Search: crawler
Search MCP servers and agent skills by name, description, category or topic — 12 results.
schemacrawler/SchemaCrawler-MCP-Server-Usage
Connect to any relational database, and be able to get valid SQL, and ask questions like what does a certain column prefix mean.
Mingye-Lu/AgenticCrawler
Autonomous LLM-powered web crawler with 17 browser tools and goal-driven agent. Single binary, stealth browsing, 25 LLM providers.
us/crw
fastCRW — open-source (AGPL-3.0), self-hostable Rust web crawler & search API for AI agents. Tools: `scrape`, `crawl`, `map`, and SearXNG-backed `search`. Single ~6MB static binary; reproducible 1K-URL benchmarks faster than hosted alternatives. Hosted MCP at [fastcrw.com/mcp](https://fastcrw.com/mcp) (Streamable HTTP, OAuth) or self-host.
0xMassi/webclaw
Web content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. TLS fingerprinting bypasses anti-bot without a browser. 67% fewer tokens than raw HTML. `npx create-webclaw` auto-configures Claude, Cursor, Windsurf, Codex, OpenCode.
just-every/mcp-read-website-fast
Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown while preserving links. Features Mozilla Readability, smart caching, polite crawling with robots.txt support, and concurrent fetching.
alex-on-ai/WebReaper
AI-native web scraper MCP server. Single binary, returns clean markdown, MIT-licensed Firecrawl alternative.
pragmar/mcp-server-webcrawl
Advanced search and retrieval for web crawler data. Supports WARC, wget, Katana, SiteOne, and InterroBot crawlers.
segentic-lab/periscope-mcp
Website & web-app testing built for AI agents rather than raw browser bindings: 66 Playwright tools with hard assertions, auto form-fill, persistent authenticated sessions (incl. interactive login for 2FA/SSO), network mocking, real INP, and accessibility/SEO/GEO (llms.txt, AI-crawler access, WebMCP) + Lighthouse audits.
serkan-ozal/driflyte-mcp-server
The Driflyte MCP Server exposes tools that allow AI assistants to query and retrieve topic-specific knowledge from recursively crawled and indexed web pages.
maxaeo/maxaeo-ai-visibility-mcp
Local-first AI visibility, GEO/AEO, and llms.txt audit MCP server for Claude, Codex, Cursor, and other agents. Checks AI crawler access, robots, sitemap, canonical, metadata, noindex, and JSON-LD; returns local-only and technical foundation scores, top issues, and action plans. Install: `npx -y maxaeo-ai-visibility-mcp`.
epistemedeus/ai-readiness
Check whether a website is visible to AI search (ChatGPT, Perplexity, Claude, Google AI Overviews). Scores AI-crawler access, JSON-LD structured data, title/meta, Open Graph, sitemap, and llms.txt, with a specific fix for each gap. Dependency-free, no API keys. Install: `npx -y github:epistemedeus/ai-readiness mcp`.
hanselhansel/aeo-cli
Audit URLs for AI crawler readiness — checks robots.txt, llms.txt, JSON-LD schema, and content density with 0-100 AEO scoring.