Features

CRAWL4AI_MCP comes packed with advanced features tailored for LLMs and AI Assistants.

• Native MCP Integration: Works seamlessly with Claude Desktop, Cline, and any MCP-compatible AI assistant.
• LLM-Optimized Markdown: Returns scraped content directly in structured Markdown.
• Link Surfacing: Automatically extracts and surfaces internal and external links for continuous AI navigation.
• Deep Crawling: Website crawling with configurable depth.
• Analytics: Detailed crawl result statistics.
• Error Handling: Robust error and not found page handling.

Advanced Scraping Capabilities

Magic Mode

Bypass anti-bots (like Cloudflare) and simulate real browser behavior seamlessly using Playwright.

Targeted Extraction

Fetch only what you need using specific CSS selectors.

Custom JavaScript

Execute custom code before extraction (such as clicks, scrolls, or form fills).

Persistent Sessions

Keep cookies and state across requests for authenticated sites.

SPA Support

Wait for dynamic CSS selectors or set explicit pre-extraction delays, perfect for React/Vue/Angular apps.