Give your AI the power
to surf the web.

The ultimate Model Context Protocol (MCP) server for Crawl4AI.
Transform any website into LLM-optimized Markdown with structured links, enabling continuous and autonomous AI navigation.

GET STARTED VIEW SOURCE

Claude Desktop — MCP Session

User:Can you read the developer documentation at https://api.stripe.com/docs and explain their authentication flow?

Claude:

Calling tool `crawl` with args:
{ "url": "https://api.stripe.com/docs", "wait_for_selector": "#main-content", "magic": true }
✓ Page loaded successfully (Playwright Magic Mode bypassed bot protections)
✓ Extracted 42KB of pristine Markdown content
✓ Surfaced 15 internal links for further navigation

I have read the documentation. The authentication flow uses Bearer tokens. Here is how you implement it...

Built for AI Workflows

Real-world scenarios where CRAWL4AI_MCP unlocks new capabilities for your AI assistant.

Automated Research

Ask Claude to analyze multiple competitor websites. The crawler extracts pricing pages and feature matrices into clean Markdown, allowing the AI to build a comprehensive competitive analysis table in seconds.

Use case: Competitive Analysis

Deep Tech Support

Give your AI a starting documentation URL. Thanks to automatic link surfacing, the AI autonomously navigates through nested pages to find the exact API endpoint you need to solve your code error.

Use case: Autonomous Navigation

Market Intelligence

Bypass aggressive bot protections (like Cloudflare) using Magic Mode. Extract financial news articles cleanly by targeting specific CSS selectors like article.main-content, ignoring ads and noise.

Use case: Targeted Extraction

Engineered for LLMs

Not just a scraper. A fully integrated bridge between your AI assistant and the dynamic, modern web.

LLM-Optimized Markdown

Raw HTML is noisy and expensive for LLMs. Crawl4AI-MCP parses DOM elements and returns pristine, structured Markdown that fits perfectly within the context window, drastically reducing token usage.

Magic Mode

Bypass Cloudflare and anti-bot protections seamlessly by simulating real browser behavior via Playwright. Never get blocked by a 403 again.

SPA & React Support

Modern web apps are heavily reliant on JavaScript. Easily wait for specific CSS selectors or inject custom JS before extracting content.

Autonomous Navigation

Every crawl surfaces both internal and external links found on the page. This allows Claude or Cline to autonomously click through documentation, follow paginations, and navigate deeply without human intervention.

Zero Configuration.

Run the server directly using `uvx` without installing any complex dependencies. Just add it to your Claude Desktop configuration and you're ready to go.

Read the Setup Guide

// claude_desktop_config.json

{
  "mcpServers": {
    "crawl4ai": {
      "command": "uvx",
      "args": [
        "crawl4ai-mcp-llm"
      ]
    }
  }
}

Give your AI the power to surf the web.