🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

Crawl4AI MCP Server

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents.

Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

Features

• 🔧 MCP Tools: Exposes 4 powerful tools: scrape, crawl, crawl_site, crawl_sitemap via stdio MCP server
• 🌐 Web Scraping: Single-page scraping with markdown extraction
• 🕷️ Web Crawling: Multi-page breadth-first crawling with depth control
• 🧠 Adaptive Crawling: Smart crawling that stops when sufficient content is gathered
• 🛡️ Safety: Blocks internal networks, localhost, and private IPs
• 📱 Agent Ready: Works with OpenAI Agents SDK, Cursor, and Claude Code
• ⚡ Fast: Powered by Playwright and Crawl4AI's optimized extraction

🚀 Quick Start

Choose between Docker (recommended) or manual installation:

Option A: Docker (Recommended) 🐳

Docker eliminates all setup complexity and provides a consistent environment:

Option A1: Use Pre-built Image (Fastest) ⚡

# No setup required! Just pull and run the published image
docker pull uysalsadi/crawl4ai-mcp-server:latest

# Test it works
python test-config.py

# Use directly in MCP configurations (see examples below)

Option A2: Build Yourself

# Clone the repository
git clone https://github.com/uysalsadi/crawl4ai-mcp-server.git
cd crawl4ai-mcp-server

# Quick build and test (simplified)
docker build -f Dockerfile.simple -t crawl4ai-mcp .
echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "test", "version": "1.0"}}}' | docker run --rm -i crawl4ai-mcp

# Or use helper script (full build with Playwright)
./docker-run.sh build
./docker-run.sh test
./docker-run.