site-crawler-mcp

A site-wide crawler MCP server built on crawl4ai (Apache-2.0). It crawls every internal link of a website like a search-engine spider (BFS), filters pages by publish time / title / URL, and can operate on crawled pages (summarize, click links, download files). Results are returned as JSON and persisted to SQLite for reuse by any project.

中文简介：一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装， 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作（LLM 概括/点击链接/下载文件）、 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意 MCP 客户端即可全局复用。

Features

• ✅ Site-wide BFS crawl — traverse all internal links (max_depth / max_pages controllable)
• ✅ Time filter — extract publish time from page meta / JSON-LD / URL first, fall back to crawl time when unavailable (result tagged with time_source)
• ✅ Title filter — include / exclude by title keyword (case-insensitive)
• ✅ URL pattern & domain filters — glob/regex URL matching, same-domain restriction
• ✅ Polite crawling — respects robots.txt and rate-limits by default (toggleable)
• ✅ Page operations — LLM summarize (LiteLLM: DeepSeek / GLM / OpenAI…, local fallback), click a specific link (CSS selector or link text), download files
• ✅ SQLite persistence — query_crawls tool to reuse crawled data in later projects

Tools

Tool	Description
crawl_site	BFS site-wide crawl. Params: start_url, max_depth, max_pages, published_after/before, title_contains/title_exclude, url_pattern, include_external, respect_robots, rate_limit
scrape_page	Scrape a single page (markdown / title / publish time / links)
summarize_page	Summarize page content. mode=auto (LLM first → local fallback) / llm / local; llm_provider (LiteLLM format, e.g. deepseek/deepseek-chat), llm_api_key_env (default DEEPSEEK_API_KEY)
click_link	Click a link inside the page (selector CSS or link_text) and scrape the target page
download_file	Download page files to output dir (default E:\Reasonix-项目\crawler-output)
query_crawls	Query persisted crawl results from SQLite (title/URL/time filters)

Requirements

• Python ≥ 3.12 (tested on 3.12.13)
• uv recommended (optional — plain pip works too)
• Playwright browsers: python -m playwright install chromium (or point PLAYWRIGHT_BROWSERS_PATH at an existing browser install)

Install & Register

# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .

# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium

# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name    = "site-crawler-mcp"
type    = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args    = ["-m", "site_crawler_mcp.server"]

For Claude Desktop / Cursor, add the same command/args under mcpServers in their config files.

Quick Start (Python API)

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("crawl_site", {
                "start_url": "https://example.com",
                "max_depth": 2,
                "max_pages": 20,
                "title_contains": "Example",
            })
            print(res.content[0].text)

asyncio.run(main())

How the time filter works

crawl4ai's URL filters work on URLs only, so content-level filtering is implemented here:

1. URL-level pruning — TimeRangeFilter / TitleFilter (URL date patterns, URL keywords)
2. Content-level — after each page is fetched, publish time is extracted from <meta property="article:published_time">, JSON-LD datePublished, <time datetime>, or YYYY/MM/DD in the URL. If none found, crawl time is used (decision documented as time_source).

Compliance

• Respects robots.txt and rate limits by default to avoid stressing target sites or getting IP-banned.
• For learning / research / your own sites; please follow target sites' ToS and local laws.

License

MIT

Built on crawl4ai (Apache-2.0).