searxng-mcp-scraper

MCP server exposing SearXNG search and HTTP fetch tools for AI agents.

A self-contained Model Context Protocol server that turns SearXNG (a self-hosted metasearch engine) into two MCP tools you can wire into Claude Desktop, Cursor, or any MCP-compatible client:

• search(query, ...) — ranked web results from SearXNG's JSON API
• fetch(url) — cleaned main text of any public http(s) URL

No API keys, no third-party tracking, no rate limits beyond what your SearXNG instance imposes.

Installation

# From source (recommended during dev)
git clone https://github.com/your-org/searxng-mcp-scraper
cd searxng-mcp-scraper
uv sync --extra dev
uv pip install -e .

# Or with plain pip
pip install -e .

Configuration

The server reads configuration from environment variables (or a .env file at startup):

Variable	Required	Default	Description
SEARXNG_BASE_URL	yes	—	Base URL of your SearXNG instance, e.g. http://localhost:8888
SEARXNG_DEFAULT_CATEGORY	no	general	Default search category
SEARXNG_DEFAULT_ENGINES	no	wiby,naver,seznam,marginalia,wikipedia,duckduckgo_html	Default engine list when a caller doesn't pass engines. Curated to the keyless engines that return results from this machine's egress IP (Brave/Google/Startpage/DuckDuckGo are rate-limited here). Set to empty to use SearXNG's own default set.
SEARXNG_DEFAULT_LANGUAGE	no	en	Default language code
SEARXNG_DEFAULT_SAFESEARCH	no	0	Default safesearch: 0/1/2
SEARCH_MAX_RESULTS_CAP	no	50	Hard cap on results per call
SEARCH_TIMEOUT_S	no	30	HTTP timeout (seconds) for search
FETCH_TIMEOUT_S	no	20	HTTP timeout (seconds) for fetch
FETCH_MAX_BYTES	no	5_000_000	Refuse to read more than this many bytes per fetch
FETCH_MAX_REDIRECTS	no	5	Max HTTP redirects to follow
FETCH_ALLOW_PRIVATE	no	false	Allow fetching private/loopback URLs (SSRF guard, default on)
USER_AGENT	no	searxng-mcp-scraper/0.1	User-Agent header for all requests
LOG_LEVEL	no	INFO	Stderr log verbosity: DEBUG/INFO/WARNING/ERROR
MCP_HOST	no	127.0.0.1	Bind host when --transport streamable-http
MCP_PORT	no	8765	Bind port when --transport streamable-http
MCP_CORS_ORIGINS	no	http://localhost,http://localhost:*,http://127.0.0.1,http://127.0.0.1:*	Comma-separated CORS allow-origins. Use * to allow any origin (not recommended on LAN).
BLOG_SCRAPE_OUTPUT_DIR	no	~/scraped-blogs	Directory for scrape_blog / deep_scrape Markdown output and assets
BLOG_SCRAPE_MAX_POSTS	no	200	Max post pages per blog scrape
BLOG_SCRAPE_CONCURRENCY	no	5	Max parallel post fetches
BLOG_SCRAPE_DISCOVERY_PAGES	no	10	Max HTML index pages to scan when no feed is found
DEEP_SCRAPE_MAX_DOCUMENTS_PER_BLOG	no	100	Max linked documents downloaded/extracted by one deep_scrape call
DEEP_SCRAPE_MAX_DOCUMENT_BYTES	no	50_000_000	Refuse a single linked document above this size
DEEP_SCRAPE_MAX_IMAGES_PER_BLOG	no	200	Max images downloaded by one deep_scrape call
DEEP_SCRAPE_MAX_IMAGE_BYTES	no	15_000_000	Refuse a single image above this size
DEEP_SCRAPE_CONCURRENCY	no	5	Max parallel document/image downloads during deep_scrape
FIRECRAWL_FALLBACK_ENABLED	no	true	Retry a failed/thin static fetch through Firecrawl (see below)
FIRECRAWL_BASE_URL	no	http://127.0.0.1:8788	Firecrawl API base — the local keyless proxy by default
FIRECRAWL_API_KEY	no	—	Optional. Empty uses the keyless free tier
FIRECRAWL_TIMEOUT_S	no	60	Timeout for a fallback scrape (Firecrawl renders before responding)
FIRECRAWL_MIN_TEXT_CHARS	no	500	Static extractions shorter than this are treated as failures and retried

Note: your SearXNG instance must have json enabled in search.formats under settings.yml, or search will return searxng_unavailable with a 403.

PDF extraction (optional, local, keyless)

deep_scrape downloads linked PDFs and extracts their text. By default it uses markitdown. For a much faster and cleaner text extraction on text-based PDFs, install the optional extra:

uv sync --extra pdf-extras          # or: pip install "searxng-mcp-scraper[pdf-extras]"

This pulls in pdf-inspector — Firecrawl's open-source Rust PDF engine (no API key, no cloud, runs locally). When installed, PDF extraction routes through it first:

• text-based PDFs are read straight from the PDF internals (fonts, text operators) in milliseconds.
• scanned / image-only PDFs are correctly classified as needing OCR (the library reports pdf_type: scanned). It has no OCR of its own, so those fall through to markitdown (which may have an OCR backend); if markitdown also returns nothing, the document surfaces an honest DocumentExtractionFailed instead of a silent empty string.

The dependency is optional: without it, PDFs simply use markitdown. Nothing in the core package depends on a native wheel.

Firecrawl fallback (JS-rendered pages)

fetch does a plain HTTP GET and runs trafilatura over the HTML. That is fast and free, but it returns nothing useful for client-rendered pages: the served body is an empty root div, so trafilatura scrapes up the meta description and a nav label and returns a couple hundred characters of noise. Measured on real sites:

URL	static	with fallback
vercel.com/templates	208 chars	15,972
notion.so/product	156 chars	7,737
linear.app/method	326 chars	1,126
excalidraw.com	57 chars	655
app.slack.com	2,764 chars	2,764 (static kept — already good)

When a fetch errors, returns empty text, or returns fewer than FIRECRAWL_MIN_TEXT_CHARS, the URL is retried once through Firecrawl's /v2/scrape, which renders the page server-side and returns clean markdown. Results carry extractor: "firecrawl" so you can tell which path produced them.

Firecrawl's hosted API has a keyless free tier for /v2/scrape and /v2/search, so this costs nothing and needs no account. The catch: the tier rejects any Authorization header, while most clients insist on sending one. FIRECRAWL_BASE_URL therefore points at a small local proxy that strips the header:

python3 scripts/firecrawl_keyless_proxy.py   # 127.0.0.1:8788

The proxy is vendored here (no external dependency) and forwards a real key if you set FIRECRAWL_UPSTREAM_KEY in its environment — which unlocks /v2/map, /v2/crawl, and /v2/extract through the same URL. Key-gated paths without a key get a clear error rather than a confusing upstream 403.

Set FIRECRAWL_API_KEY and point FIRECRAWL_BASE_URL at https://api.firecrawl.dev to skip the proxy and use a real account instead.

The fallback never overrides a guard. invalid_url and private_network_blocked are SSRF/scheme decisions, not extraction failures — routing those through a third-party renderer would defeat the guard, so they are excluded. oversize is excluded too (re-rendering the same huge page just burns the cap again). If the fallback fails or returns less text than the static path, the original result is kept unchanged. Set FIRECRAWL_FALLBACK_ENABLED=false for fully air-gapped runs.

Running

The server supports two transports. Pick the one your MCP client speaks:

stdio (default — Claude Desktop, Cursor, etc.)

# From a checkout, with .env in the cwd
SEARXNG_BASE_URL=http://localhost:8888 searxng-mcp-scraper

# Or via uv
SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper

# Or as a module
SEARXNG_BASE_URL=http://localhost:8888 python -m searxng_mcp_scraper

The server speaks MCP over stdio. It writes logs to stderr; stdout is reserved for the JSON-RPC stream and must not be polluted.

streamable-http (llama-ui, Open WebUI, browser-based clients)

SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper --transport streamable-http

This binds to http://127.0.0.1:8765/mcp by default. Override with MCP_HOST / MCP_PORT env vars. CORS is pre-configured for http://localhost:* and http://127.0.0.1:* — any browser on your machine can connect.

If you need to bind to 0.0.0.0 (LAN), set MCP_CORS_ORIGINS to a narrower list. There is no auth in v1 — do not expose this to the public internet without putting it behind a reverse proxy with auth.

llama-ui / Open WebUI config

For browser-based MCP clients, run the server in HTTP mode and paste the URL into the client's "MCP server URL" field:

1. In one terminal:

   SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper --transport streamable-http
   

2. In the UI, add a new MCP server with URL http://127.0.0.1:8765/mcp (no auth headers required for local).
3. The search and fetch tools will appear in the model's tool list.

Claude Desktop config

Add this to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "searxng-scraper": {
      "command": "searxng-mcp-scraper",
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8888"
      }
    }
  }
}

If you installed from source and the binary isn't on your PATH, point command at uv:

{
  "mcpServers": {
    "searxng-scraper": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/searxng-mcp-scraper", "run", "searxng-mcp-scraper"],
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8888"
      }
    }
  }
}

Restart Claude Desktop; you should see search and fetch tools appear.

Tool reference

search(query, ...) -> {results, suggestions, number_of_results}

Parameter	Type	Default	Description
query	string	(required)	Search query string
categories	list[string]	["general"]	SearXNG categories, e.g. ["general"], ["images"]
engines	list[string]	null	Restrict to specific engines, e.g. ["google", "bing"]
language	string	"en"	Language code
pageno	int	1	Page number (1-indexed)
time_range	"day"/"month"/"year"/null	null	Time filter
safesearch	0/1/2	0	Safe-search level
max_results	int	10	Cap on results returned (1–50)

fetch(url) -> {url, final_url, content_type, text, byte_count}

Parameter	Type	Description
url	string	An http:// or https:// URL

Strips <script>, <style>, <nav>, <header>, <footer>, <aside> blocks via trafilatura, and collapses whitespace. Refuses non-http(s) schemes and (by default) private network targets. Returns a structured error on timeout, oversize, or non-2xx — never crashes.

scrape_blog(blog_url) -> {output_path, post_count, ...}

Discovers a blog's RSS/Atom feed first, falls back to scanning HTML index pages, fetches post text, and writes one Markdown file under BLOG_SCRAPE_OUTPUT_DIR.

deep_scrape(blog_url) -> {output_path, assets_dir, documents_*, images_*}

Does everything scrape_blog does, plus:

• discovers linked documents from each post (.pdf, .docx, .csv, .md, etc.)
• downloads raw document files into deep_assets/
• extracts document text with MarkItDown and embeds it in the Markdown output
• discovers images from <img src>, lazy-load attributes, srcset, and direct image links
• downloads bounded image files into deep_assets/images/
• records per-image URL, saved path, content type, byte count, and any failure in the Markdown

The returned summary includes images_found, images_saved, and image_byte_count. One bad image/document is recorded in-band and does not fail the whole blog scrape.

Development

uv run pytest          # run the full test suite
uv run mypy src        # static type checks
uv run ruff check      # lint

License

MIT.