MCP SearchEngine

An MCP server that searches & scrapes multiple sources, then renders an interactive picker UI (a FastMCP App) so you can click which sources you want to delve into. Built on the FastMCP 4 prerelease and the MCP Apps extension.

How it works

LLM ──search(query)──► server ──► queries web, DuckDuckGo, Wikipedia,
│                                  arXiv & Hacker News concurrently
│                                  and returns normalized results
│
│                        ┌──── ui://search-engine/picker.html ────┐
│                        │ interactive card grid, source filters, │
│◄────── rendered app ───┤ multi-select, "Delve into selected"   │
│                        └──────────────┬─────────────────────────┘
│                                       │ callServerTool('delve_sources')
LLM ◄── delve_sources(picks) ◄──────────┘
        scrapes each picked URL and
        returns full readable content

The flow is symmetrical: the LLM gets the same JSON the UI renders, so it can also pick sources programmatically and call delve_sources directly — the UI is additive, not a gate.

Sources

Source	Backend	Notes
web	DuckDuckGo instant-answer API	abstract + related topics
duckduckgo	DuckDuckGo HTML endpoint	general web results, no key
wikipedia	Wikipedia OpenSearch / summary API
arxiv	arXiv Atom API	papers, authors, links
hackernews	Algolia HN search API	points + comment counts
reddit	Reddit public JSON search	no API key, own UA
x	Nitter HTML instances	keyword search; nitter mirror configurable via X_NITTER_INSTANCE
perplexity	Perplexity Chat API	opt-in, needs PERPLEXITY_API_KEY; not in the defaults

A failing source never sinks the search — it comes back as an error card in the UI.

Tools

• search(query, sources?, max_per_source?) — search all sources and render the picker UI. Returns JSON { mode: "pick", results: [...] } plus a text summary.
• delve_sources(picks, max_chars?) — scrape the picked URLs concurrently and return full extracted text per page. picks items need at least a url.

The interactive UI

search_engine/ui/picker.html is served as a ui:// resource (text/html;profile=mcp-app) and rendered in a sandboxed iframe by hosts that support MCP Apps (Claude, ChatGPT, VS Code, Goose, …). It uses the @modelcontextprotocol/ext-apps SDK:

• app.ontoolresult — receives the search payload pushed by the host
• app.callServerTool({ name: "delve_sources", … }) — delves into picked sources

Click cards to multi-select, double-click to delve into a single source, filter by source chip, then press Delve into selected. The scraped pages are shown in the UI and returned to the conversation.

Install & run

python -m venv .venv
.\.venv\Scripts\python -m pip install -e .

The pyproject.toml pins the FastMCP 4 prerelease (fastmcp==4.0.0b1 + fastmcp-slim==4.0.0b1 constraint, per the v4 upgrade guide).

Run over stdio (what most MCP clients use):

.\.venv\Scripts\python -m search_engine

or test it in-process:

.\.venv\Scripts\python test_client.py

There is also a Playwright-driven UI harness (test_ui_harness.py) that renders the real picker HTML in Chromium with the SDK stubbed, clicks through select → filter → delve, and screenshots each stage. Run it with .\.venv\Scripts\python test_ui_harness.py (requires playwright install chromium).

Privacy & proxying

All network egress (search + delve_sources scraping) goes through a single privacy layer (search_engine/net.py):

• No local persistence, ever. There is no browser: no cache, history or cookies on disk. Every request gets a fresh ephemeral client with its own empty cookie jar that is discarded afterwards.
• Proxy via env — SEARCH_PROXY_URL (falling back to HTTPS_PROXY, then ALL_PROXY). Supports http://, socks5:// and socks5h:// (the h means DNS is resolved at the proxy, avoiding local DNS leaks — handy for Tor, e.g. SEARCH_PROXY_URL=socks5h://127.0.0.1:9050). Requires httpx[socks].
• External sandbox relay — set SEARCH_SANDBOX_URL to a disposable sandbox (e.g. a self-hosted agentOS/Rivet VM relay, https://agentos-sdk.dev/docs/apps/) and every fetch is rewritten to {SEARCH_SANDBOX_URL}<url-encoded target> so the local machine never touches the target site. agentOS is TS/Rivet, not Python-importable — this env-relay is the integration seam.
• VPN note — a VPN is OS-level: when the OS is on one, all traffic already routes through it; the proxy and sandbox options are orthogonal to that.

See .env.example for all options. No secrets live in code and URLs that might carry credentials are never logged.

Deploy to Prefect Horizon (hosted, off your PC)

Prefect Horizon is the managed MCP hosting platform from the FastMCP team. Deploying there runs the engine on their infra — so all searching/scraping egresses from Prefect's IPs, not your PC — and serves it at an OAuth-protected URL like https://<name>.fastmcp.app/mcp. Free for personal projects.

1. Verify the entrypoint locally (this is exactly what Horizon sees):

   .\.venv\Scripts\fastmcp inspect search_engine/server.py:mcp
   

   You should see 2 tools (search, delve_sources) and 1 resource (picker_view). server.py self-bootstraps the repo root onto sys.path so the standalone import Horizon performs resolves the package.
2. Dependencies — requirements.txt pins fastmcp==4.0.0b1 + fastmcp-slim==4.0.0b1 explicitly (pip doesn't read pyproject's [tool.uv] constraint), plus httpx[socks] and beautifulsoup4. Horizon auto-detects and installs it.
3. Push to GitHub and deploy at horizon.prefect.io: sign in with GitHub, select the repo, set entrypoint search_engine/server.py:mcp, pick a server name (sets the URL), enable Authentication (built-in OAuth), and add secrets (PERPLEXITY_API_KEY, optionally X_NITTER_INSTANCE / SEARCH_PROXY_URL / SEARCH_SANDBOX_URL).
4. Connect — Horizon redeploys on push and generates connection snippets for Claude/Cursor/VS Code. Verify with its Inspector / ChatMCP, and confirm egress via delve_sources on an IP-echo URL.

Note: Horizon is serverless (cold starts, stateless per call, ~170s timeout) — fine for this stateless engine. Its egress is Prefect's shared IP; it cannot run a host-level VPN (use a VPS instead if you ever need a dedicated VPN exit).

Example client configuration

{
  "mcpServers": {
    "search-engine": {
      "command": "C:\\Users\\kevin\\Projekt\\MCP-SearchEngine\\.venv\\Scripts\\python.exe",
      "args": ["-m", "search_engine"]
    }
  }
}

Layout

search_engine/
├── __init__.py
├── __main__.py          # python -m search_engine
├── server.py            # FastMCP server: tools + ui:// resource
├── sources.py           # multi-source search + scraping engine
└── ui/
    └── picker.html      # interactive MCP App source-picker UI