myscrape

A self-contained, single-codebase web-research MCP server for local LLM agents. It mimics the spirit of Claude's WebSearch / WebFetch and adds a cooked web_research tool that runs the whole search → fetch → synthesize loop with a local LLM inside the box.

📖 Full user guide → docs/USER_GUIDE.md — setup, running (local/Docker), the tools, configuration, and troubleshooting.

Tools

Tool	Altitude	LLM inside?
web_search	raw	no
web_fetch	raw	no
web_research	cooked	yes

See specs/SPEC.md for the interface contract, and the design docs: specs/IMPLEMENTATION_OPTIONS.md, specs/TECH_STACK.md, specs/BENCHMARK.md. Decisions and checkpoints are logged in specs/DEVLOG.md; empirical work in specs/EXPERIMENTS.md; search-provider options in specs/PROVIDERS.md.

Tooling

Stable tools only — no type checker; we rely on tests + linting instead.

• uv — env, deps, locking, running
• Ruff — lint + format
• pytest — tests (TDD: red → green → refactor)

Quickstart

uv sync                                          # create venv, install deps
uv run pytest                                    # run the test suite
uv run ruff check . && uv run ruff format --check .   # lint + format gate
uv run bench                                     # run the stability benchmark (any time)

Running the MCP server

uv run myscrape          # stdio transport — a local MCP client spawns this

web_research needs a local, OpenAI-compatible LLM endpoint (Ollama / llama.cpp / LM Studio). Configure via MYSCRAPE_* env vars (see src/myscrape/config.py):

export MYSCRAPE_LLM_BASE_URL=http://localhost:11434/v1
export MYSCRAPE_LLM_MODEL=qwen2.5:14b   # eval winner on a 12GB GPU (see specs/EXPERIMENTS.md)

web_search and web_fetch need no LLM. The proven stability operating point (8s + 2s jitter) is the default; override with MYSCRAPE_REQUEST_MIN_INTERVAL.

On a server (Docker)

Uses the host's native Ollama (GPU) — see specs/SERVING.md.

ollama pull qwen2.5:14b        # on the host (the eval winner)
docker compose up --build      # myscrape on :8000, talks to host Ollama

The container is one process, includes Chromium for dynamic fetch, speaks streamable-http at http://localhost:8000/mcp, and caps concurrent research at 2 (the GPU serializes synthesis — see EXPERIMENTS E-007). For a self-contained Ollama-in-container setup with GPU passthrough: docker compose --profile ollama up.

Use it from a coding session (Claude Code)

A project .mcp.json points at the running server:

claude mcp add --transport http myscrape http://localhost:8000/mcp   # or use .mcp.json

Then web_search, web_fetch, and web_research are available as tools in the session. (For local use without Docker, run uv run myscrape over stdio instead.)

Tools

Tool	Input (key fields)	Returns
web_search	query, max_results	ranked results (no fetch, no LLM)
web_fetch	url, max_tokens	clean markdown + metadata (no LLM)
web_research	question, effort, return_mode	cited answer + sources + coverage

Status

Implemented end-to-end: all three tools, the full search → fetch → rank → synthesize loop, behind a stability gate that passes (ratelimit_rate == 0, extraction 100% over fetchable pages). Static and dynamic (Playwright) fetch work — web_fetch auto-escalates to a headless browser for JS-rendered pages (live-validated on quotes.toscrape.com/js). See specs/DEVLOG.md for the full build log and specs/EXPERIMENTS.md for the empirical work.