MCP 服务器

scout

MCP server that fetches web pages, extracts clean markdown (reducing token count), caches results, and provides searchable reading history.

README

🧭 scout

The agent's web reader.

Raw HTML is a terrible thing to feed a model — a 380 KB Wikipedia page is ~95k tokens of markup for a few KB of prose. scout fetches a URL and gives back clean, readable markdown (headings, links, code, lists — the substance, none of the chrome), typically ~90% smaller than the HTML. Every page is cached, so re-reading is free and your whole reading history is searchable.

Part of tools-for-agents. Zero dependencies — Node's built-in fetch + a regex "readability-lite" extractor + node:sqlite (FTS5). Pairs naturally with cortex: scout clips the web, cortex files it into your second brain.

Why

Without scout	With scout
Feed raw HTML to the model → ~95k tokens of `<div>`s	`scout_fetch` → ~5k tokens of clean prose
Re-fetch the same page every time you need it	Cached — re-reads are free (`--fresh` to bust)
"What did that article say about X?" → fetch again, re-read	`scout_search "X"` across everything you've read
No memory of what you've researched	A searchable reading history on disk

CLI

scout fetch https://en.wikipedia.org/wiki/Zettelkasten     # → clean markdown (cached)
scout fetch https://example.com/post --tokens 3000         # cap the returned size
scout fetch https://api.example.com/data.json --raw        # skip extraction
scout search "luhmann note linking" -k 5                   # search your reading history
scout links https://news.ycombinator.com --limit 30        # outbound links to crawl next
scout list | scout forget https://old.example.com | scout stats

Cache location: $SCOUT_DB (default ./.scout/cache.db).

MCP server (for agents)

{
  "mcpServers": {
    "scout": { "command": "node", "args": ["/abs/path/to/scout/mcp/mcp-server.js"],
               "env": { "SCOUT_DB": "/abs/path/to/.scout/cache.db" } }
  }
}

Tools

Tool	Use it to…
`scout_fetch`	Read a web page as clean, token-budgeted markdown (cached; `fresh` to re-fetch).
`scout_search`	Search every page you've already read — ranked snippets, no re-fetch.
`scout_links`	Extract a page's outbound links (absolute URLs + text) to decide where to go next.
`scout_list`	Your recent reading history.
`scout_forget`	Drop a page from the cache.
`scout_stats`	Pages cached, bytes stored, last fetch.

The research loop (with cortex)

scout_fetch the page → clean markdown.
scout_search your history to connect it to what you've already read.
cortex_capture the useful parts into your second brain, then cortex_write distilled, [[linked]] notes.
Next time, cortex_search / scout_search recall it instead of fetching the web again.

How it works

Fetch uses Node's global fetch (follows redirects, 20 s timeout, a plain user-agent).
Extraction is regex-based readability-lite: strip <script>/<style>/<nav>/<footer>/…, pick the densest <article>/<main>/<body> region, convert headings, links (resolved to absolute), code, lists, bold/italic, and decode HTML entities. Not a full DOM parse — but it reliably turns an article into readable prose at a fraction of the tokens.
Cache is a node:sqlite table keyed by URL; the same URL returns instantly unless fresh. An FTS5 mirror makes the whole history searchable by bm25, filled to a token budget (≈4 chars/token) — the same discipline as lens and cortex.
Non-HTML responses (JSON, plain text) are stored verbatim.