HyperCrawl

Self-hosted, cookie-aware web-automation engine. Turns any website into callable MCP tools (DOM-selector based), runs them headlessly with stealth on our own servers, and pulls live auth cookies on-demand from a real logged-in browser. Internal NEB tool — works alongside Apify.

Status: production on the hypercrawl-vm GCE VM (Caddy auto-TLS + Firebase-Auth gate, hypercrawl.service). Capabilities: the acquire() capability ladder (hypercrawl_acquire — API replay/hydration → learned tool → DOM agent → set-of-marks → vision → specialist), multi-page discovery (hypercrawl_map, hypercrawl_crawl_site), cross-crawl change tracking (stateless previousHashes or DB-backed track streams, both intents), web search (hypercrawl_search), plus HTTP MCP transport, ProxyPool, and the Apify fallback adapter. See docs/USAGE.md (tools) and deploy/gcp/README.md (deploy/update — includes the required npm install + npx playwright install steps).

Browser sessions

Beyond one-shot acquisition, HyperCrawl exposes a stateful browser surface: 15 hypercrawl_browser_* MCP tools that drive a single live page across many calls, so an agent can read a page, click into it, fill a form, and submit — each step landing on the same tab. hypercrawl_browser_open returns a sessionId every other tool takes.

Sessions are a bounded resource: each pins a real Chromium context, so the server caps concurrency (HYPERCRAWL_MAX_SESSIONS, default 4) and sweeps sessions idle past five minutes. Close yours when done. An unknown sessionId is a loud error, never a silently fresh page.

Alongside them: hypercrawl_analyze / hypercrawl_quality (page structure, and whether a stored template's selectors still resolve against the live page — the signal a redesign broke it), hypercrawl_auth_save / _auth_list, and hypercrawl_metrics / _events / _speed / _setup for in-process observability. Tool reference and every env var: docs/USAGE.md.

Two capabilities are wired only when the server is started via hypercrawl serve — the auth pair and hypercrawl_setup — because their implementations live in the CLI package. Unwired, they say so rather than returning an empty result that would read as "nothing configured".

Deploy to GCP

To run the full engine (including authed cookie-pull scraping) on a dedicated GCE VM with Caddy auto-TLS and a Firebase Auth token gate, use the idempotent provisioning scripts and operator runbook in deploy/gcp/README.md. Design: docs/superpowers/specs/2026-05-29-hypercrawl-gcp-deploy-design.md.