Keytomic MCP

Keytomic MCP gives AI clients free, public remote access to bounded website Quick Audits. It translates MCP tool calls into requests to Keytomic's private crawler API without owning crawl execution or storing audit data.

The production MCP endpoint is planned for:

https://mcp.keytomic.com/

The operational health endpoint is:

https://mcp.keytomic.com/health

What this server does

The server exposes one focused workflow:

1. An MCP client starts a Quick Audit for a public HTTP or HTTPS URL.
2. The crawler API asynchronously audits up to 25 pages.
3. The client polls the Audit Job using its opaque auditId.
4. When complete, the client reads the summary and paginated category issues.

Every Quick Audit covers three fixed categories:

• crawlability
• indexability
• metadata

The MCP Adapter is intentionally small and stateless. The separate Crawler API owns job execution, durable state, results, quotas, expiry, and recovery.

MCP tools

Tool	Input	Result
start_website_audit	url	Starts a fixed 25-page Quick Audit and returns an auditId
get_audit_status	auditId	Returns lifecycle state and crawl progress
get_audit_summary	auditId	Returns the completed score and category totals
get_audit_category	auditId, category, optional cursor	Returns up to 20 issues and an optional next cursor

Typical tool sequence

start_website_audit
        |
        v
 get_audit_status ----> queued / running
        |
        v
     completed
        |
        +------> get_audit_summary
        |
        +------> get_audit_category
                    |
                    +------> repeat with nextCursor

Audit Jobs expire after 24 hours. Clients should treat auditId and pagination cursors as opaque values.

Scope

Included

• Remote MCP over Streamable HTTP at /
• Compatibility with legacy stateless MCP clients
• Structured and text tool results
• Strict validation at MCP and crawler boundaries
• Stable, client-safe error codes
• Internal bearer authentication to the Crawler API
• Health endpoint at /health
• Production multi-stage Docker image
• CI checks for formatting, types, tests, build, and Docker
• Stateless horizontal scaling

Deliberately excluded

• Website crawling inside this repository
• Local job ownership, persistence, queues, or caches
• Caller accounts, API keys, or login
• Configurable page limits or audit categories
• MCP resources, prompts, cancellation, or stdio transport
• Billing, subscriptions, or usage entitlements
• Browser extensions or npm package publication

These boundaries keep the public adapter independently deployable while the Crawler API remains the system of record.

Architecture

MCP client
    |
    | Streamable HTTP JSON-RPC
    v
Keytomic MCP Adapter
    |
    | private HTTP + internal bearer token
    v
Keytomic Crawler API
    |
    +---- execution
    +---- job state
    +---- results
    +---- quotas and recovery

The code follows four direct module boundaries:

server.ts ──> mcp/module.ts ──> crawler/port.ts <── crawler/http-adapter.ts
    │                                                    │
    └──────────────── config.ts ─────────────────────────┘

• src/server.ts composes dependencies and mounts / and /health.
• src/mcp/module.ts owns the four public tools and error mapping.
• src/crawler/port.ts defines the crawler capabilities needed by MCP.
• src/crawler/http-adapter.ts owns private HTTP details and response validation.
• Contract files keep Zod schemas beside their respective boundaries.

See Repository Design for the authoritative architecture and trade-offs.

Error contract

Tool failures remain valid MCP responses and use a small public vocabulary:

Code	Meaning
not_found_or_expired	The Audit Job is unknown or has expired
audit_not_ready	Results were requested before the job completed
capacity_rejected	Temporary crawler capacity is exhausted
crawler_unavailable	The private crawler could not complete the request
invalid_crawler_response	Crawler output failed strict validation
internal_error	An unexpected adapter failure occurred

When the Crawler API supplies a safe numeric retry delay, the error includes retryAfterSeconds. Private response bodies, internal credentials, and implementation details are never returned to MCP clients.

Security model

The public MCP endpoint is authless. Its risk is bounded by the fixed 25-page Quick Audit contract and capacity controls owned by the Crawler API.

• The internal crawler token comes only from runtime configuration.
• The token is never accepted from clients, returned in results, or logged.
• Crawler responses fail closed when their schemas do not match.
• Unknown and expired jobs share one error to avoid disclosing job existence.
• The runtime uses an unprivileged container user.
• No caller or audit data is persisted by this service.

Report vulnerabilities using the process in SECURITY.md.

Requirements

• Node.js 24
• pnpm 10
• Access to a compatible Keytomic Crawler API

Local development

Install dependencies and create local configuration:

pnpm install
cp .env.example .env

Set the required values:

CRAWLER_API_BASE_URL=https://crawler-api.example.com
CRAWLER_API_TOKEN=replace-me
PORT=3000

Start the development server:

pnpm dev

Local endpoints:

• MCP: http://localhost:3000/
• Health: http://localhost:3000/health

Commands

Command	Purpose
pnpm dev	Run the TypeScript server in watch mode
pnpm check	Check formatting and lint rules with Biome
pnpm typecheck	Type-check without emitting files
pnpm test	Build and run the Vitest suite
pnpm build	Compile production JavaScript into dist/
pnpm start	Run the compiled production server

Run the complete local quality gate:

pnpm check
pnpm typecheck
pnpm test
pnpm build
docker build --tag keytomic-mcp .

Docker

Build and run the production image:

docker build --tag keytomic-mcp .
docker run --rm \
  --publish 3000:3000 \
  --env CRAWLER_API_BASE_URL=https://crawler-api.example.com \
  --env CRAWLER_API_TOKEN=replace-me \
  keytomic-mcp

The image uses Node.js 24 Alpine, contains production dependencies only, runs as an unprivileged user, and checks /health for container health.

Testing

The test suite exercises observable behavior at three seams:

1. MCP JSON-RPC through the in-memory Hono handler.
2. Crawler HTTP through an injected fetch.
3. The compiled Node server through real loopback HTTP.

Remote crawler behavior is the only mocked boundary. Private helpers are tested through public outcomes.

Deployment

The intended production topology is one stateless container deployed through Dokploy:

1. Dokploy builds the root Dockerfile.
2. Runtime configuration supplies the crawler URL and token.
3. Traefik routes mcp.keytomic.com to container port 3000.
4. Route53 points the public hostname to the Dokploy server.
5. Dokploy monitors /health.

CI validates pushes and pull requests but does not deploy.

Product roadmap

The current four-tool Quick Audit is the stable product core. Future work is prioritized by user value and operational evidence; roadmap items are direction, not a compatibility promise.

1. Production launch and reliability

• Deploy and verify mcp.keytomic.com.
• Add end-to-end smoke checks against the production MCP and Crawler API.
• Define availability, latency, capacity, and error-rate service indicators.
• Document incident response, token rotation, rollback, and recovery.

Exit criterion: the public endpoint is observable, supportable, and safe to operate continuously.

2. MCP client experience

• Publish setup examples for major remote MCP clients.
• Add copy-paste tool workflows and troubleshooting guidance.
• Validate structured outputs and legacy compatibility across supported clients.
• Gather failure and completion feedback without storing audit content.

Exit criterion: a new user can connect, run an audit, and understand the result without operator help.

3. Audit usefulness

• Use real Quick Audit feedback to identify missing explanations or result context.
• Improve issue titles, descriptions, and remediation guidance at the Crawler API boundary.
• Evaluate additional bounded categories only when the Crawler API owns their contracts and execution.
• Preserve predictable limits and explicit pagination as result depth grows.

Exit criterion: expansions solve demonstrated user problems and retain a small, stable MCP surface.

4. Sustainable public access

• Measure aggregate capacity pressure and abuse patterns outside audit content.
• Tune domain and global capacity limits in the Crawler API.
• Evaluate fair-use controls only if authless access becomes operationally unsafe.
• Keep client-facing capacity errors actionable and retryable.

Exit criterion: free access remains reliable without moving crawler ownership or durable user state into this adapter.

5. Protocol and platform maturity

• Review the pinned MCP SDK when its required APIs become stable.
• Add protocol capabilities only when they materially improve the audit workflow.
• Keep dependency upgrades reviewed, reproducible, and covered by compatibility tests.
• Revisit deployment topology only when production evidence exceeds the limits of one stateless container.

Exit criterion: platform changes reduce operational risk or improve the user workflow without speculative architecture.

Roadmap decision rules

Every proposed capability must answer:

1. What observed user or operational problem does it solve?
2. Does it belong in the MCP Adapter or the Crawler API?
3. Can the adapter remain stateless?
4. Does it preserve a bounded, understandable public contract?
5. How will compatibility, security, and operations be verified?

Hard-to-reverse changes require an ADR before implementation.

Project documentation

• Domain language and invariants
• Repository design
• Implementation plan
• Architecture decisions
• Security policy
• License