mcp-queue-doctor

An MCP server that diagnoses Postgres job queues — pg-boss and graphile-worker. Retry storms, stuck workers, missed schedules, expiry overruns: what is broken, why, and the safest way to recover.

❌ "3 jobs in enrichment/corpus-fill are in state 'failed'."

✅ "enrichment/corpus-fill failed 140 times over 3m, peaking at 50 failures in a
   single minute. 91% share one error, which looks like an upstream rate limit.
   This is one fault reproduced many times, not many separate faults — so the fix
   belongs at the source, and retrying the jobs individually will reproduce it.

   Recovery, safest first:
     1. Stop enqueuing to this queue — every new job feeds the same failure.
     2. Confirm when the upstream quota resets; treat that as the time to resume.
     3. Add a cooldown gate after N consecutive 429s.
     ⚠ Do NOT bulk-retry yet — the upstream is still limited.

   Evidence: 140 failures, 91% 'HTTP 429 Too Many Requests (daily quota
   exceeded)', peak 50/min, busiest minutes [...], 12 other errors [...]"

The second answer is the product. Every finding carries the evidence it was drawn from, so you — or an agent — can check the reasoning instead of trusting it.

Where the heuristics come from

The rules are extracted from a morning health check that has run daily in production since April 2026 against a pg-boss instance driving ~30 cron queues. Every threshold was tuned by a real false positive or a real missed failure, and each rule below names the incident that motivated it. That provenance is the point: these are not heuristics invented for a README.

Install

npm install -g mcp-queue-doctor

{
  "mcpServers": {
    "queue-doctor": {
      "command": "mcp-queue-doctor",
      "env": {
        "QUEUE_DOCTOR_DATABASE_URL": "postgres://readonly:pw@localhost:5432/app"
      }
    }
  }
}

Then ask: "Is anything wrong with my job queue?"

Want to see it work first? examples/demo spins up a throwaway Postgres and manufactures seven failures in about a minute.

Connecting it

The server speaks stdio, so every MCP client starts it as a subprocess. The only thing that varies is where the config lives — and whether that process can reach your database.

Claude Desktop — ~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows. Use the mcpServers block above, then restart the app.

Desktop launches its subprocesses from the app bundle, not a login shell, so PATH is minimal and a bare mcp-queue-doctor or npx often fails to resolve. Give it an absolute path — which mcp-queue-doctor after a global install, or the absolute path to npx with ["-y", "mcp-queue-doctor"] as its args.

Claude Code — one command, no file editing:

claude mcp add queue-doctor -e QUEUE_DOCTOR_DATABASE_URL=postgres://... -- npx -y mcp-queue-doctor

Add -s project to write a checked-in .mcp.json at the repo root instead of your personal config, so everyone working in that repo gets the tool.

Cloud / remote sessions (Claude Code on the web, and any other headless runner) — a checked-in .mcp.json is the only mechanism that works, because nobody is there to answer an approval prompt. Reference the connection string rather than committing it; Claude Code expands ${VAR} and ${VAR:-default} in .mcp.json:

{
  "mcpServers": {
    "queue-doctor": {
      "command": "npx",
      "args": ["-y", "mcp-queue-doctor"],
      "env": { "QUEUE_DOCTOR_DATABASE_URL": "${QUEUE_DOCTOR_DATABASE_URL}" }
    }
  }
}

Project-scoped servers still need to be trusted before they start. In a headless session that means setting enableAllProjectMcpServers: true in the repo's .claude/settings.json, since the interactive approval never arrives.

Reachability is the real constraint, not the config. The server runs wherever the client runs, and it connects to Postgres directly — there is no hosted component in between. A cloud sandbox can therefore only diagnose a database inside that sandbox: the demo stack, or a dev stack the session brought up itself. A production queue bound to loopback on your own host is not reachable from a sandbox at all, and exposing it to make it reachable is the wrong trade.

Diagnose production from a client on a machine that already has a route to it — your laptop, over an SSH tunnel:

ssh -N -L 5432:127.0.0.1:5432 prod-host

and point QUEUE_DOCTOR_DATABASE_URL at 127.0.0.1:5432. The tunnel is the access grant, it lasts exactly as long as the terminal stays open, and the credentials never leave your machine.

Tools

Tool	Answers
diagnose	Start here. Runs the whole rule catalog, returns ranked findings with evidence and recovery steps
queue_overview	Per-queue counts by state, stuck jobs, and each queue's expiry/retention/retry config
failed_jobs	Failures in a window with error messages, plus a per-queue error-frequency breakdown
stuck_jobs	Jobs active past a threshold, with age, expiry, and heartbeat staleness
missed_schedules	Cron queues whose latest firing is older than their expression implies
schedule_status	Every registered schedule with cron, timezone, last firing, and next expected
job_detail	One job's full record: state, timings, retries, payload, output
server_info	Connectivity, detected schema, matched dialect, and reduced capabilities

Schedule expectations are derived from pg-boss's own schedule table by parsing each cron expression, so the common case needs no configuration. The health check this was extracted from carried a hand-maintained list of expected jobs that silently stopped covering whatever nobody remembered to add.

The rule catalog

Rule	Fires when	Motivating incident
retry-storm	Many failures, densely packed, dominated by one error	A daily API quota tipped over and 875 corpus-fill jobs failed in one night. The count suggested 875 problems; the shape showed one
expiry-overrun	Failure durations cluster at the queue's expiry	A full-corpus sweep couldn't finish inside a 30-minute expiry once upstream throttling slowed it. It reported as a job failure nightly; the fix was an internal wall-clock budget
stuck-jobs	Jobs active far too long, or heartbeats stopped	A worker killed without graceful shutdown leaves rows active until maintenance reclaims them
missed-schedule	Latest firing predates the last expected tick	Distinguishes "never fired" (scheduler never booted) from "stopped firing"
duplicate-registration	A cron queue enqueued twice for one tick	An instrumentation hook invoked job registration twice per process, so every cron ran double for weeks
retention-window	Failed-row count disagrees with the windowed count	A health email stayed yellow for days after the bug was fixed, counting rows that failed days earlier
dead-queue	Registered long ago, unscheduled, holds nothing	A producer that stopped, or a registration dropped in a refactor

Failures are classified (rate_limit, transient_transport, auth, not_found) because the class changes the advice: the right response to a storm of 429s is close to the opposite of the right response to connection resets.

Backends

Backend	Support	Verified against
pg-boss v11+	Full	11.1.2 (schema 26), 12.27.0 (schema 37)
pg-boss v10	Recognised, refused — see below	10.4.2 (schema 24)
pg-boss v9 and earlier	Recognised, refused	—
graphile-worker 0.17	Partial, capability-declared	0.17.3

Select with QUEUE_DOCTOR_BACKEND=pgboss (default) or graphile; the schema default follows the backend.

Capabilities, not zeros

Backends don't just name columns differently — they model work differently. graphile-worker deletes a job when it succeeds, has no per-job expiry, no worker heartbeats, and keeps cron expressions in a file rather than the database. So "how many completed in the last day" has no answer there at any price.

Every backend therefore declares what it can answer, and rules that depend on missing data stay silent rather than reporting a zero — a zero reads like a measurement.

Rule	pg-boss v11+	graphile-worker
retry-storm	✅	✅
stuck-jobs	✅ (with heartbeats)	✅ (age only)
expiry-overrun	✅	— no expiry exists
missed-schedule	✅	— cron lives in a file
duplicate-registration	✅	— no firing history
retention-window	✅	— nothing is retained
dead-queue	✅	— no queue registry

server_info reports the capability set and spells out each limitation.

Versioned against pg-boss

pg-boss's tables are not a stable API. Across versions it has renamed every timestamp column (createdon → created_on), dropped a whole table (archive, removed in v11), changed a duration from an interval to an integer (expire_in → expire_seconds), partitioned the job table, and added columns (heartbeat_on) that materially change what can be diagnosed.

A tool that hard-codes one shape breaks on the next upgrade — silently, if it is unlucky. That is exactly how the health check this is extracted from spent weeks emitting a confident, wrong "missed schedules" warning that was really SQLSTATE 42P01 after pgboss.archive disappeared.

So schema knowledge lives in one file, src/pgboss/dialect.ts, as data:

• Every relation and column name is declared in a dialect. Query builders emit identifiers from it, so supporting a new pg-boss layout is an edit to that file — no SQL elsewhere mentions a pg-boss table by name.
• Dialects are matched on observed shape, not on a version number. pg-boss's release→schema-version mapping is not published as a contract, and a guessed mapping would reintroduce the very failure this guards against. The version integer is read, reported, and used to say "this is newer than anything we have verified" — but it never decides which SQL runs.
• Optional columns are feature-detected. No heartbeat_on? Stuck-job detection degrades to age-based and says so, instead of failing.
• Unknown layouts are refused, by name. A pre-v10 schema is recognised specifically and rejected with the reason, because diagnosing it against modern queries would silently miss every archived job. A wrong diagnosis is worse than a refusal.

server_info reports the matched dialect, the schema version, whether that version has been verified against real pg-boss, and any reduced capabilities.

This is not a theoretical concern — it has already caught a real bug. The dialect originally claimed a v10 floor, on the belief that v10 removed the archive table. Booting pg-boss 10.4.2 showed the archive table still present and expiry still an expire_in interval, so the dialect was rejecting v10 outright and matching nothing at all for it. The real floor is v11, and v10 now has its own dialect: recognised, and refused by name, because reading the job table alone on v10 silently misses everything already archived.

CI keeps this honest. The integration suite boots pg-boss 10, 11 and 12 into separate schemas and asserts that the observed schema version appears in the dialect's verified list — so a future pg-boss that changes the schema fails loudly rather than running unverified SQL.

Read-only, by construction

Every query runs inside a BEGIN READ ONLY transaction with a statement_timeout and a row cap, and is always rolled back. Recovery actions are recommended, with exact commands — never executed. A confused agent cannot purge your queue, because the database itself refuses the write.

Three independent guarantees, because the failure being guarded against is writing to someone's production queue:

1. BEGIN READ ONLY on every transaction
2. default_transaction_read_only=on at connection level
3. The docs tell you to connect as a least-privilege role — the only guarantee that does not depend on this code being correct

Timeouts bind as parameters via set_config(..., is_local => true) rather than being interpolated into SQL. The schema name — the one identifier that cannot be a bind parameter — is validated against an identifier grammar and quoted.

Log correlation (optional)

Queue state says that a job failed; application logs usually say why. Point the server at a log backend and findings quote the lines behind a failure.

QUEUE_DOCTOR_AXIOM_TOKEN=xapt-...      # read-capable PAT
QUEUE_DOCTOR_AXIOM_DATASET=app-prod
QUEUE_DOCTOR_AXIOM_ORG_ID=your-org
QUEUE_DOCTOR_AXIOM_QUEUE_FIELD=job     # field carrying the queue name

Deliberately optional, and deliberately unable to break anything: a dead log backend never turns a working diagnosis into a failed one, and "we did not look" stays distinguishable from "we looked and found nothing" — otherwise an absent log line reads as evidence of absence. Half-configured settings are a startup error rather than a silent downgrade.

Reaching a database you cannot connect to

Production queues are often the ones you most want diagnosed and least able to reach: Postgres bound to loopback, no port forwarding, only the application in front of it exposed. Opening the database to the network so a diagnostic can connect is a poor trade — the grant is permanent and far wider than the need.

So the server can run its SQL over HTTPS against a read-only SQL endpoint instead:

QUEUE_DOCTOR_HTTP_SQL_URL=https://your-app.example/api/admin/sql
QUEUE_DOCTOR_HTTP_SQL_TOKEN=...

Set these and no connection string is needed; set both and the HTTP transport wins, so an ambient DATABASE_URL cannot quietly become the target. The endpoint must accept {"query": "...", "params": [...]} and answer with {"rows": [...], "truncated": bool}. Reference implementation: showbook's /api/admin/sql.

The safety properties move to the far end, which is an improvement rather than a compromise. The endpoint opens its own read-only transaction, enforces its own timeout and row cap, can rate-limit, can log every query, and can be backed by a role with narrower grants than the application's own — none of which depend on this client being correct. What changes for you: the endpoint's statement_timeout and row cap win over QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS and QUEUE_DOCTOR_MAX_ROWS, a truncating endpoint is reported as truncated rather than silently short, and one diagnose costs roughly a dozen requests against whatever rate limit is in force.

Bind parameters are required, not optional: a client forced to inline its own literals to reach a read-only endpoint would be building an injection sink to get there.

Configuration

Variable	Default	Purpose
QUEUE_DOCTOR_DATABASE_URL / DATABASE_URL	—	Required, unless the HTTP transport is used. Postgres connection string
QUEUE_DOCTOR_HTTP_SQL_URL	—	Read-only SQL endpoint to query through instead of connecting
QUEUE_DOCTOR_HTTP_SQL_TOKEN	—	Bearer token for that endpoint
QUEUE_DOCTOR_BACKEND	pgboss	pgboss or graphile
QUEUE_DOCTOR_SCHEMA	per backend	Schema the queue was installed into
QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS	5000	Per-query timeout (100–120000)
QUEUE_DOCTOR_MAX_ROWS	500	Row cap per query (1–10000)
QUEUE_DOCTOR_LOG_LEVEL	info	debug/info/warn/error/silent (stderr)
QUEUE_DOCTOR_THRESHOLDS	—	JSON object overriding rule thresholds (see below)

See .env.example. Requires Node.js ≥ 20.11.

Tuning the rules

The thresholds are tuned to the queue these rules were extracted from. That is a defensible starting point and a poor universal answer: a queue that legitimately fails fifty times an hour against a flaky upstream does not have a retry storm, and being told it does every time teaches you to stop reading.

Override any of them with a JSON object — only the keys you set change:

QUEUE_DOCTOR_THRESHOLDS='{"stormMinFailures":50,"idleQueueSeconds":2592000}'

Key	Default	Governs
stormMinFailures	20	Failures before a burst counts as a storm
stormDominantShare	0.5	Share one error must hold to be called dominant
stormPeakPerMinute	5	Failures in a minute that mark a burst, not a trickle
stormCriticalFailures	100	Above this a storm is critical, not a warning
expiryProximity	0.95	Fraction of expiry that looks killed rather than failed
expiryMinJobs	3	Jobs at expiry before it is a pattern
heartbeatMissedMultiplier	3	Missed heartbeats before a worker counts as gone
missedScheduleCriticalSeconds	86400	Lateness beyond which a miss is critical
retentionMismatchMin	5	Extra stale failed rows before flagging retention
duplicateTickMin	2	Ticks with duplicate firings before suspecting double registration
idleQueueSeconds	604800	Age at which an empty queue is worth mentioning
correlatedLogSample	5	Log lines attached to a finding as evidence

An unknown key is a startup error, not a warning — a typo that silently leaves the default in place is the failure this prevents. server_info reports the effective values and which ones you set, so you can confirm an override took.

What this is not

• Not a queue browser. To page through jobs, psql is better.
• Not a dashboard. This is agent infrastructure; your MCP client is the UI.
• Not a Redis queue tool. Both supported backends are Postgres-native, which is what makes the read-only transaction guarantee possible at all. BullMQ and Celery would need a different safety story.
• Not a writer. It will not retry, cancel, or purge anything.

Roadmap

• [x] Read-only database layer, schema probe, CI
• [x] The read-only tool surface
• [x] The diagnosis engine
• [x] A docker compose up demo with a chaos worker
• [x] Integration tests against real pg-boss 10/11/12 in CI
• [x] Log correlation, so findings can cite application logs
• [x] A second adapter (graphile-worker)
• [ ] Opt-in write tools (retry_job, cancel_job) behind an explicit flag
• [ ] Configurable rule thresholds

Development

npm install
npm run verify     # lint + typecheck + test + build

The unit suite drives the database layer through a scripted fake client and the rules through fixtures reconstructing each motivating incident, so npm test runs with no Postgres, no containers, and no network.

The integration suite boots real pg-boss (v10, v11, v12) and real graphile-worker against a live Postgres:

docker run -d -p 55432:5432 -e POSTGRES_USER=qd -e POSTGRES_PASSWORD=qd \
  -e POSTGRES_DB=qd postgres:16-alpine
QUEUE_DOCTOR_TEST_DATABASE_URL=postgres://qd:qd@127.0.0.1:55432/qd \
  npm run test:integration

It skips itself when that variable is unset, so a contributor without Postgres is never blocked. For a hands-on run, use examples/demo.

License

MIT