qa-mcp

Automated QA for a .NET microservice backend (several independent services, each with its own Swagger/OpenAPI document) and a React frontend, running against stage, exposed to an MCP client over stdio.

The one architectural rule

MCP is a thin layer on top of the test engine. The engine does not depend on the model.

Tests are never generated by an LLM at call time. They are either derived deterministically from OpenAPI, or written by hand as YAML flows / Playwright specs. That is what makes two consecutive runs comparable — and comparability is the entire value of the tool. If every run produced a different set of assertions, qa_diff could not tell a regression from noise.

Consequence: everything under src/api, src/flows, src/ui and src/report runs perfectly well from CI without an MCP client. src/index.ts only registers tools and shrinks the output.

Install

npm install
npm run build

Requires Node ≥ 20. npm install also pulls Playwright; download the browser once with:

npx playwright install chromium

Configure

qa.config.json is the single source of truth. Its path comes from QA_MCP_CONFIG, defaulting to ./qa.config.json. Start from the example:

cp qa.config.example.json qa.config.json

Section	What it does
environment	Free-form label stored in every run report.
services[]	{ name, baseUrl, openapi, tags[], headers{} }. openapi is either an absolute URL or a path relative to baseUrl.
frontend	baseUrl, loginPath, loginSelectors{username,password,submit,successUrlPattern}, crawl{maxPages,ignore[],viewport}.
auth	tokenUrl, grantType, clientId, clientSecret, tokenField, and one entry per role under roles.
policy	readOnly, allowedMutationPaths[], excludePaths[], maxResponseMs, concurrency, requestTimeoutMs.
samples	Real values used to fill required parameters, keyed by parameter name.
artifactsDir	Where runs, HTML reports and screenshots are written. Default .qa-runs.

Secrets never live in the config

Any string starting with env: is read from the environment; if the variable is unset, startup fails with an explicit message naming the field and the variable.

"password": "env:QA_ADMIN_PASSWORD"

export QA_ADMIN_PASSWORD=...
export QA_CUSTOMER_PASSWORD=...
export QA_CLIENT_SECRET=...

Register the server in an MCP client

command + args, transport stdio:

{
  "mcpServers": {
    "qa-mcp": {
      "command": "node",
      "args": ["G:/mcp/qatester/dist/index.js"],
      "env": {
        "QA_MCP_CONFIG": "G:/mcp/qatester/qa.config.json",
        "QA_ADMIN_PASSWORD": "...",
        "QA_CUSTOMER_PASSWORD": "...",
        "QA_CLIENT_SECRET": "..."
      }
    }
  }
}

Claude Code CLI equivalent:

claude mcp add qa-mcp -- node G:/mcp/qatester/dist/index.js

Verify the handshake at any time without touching stage:

npm run smoke

Tools

Tool	Input	What it does
qa_discover	services?	Fetches every Swagger document, reports endpoints per service, planned assertions, operations skipped by policy with the reason, and services that could not be reached. Sends no request to the endpoints.
qa_run_api	services?, role?, pathContains?	Generates and runs the API matrix. Mutating endpoints are skipped while policy.readOnly is on.
qa_run_flows	names?, flowsDir?	Runs tests/flows/*.yaml sequentially.
qa_run_ui	mode: specs|crawl|both, role?, maxPages?, startPaths?, grep?	Playwright specs and/or the automatic crawl.
qa_run_all	role?, services?, skipUi?, maxPages?	Full pass, then an automatic diff against the previous stored run. Release gate.
qa_report	runId?, status?, severity?, suite?, service?, limit	Reads a stored run with filters. Use it when a digest says results were truncated.
qa_diff	baselineRunId?, currentRunId?	Compares two runs by stable test id.

Every run tool returns a digest, not the raw results: the summary, failures grouped by service, at most 40 failures sorted critical-first, how many were truncated, and the runId to drill into with qa_report. A full pass over several services produces thousands of assertions; returning all of them would bury the signal.

What gets tested

Generated API tests

Per operation, from the OpenAPI document:

Check	Assertion	Severity
contract	Valid request → status must be declared in the spec and the body must validate against that status' schema	major
authz	Secured operation called with no token → anything other than 401/403 is a leak	critical
robustness	Malformed value in a required typed parameter → expect 400/404/422	major
perf	Response time over policy.maxResponseMs	minor

Any 5xx is an unconditional critical failure, including when the input was deliberately broken. A well-behaved service answers 400, not 500. On a .NET backend this single rule is the highest-yield automated check there is — it drags unhandled exceptions straight out of the framework.

Parameter values come from config.samples (by parameter name) first, then enum[0], default, example, then a format/type heuristic (uuid → zero-uuid, date → today, integer → 1, …). Malformed values match the type: uuid → not-a-uuid, numeric → not-a-number, date → 31-31-9999.

Each test carries a stable id — service:METHOD:/path#check — with no timestamp and no random value in it. qa_diff depends entirely on that.

One valid request produces both the contract and perf results, so an operation costs at most three HTTP calls (valid, unauthenticated, malformed).

Safety model — read this before pointing it at anything

Running generated tests against stage can destroy data or fire notifications at real users.

• With policy.readOnly: true (the default) only GET/HEAD/OPTIONS are executed. Every mutating verb is refused unless its path starts with one of policy.allowedMutationPaths.
• policy.excludePaths is honoured always, including when readOnly is off.
• Every refused operation is reported with its reason by qa_discover, so you always know what is not covered.
• Destructive operations are only allowed inside YAML flows, where a human wrote the steps.

Cross-service flows

tests/flows/*.yaml. Real microservice bugs live here, not in single-service tests: each service passes its own tests while the order id never reaches the notification consumer.

name: order-lifecycle
description: why this scenario matters
role: customer
severity: critical
steps:
  - name: create order
    service: orders          # must match a service name in qa.config.json
    method: POST
    path: /api/orders
    role: customer           # optional, overrides the flow role
    body: { productId: "{{productId}}" }
    expectStatus: [200, 201]
    expectBody: { status: Pending }   # dot-path -> expected value
    capture: { orderId: id }          # variable name -> dot-path in the response
    waitMs: 0                         # pause before the step, for eventual consistency

{{var}} interpolation works in path, body, headers and expectBody. Dot-paths understand array indexes (items[0].status). Flows run sequentially — they mutate shared state. One flow produces exactly one result: the business scenario either completes or it does not. On failure the evidence holds the failing step name, the trace of the steps that did succeed, and every captured variable. waitMs exists for the case where the target service only learns about the change through a message broker.

Frontend — two layers

Fixed specs (tests/ui/, standalone playwright.config.ts) are run as a child process with --reporter=json and normalised into results. QA_BASE_URL, QA_LOGIN_PATH and QA_LOGIN (credentials for the requested role) are passed in as environment variables, so the project also runs directly in CI. If playwright.config.ts is missing you get a skip with a clear message, not a crash. Put @critical or @minor in a test title to set its severity.

The three shipped specs assert business outcomes rather than "did the page load": a successful login, a list that must contain rows (an empty table where data is expected is the classic silent data-layer failure), and one that fakes a 500 with page.route to prove the UI shows the user an error instead of hanging on a spinner.

Automatic crawl covers the pages nobody wrote a spec for. It optionally logs in with a role; if that login fails it returns one critical result and stops immediately, because every later result would be meaningless without a session. It then BFS-crawls same-origin links up to maxPages, skipping crawl.ignore prefixes. A route fails on navigation error, HTTP ≥ 400, console errors, an XHR ≥ 400, or an effectively empty render. Console noise (React DevTools, HMR, React Router future flags, unused-preload warnings, favicon 404s) is filtered out — without that the output is unusable. Every failure gets a full-page screenshot whose path lands in the evidence.

Runs, reports and diff

Each run is written to artifactsDir as run-<timestamp>.json plus a readable .html next to it, with a severity-coloured failure table. Colons and dots in the timestamp are replaced with - (illegal in Windows filenames); the ids stay lexicographically sortable.

qa_diff compares two runs by stable id and splits the delta:

Bucket	Meaning
regressions	Passed before, fails now. The only bucket that should block a release.
fixed	Failed before, passes now.
newFailures	Test did not exist in the baseline and fails now (new endpoint, new page).
stillFailing	Already broken before.
removed	Was in the baseline, gone now (endpoint or page deleted).

When no baselineRunId is given, the baseline is the most recent earlier run covering the same suites — comparing an API-only run against a UI-only run would mark every test as removed and mean nothing.

Expect the first run to be noisy

This is normal and it is not a bug. The first pass against a real stage environment will report plenty of failures that are configuration, not defects: endpoints needing an id you did not provide, admin-only paths, export jobs, health probes.

Work through it in this order:

1. Run qa_discover and read what was skipped by policy — that is your coverage gap.
2. Run qa_run_api, then push obvious non-defects into policy.excludePaths.
3. Fill samples with real stage ids (orderId, customerId, productId, …) so path and required-query parameters resolve to rows that actually exist. Most 404 noise disappears here.
4. Raise maxResponseMs if stage is simply slower than production.
5. Re-run until what remains is genuinely interesting.
6. Keep that run as the baseline. From then on qa_diff / qa_run_all answer the only question that matters before a release: did anything that used to work stop working?

Layout

src/
  config.ts               zod config schema + env: resolution
  types.ts                TestResult / RunReport / summarize()
  discovery/openapi.ts    fetch + normalise specs, resolve $ref
  auth/session.ts         per-role token with cache
  api/generator.ts        test matrix per operation + safety gate
  api/runner.ts           bounded-concurrency execution + schema validation
  flows/runner.ts         cross-service YAML scenarios
  ui/crawler.ts           automatic Playwright crawl
  ui/specs.ts             runs Playwright specs, normalises the JSON report
  report/store.ts         store, diff, render HTML
  index.ts                MCP tool registration
tests/
  flows/*.yaml
  ui/                     playwright.config.ts + *.spec.ts

Notes for whoever extends this

• stdio transport: never write to stdout. console.log corrupts JSON-RPC and the client disconnects with an opaque error. Every log goes to process.stderr (see log() in src/index.ts). npm run smoke fails if anything non-protocol reaches stdout.
• ajv/ajv-formats are CommonJS. Under NodeNext the constructable sits on .default, and import type Ajv from "ajv" is unusable as a type — src/api/runner.ts shows the working pattern.
• Relative ESM imports need the .js extension even from .ts sources.
• Every tool handler is wrapped in try/catch and returns isError: true rather than throwing.
• Every fetch carries AbortSignal.timeout(policy.requestTimeoutMs).
• Uncompilable response schemas are treated as "no schema", not as an endpoint failure — the defect is in the spec, not in the service.
• $ref resolution has both a depth cap and a per-branch seen-set; self-referencing DTOs are common in .NET and would otherwise hang the process.