AgentVision

A debug flight recorder for AI coding agents. It watches one program you are debugging — screenshots its window on a timer, parses its logs with 658 format adapters, and time-aligns the two — then exposes the result to an AI agent as 90 MCP tools.

Everything runs locally on your own CPU. Capturing, hashing, diffing and parsing are free; the only expensive resource is the agent's context, so AgentVision spends its own compute to keep that small.

your program ──┬── window screenshots ──┐
               │                        ├── one time-aligned timeline ── 90 av_* MCP tools ── your AI agent
               └── logs (658 adapters) ─┘

---

Why this exists

An AI agent debugging a running program is working blind. It can read your source and it can run commands, but it cannot see the window, and it only learns what a log says if it thinks to go and read the right file at the right moment.

AgentVision gives it both halves at once, on one timeline: what the program printed, and what the screen looked like when it printed it.

The hard part is not collecting that. It is handing it over without corrupting it. Long contexts measurably degrade — every frontier model loses accuracy as input grows, well before its window fills, and a model that takes a wrong turn in a multi-turn conversation tends not to recover. A debug tool that invents one detail is worse than no tool, because the agent commits to it. So the governing rule here is:

AgentVision may not assert anything it did not verify.

That rule is not aspirational. It is enforced in code and in 54 test suites, most of which exist because a specific version of this tool once said "healthy" while the program was failing.

What that looks like in practice

• A summary never replaces evidence. Collapsing 21,982 identical log lines to line [x21982] is allowed — nothing is lost. Dropping, re-levelling or ranking a distinct line is not. Raw output is always available and is never withheld.
• Silence is distinguished from absence. "OCR read the screen and found no error" and "OCR could not read the screen" are different facts, tracked separately, because conflating them once made a frame showing an error dialog the first thing deleted.
• Machine-read values are corroborated, not quoted. OCR misreads 0x5D80000 as 0x5080000 and reports full confidence while doing it, so screen-read hex, ids and counts are checked against the time-aligned log and marked appears_in_log: true|false — never verified.
• Retractions are as loud as claims. If AgentVision told your agent the program died and it is now running, it says so, quotes its own earlier words, and states a counted observation — and stays silent when it merely stopped looking.
• Truncation is always reported. A scan that did not finish says so, because "signal absent" and "file never opened" must not look the same.

The part that surprises people

AgentVision does not decide what to install into your program. Your agent does, once. On first connection it refuses to guess and returns a catalog: every emitter, adapter and tool it could use, alongside real evidence scanned from your code (71 signals — does it read keypresses, own a render loop, swallow exceptions, drive a GPU from Python, run in a browser). The agent commits a plan naming what it chose and why. AgentVision builds exactly that.

It refuses lazy plans: a stale catalog token, a selection with no reason, or "install everything" is rejected. The gate fires once per program, ever.

---

Install

Requires Python 3.11+.

pip install -r requirements.txt

Then follow the guide for your platform — each covers registering the MCP server with Claude Code and granting the OS screen-capture permission, both of which fail silently if skipped:

• macOS — SETUP.md
• Windows — SETUP-Windows.md
• Linux — dist/linux/ (X11 and Wayland notes, init templates)

Optional: pip install -e . gives you the agentvision command (agentvision doctor, agentvision run -- <cmd>). Without it, use python3 -m python_backend.cli <subcommand>.

Ships with no saved programs — you add your own.

If you are an AI agent reading this

Call av_start_here(). It reports the target program, whether the bridge is built, and your exact next call. Then read docs/AI_START_HERE.md and docs/BRIDGE_PROTOCOL.md — the first-connection contract is written for you, including what to do when the active profile is not the program you were asked about.

Documentation

What it is, in depth	the whole design, code-grounded
AI start here	cold-start guide for an agent
Bridge protocol	the first-connection contract
Logs & emitters	what to build into a program, and why
Adapters guide	the 658 log formats, and adding your own
MCP tools reference	all 90 tools (generated from code)
Push mode	how state reaches an agent without being asked
Token efficiency research	the measurements behind the design
Troubleshooting	when it is not working

Tests

python3 run_all_tests.py

54 suites. They are worth reading as documentation of real failures: each one tends to encode a specific way this tool once misled its caller.

One of them, bridge_gate, tests the first-connection contract and so needs a live bridge; it SKIPs (loudly, saying how to run it) if none is listening. To include it:

python3 python_backend/api/bridge_server.py --no-autocapture &
python3 run_all_tests.py

Platform support

	capture	input recording	notes
macOS	window backing store — survives occlusion and minimisation	✅	best supported; the platform it is developed on
Windows	mss region grab of the window rect	✅	occlusion-sensitive; blind to minimised windows
Linux/X11	region grab	✅ (evdev)
Linux/Wayland	portal-based	✅ (evdev)	no per-window enumeration — a compositor restriction

The Windows and Linux ports are built and unit-tested but have not been re-run end to end since a recent large refactor. macOS is the verified path.

If you are on Windows and want to help close that gap, start with docs/WINDOWS_PORT_COMPLETION.md — it lists what is done, the one highest-risk path to test first, and the traps that already cost time so you do not rediscover them.

Licence

MIT — see LICENSE.