Experiment Tracker MCP

An MCP (Model Context Protocol) server that lets you query your machine-learning experiments in natural language. Train models, log every run to a local store, then ask an AI assistant — Claude Desktop, VS Code Copilot, or any MCP client — questions like "which run had the best recall?" and have it call your tools to answer.

The project is a small, self-contained demonstration of three things: a clean experiment-tracking data model, a backend-agnostic storage layer, and an intent-oriented MCP server on top of it.

---

What it does

1. train.py trains scikit-learn models on a dataset and logs each run — hyperparameters, metrics, and the saved model file — through a tracker.
2. The store persists runs in SQLite behind a swappable interface.
3. server.py exposes the store as MCP tools, so an LLM can list, inspect, rank, and compare your experiments on request.

train models ──▶ log to store (SQLite) ──▶ MCP server ──▶ "which run is best?" (any MCP client)

---

Architecture

The codebase is layered by responsibility, and every dependency points inward toward an interface rather than toward a concrete implementation:

server.py ─┐
           ├─▶ store/base.py  ◀── store/sqlite_store.py ──▶ db/connection.py ──▶ schema.sql
train.py ──┘   (Tracker: the        (SQLiteTracker:          (connection +        (the DDL)
                interface)           an implementation)       initialization)

server.py and train.py — the application edges — depend only on the abstract Tracker interface, never on SQLite. SQLite is an adapter plugged in behind the interface. This is the Dependency Inversion Principle in practice: swapping the backend (e.g. for MLflow) means adding one class in store/, and nothing above the interface changes.

---

Project structure

tracker-mcp/
├── pyproject.toml          # dependencies + package metadata
├── .gitignore              # keeps tracker.db, models, __pycache__ out of git
├── README.md
├── .vscode/
│   └── mcp.json            # VS Code MCP server config
├── tracker_mcp/            # the package
│   ├── __init__.py         # re-exports Tracker, SQLiteTracker
│   ├── config.py           # single source of truth for filesystem paths
│   ├── db/                 # ── infrastructure layer ──
│   │   ├── schema.sql      #    the database structure, as real SQL
│   │   └── connection.py   #    connection config + schema initialization
│   ├── store/              # ── repository layer ──
│   │   ├── base.py         #    the Tracker interface (abstract)
│   │   └── sqlite_store.py #    the SQLite implementation
│   ├── train.py            # ── application: trains models, logs runs ──
│   └── server.py           # ── application: the FastMCP server ──
├── models/                 # saved .joblib model artifacts (gitignored)
└── tracker.db              # the SQLite store (gitignored)

---

Setup

Requires Python 3.12+.

# from the project root
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install fastmcp scikit-learn joblib
pip install -e .                   # editable install: makes `python -m tracker_mcp.*` work anywhere

The editable install is what lets the MCP server be launched by name (-m tracker_mcp.server) from any working directory — important because MCP clients spawn the server from an unpredictable location.

---

Usage

1. Train some models

python -m tracker_mcp.train

This trains four models on the scikit-learn breast-cancer dataset (a shallow decision tree as a weak baseline, a default and a tuned random forest, and gradient boosting), logs each run to tracker.db, and saves each model to models/.

2. Run the MCP server

python -m tracker_mcp.server      # stdio transport — what MCP clients expect

Or debug it interactively with the MCP Inspector:

fastmcp dev tracker_mcp/server.py

3. Connect it to a client

Claude Desktop — add to claude_desktop_config.json:

{
  "mcpServers": {
    "experiment-tracker": {
      "command": "/absolute/path/to/tracker-mcp/.venv/bin/python",
      "args": ["-m", "tracker_mcp.server"]
    }
  }
}

VS Code (GitHub Copilot, agent mode) — .vscode/mcp.json. Note VS Code uses the servers key, not mcpServers:

{
  "servers": {
    "experiment-tracker": {
      "type": "stdio",
      "command": "/absolute/path/to/tracker-mcp/.venv/bin/python",
      "args": ["-m", "tracker_mcp.server"]
    }
  }
}

Then ask, in natural language:

• "List my experiment runs."
• "I'm building a cancer screening model — which run should I use?"
• "Compare the tuned random forest and gradient boosting."
• "Show me the details of run 910ced952bb6."

---

The tools

The server exposes tools designed around what a user asks, not a 1:1 mirror of the store's methods:

Tool	Purpose
list_experiments(limit)	Recent runs, newest first (summary).
get_experiment(run_id)	Full detail of one run, including all metrics.
find_best_model(metric, mode)	The run with the best value for a metric (max/min).
compare_experiments(run_ids)	Several runs side by side, by params and metrics.

It also exposes each run as a read-only resource at experiment://{run_id}, and ships a server-level instructions block — domain context (what a run is, which metric fits which goal) delivered to the model during the MCP handshake, so it reasons about experiments correctly without the user re-explaining each turn.

---

Design decisions

These are the choices worth understanding, and the reasoning behind them.

Two tables, split by access pattern. Runs get one table; metrics get their own. Hyperparameters are heterogeneous across model types and rarely queried individually, so they're stored as a single JSON column. Metrics are the opposite — you constantly rank and compare across them, and they can form time series over training steps — so they live in a normalized table where MAX(value) WHERE key = 'accuracy' is trivial. This params-as-JSON, metrics-as-rows asymmetry is the core of the data model.

Backend-agnostic storage. The server and training code depend on the abstract Tracker interface, so the SQLite backend is a detail, not an assumption. A different backend (MLflow, Postgres) would be a new class behind the same interface.

Intent-oriented tools. MCP tools are shaped around user questions (find_best_model, compare_experiments) built on top of the store's primitives, rather than mechanically exposing get_run/list_runs. Clear, distinct tool descriptions are what let the model pick the right tool.

Single source of truth for paths. config.py anchors the database and model directory to the project root via __file__, so the training script (writer) and the server (reader) always resolve to the same files regardless of the working directory a command is launched from. Env vars (TRACKER_DB, TRACKER_MODELS) override when needed.

Reproducibility as a logged value. The random seed lives inside each run's logged params, so a stored run can be recreated exactly — the same dict both builds the model and gets recorded.

Failures are recorded. A run is created before training starts and marked failed if training raises, so the experiment history reflects reality rather than only successes.

---

Tech stack

• FastMCP (3.x) — the MCP server framework
• scikit-learn — model training
• SQLite (stdlib sqlite3) — the tracking store
• joblib — model serialization

---

Possible extensions

• An MLflowTracker implementation behind the same Tracker interface.
• A test suite under tests/ (the in-memory fastmcp.Client makes end-to-end tool tests straightforward).
• Per-step metric logging to support training curves, and a best_run that ranks by each run's final step.
• A delete_run / archival tool, and filtering list_experiments by model type or dataset.