StudyPilot

StudyPilot is a course-material learning agent project. It uses an MCP server to expose course documents as resources and provide search/read tools.

Project references: architecture, evaluation protocol, demo script, and release checklist.

Current capabilities

• Import local Markdown, text, and text-based PDF documents.
• Preserve one-based PDF page numbers in search results and citation details.
• Store material chunks and their Qwen3-Embedding-0.6B vectors in a persistent FAISS IndexFlatIP index (exact cosine similarity after L2 normalization).
• Retrieve with hybrid search: dense vector candidates plus an in-process lexical rank, fused with reciprocal-rank fusion (RRF).
• Keep study tasks and quiz results in PostgreSQL; material retrieval no longer depends on PostgreSQL full-text search.
• Read a complete document by its stable document ID.
• Expose the same operations as MCP tools and resources.
• Create, list, and update persistent study tasks through MCP tools.
• Provide a source-grounded quiz prompt through MCP.
• Let DeepSeek discover and call MCP tools in a bounded Agent loop.
• Route requests between a Tutor Agent and Planner Agent with isolated MCP tool permissions.
• Select DeepSeek or a release-gated local Ollama model for Tutor requests while Planner remains on DeepSeek.
• Persist the assistant display name per browser conversation in PostgreSQL.

Multi-Agent routing

StudyPilot uses deterministic routing instead of spending another model call on a Supervisor Agent. Ordinary course questions go directly to the Tutor Agent. Task-only commands go directly to the Planner Agent. Composite requests such as "analyze my weak points from the materials and schedule revision" run Tutor first, then pass its cited diagnosis to Planner as a structured handoff.

• Tutor tools: search_materials, get_document
• Planner tools: create_study_task, list_study_tasks, update_study_task_status
• Single-purpose requests invoke one Agent; only composite requests invoke both.

Each Agent run is bounded by both model steps and total tool calls. Composite requests pass a JSON Tutor handoff containing the original request, grounded diagnosis, and extracted citation IDs to the Planner.

Within one Agent run, an identical mutating tool call is executed at most once. This protects task creation and status updates from repeated model tool calls; read-only tools may still be called again when needed. Web Agent task creation also receives a persistent idempotency key derived from the conversation, turn, and normalized business arguments. Concurrent or retried creation with that key returns the original task. Direct MCP/CLI creation without a key keeps normal create-each-time behavior.

The Web client stores a random conversation_id in localStorage. Explicit renames such as "以后你叫不亮" are stored in PostgreSQL and injected into both Agent prompts on later turns. Clearing the chat creates a new conversation and resets the display name.

Persistent conversation context

Every user and assistant message is stored in PostgreSQL with its conversation_id, turn_id, role, timestamp, and executing Agent. The browser restores the complete saved history through GET /api/conversations/{conversation_id} after a refresh. Full history remains in the database; only the latest 20 messages within a 12,000-character budget are sent to the model on each turn.

If a completed conversation_id and turn_id is submitted again with the same user message, both chat endpoints replay the persisted assistant response without running the Agent again. Reusing a turn ID for different content returns HTTP 409. This covers completed-request retries; it is not a lock for two concurrent requests that arrive before the first response is persisted. Persistent task idempotency still prevents identical task creation within that concurrent turn.

StudyPilot records user preferences only from explicit statements, including preferred name, language, response detail, and step-by-step teaching style. Its learning profile separates explicit goals and weak points from observed activity such as retrieval queries, source documents, and study tasks. Observed activity is derived from real MCP traces, not model guesses. Clearing the chat starts a new conversation but does not delete the old history from PostgreSQL.

Setup

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"

Run the MCP server

Start PostgreSQL for study-task persistence, migrate the schema, and index sample materials into FAISS. The first indexing run downloads Qwen/Qwen3-Embedding-0.6B from Hugging Face.

docker compose up -d postgres
.venv\Scripts\alembic.exe upgrade head
.venv\Scripts\studypilot-index.exe

FAISS persists the index and chunk metadata under data/faiss/. Existing Chroma cache files are not read after this migration, so run studypilot-index once after upgrading to rebuild the index from data/materials/. Set FAISS_DIR only when the index must live elsewhere, such as an isolated test or deployment data volume; the default remains data/faiss/.

Document IDs are derived from normalized file stems and must be unique. Files such as Course Notes.md and course-notes.txt conflict; indexing rejects the set and reports both names instead of silently merging their chunks or producing ambiguous citations.

Blank-line paragraph boundaries remain stable for ordinary material. A single paragraph longer than 1,200 characters is split into bounded chunks, preferring sentence endings in the final 40 percent of each window. This rule does not add overlap because complete documents are reconstructed from stored chunks. Re-run studypilot-index after upgrading from an index built before this rule.

The default transport is stdio, which is convenient for a local Agent host:

studypilot-mcp

To inspect it interactively:

mcp dev app/studypilot/mcp_server.py

Test

.venv\Scripts\python.exe -m pytest

The default suite uses local fakes for model calls and an isolated repository temporary directory. PostgreSQL integration tests are skipped when the test database is unavailable; a green local run therefore does not by itself prove that PostgreSQL or a real model endpoint is reachable.

Fixed retrieval evaluation

The fixed set is data/eval/retrieval_cases.json. It uses source chunk IDs as relevance judgments and writes a reproducible JSON report with Recall@1, Recall@3, Recall@5, MRR, and per-case bad cases.

.venv\Scripts\studypilot-eval.exe --k 1 3 5

To score answers and citation faithfulness as well, provide real Agent output in this format (one record per fixed case):

[
  {
    "id": "round-robin",
    "answer": "时间片过小会造成频繁进程切换，增加系统开销。",
    "citations": ["operating-systems:2"],
    "refused": false,
    "latency_ms": 840.5,
    "total_tokens": 320,
    "error": null
  }
]

.venv\Scripts\studypilot-eval.exe --answers .\agent_answers.json

data/eval/latest_report.json records every retrieval list and identifies failures in bad_cases. Cases marked should_refuse evaluate refusal separately from retrieval. When answer records include runtime fields, the report also calculates P50/P95 latency, mean token count, and failure rate. It never fabricates metrics when real Agent data is not supplied.

T2Ranking fixed rerank set

The project uses the official THUIR/T2Ranking dev split as its external Chinese rerank benchmark. The checked-in fixed subset contains 200 query IDs selected from dev using a documented SHA-256 seed, retrieval qrels, graded rerank qrels, and hashes of all three official source files.

Build or verify the subset after downloading the official small metadata files into data/eval/t2ranking/source/:

.venv\Scripts\python.exe .\tools\build_t2ranking_subset.py

The fixed query/qrels set alone is not enough to report rerank quality. A valid Recall@3 comparison also needs a persisted Top-K candidate pool and the corresponding passages from the same collection.tsv revision. The baseline and reranker must reorder the same candidates. Until those artifacts and both run outputs exist, this project does not claim any Recall@3 improvement.

The official dev.bm25.tsv candidate list produces a persisted 100-candidate pool per fixed query. Its measured baseline is Recall@3 0.5250, MRR@10 0.4268, and nDCG@10 0.3050. A completed local BAAI/bge-reranker-v2-m3 run reranked the first 20 of the same 100 candidates for all 200 queries in one 4,000-pair batched call. The measured rerank result is Recall@3 0.6000, MRR@10 0.4812, and nDCG@10 0.3495, for deltas of +0.0750, +0.0544, and +0.0445 respectively. The checked-in report and manifest identify the exact artifacts. To reproduce it, run:

.venv\Scripts\python.exe .\tools\extract_t2ranking_passages.py
.venv\Scripts\python.exe .\tools\rerank_t2ranking.py --model .\data\models\bge-reranker-v2-m3 --top-n 20 --batch-size 16 --progress
.venv\Scripts\python.exe .\tools\evaluate_t2ranking.py --rerank .\data\eval\t2ranking\fixed-200\rerank.cross_encoder.tsv --report .\data\eval\t2ranking\fixed-200\rerank.report.json

The evaluator rejects a rerank file if it changes any query's candidate set. It therefore reports a genuine reranking comparison, not a candidate-generation comparison.

Ask the Agent

Configure DEEPSEEK_API_KEY and DEEPSEEK_MODEL in .env, then run:

.venv\Scripts\studypilot-chat.exe "时间片为什么不能设置得太小？"

Model reliability and observability

DeepSeekChatModel enforces a per-attempt timeout, retries transient timeout, connection, rate-limit, and server errors with exponential backoff, then tries the explicitly configured fallback models. It does not silently switch to an invented provider or model.

For streaming responses, retry and fallback are allowed only before the first chunk is emitted. Every wait for the next chunk uses the same per-attempt timeout. If a stream fails after partial content has reached the client, the request ends with an error instead of replaying duplicated text.

Configure the primary and fallback sequence in .env:

DEEPSEEK_MODEL=deepseek-v4-flash
DEEPSEEK_FALLBACK_MODELS=your-confirmed-fallback-model
DEEPSEEK_TIMEOUT_SECONDS=30
DEEPSEEK_MAX_RETRIES=2
DEEPSEEK_RETRY_BASE_SECONDS=0.5

The system prompt carries the code-defined version studypilot-2026-07-30.2. Every model attempt is appended to data/logs/model_calls.jsonl with its provider, model, model artifact hash (when configured), prompt version, retry/fallback position, latency, token usage, outcome, and error type. Web calls also include request_id, conversation_id, and turn_id so an HTTP response can be correlated with its model attempts. These identifiers are propagated through both JSON and NDJSON responses. Prompt text, user questions, API keys, and material content are not written to this log. To calculate cost, provide verified per-million-token rates for the exact model names; without them estimated_cost_usd remains null.

MODEL_PRICING_USD_PER_MILLION_JSON={"deepseek-v4-flash":{"input":0.0,"output":0.0}}

Run the web workspace

.venv\Scripts\studypilot-web.exe

Open http://127.0.0.1:8787 to chat, import materials, inspect sources, and manage study tasks.

For a containerized application process, keep Ollama on the host and run:

docker compose build app
docker compose run --rm app alembic upgrade head
docker compose run --rm app studypilot-index
docker compose up -d app

The app container reaches Ollama through host.docker.internal. PostgreSQL, FAISS data, Hugging Face cache, and application health have separate Compose volumes/checks. The native and containerized Web processes both use port 8787, so run only one of them at a time.

GET /api/health reports the application, PostgreSQL connection, and FAISS material-index state separately. An index can be ready, empty, inconsistent, or unavailable; empty remains usable because documents can still be imported, while inconsistent or unavailable indexes make the top-level health status false. The health payload also reports the current index-format version. Metadata from an older or malformed format is rejected with an explicit studypilot-index rebuild instruction.

Hybrid retrieval and streaming chat

search_materials keeps its existing MCP tool name and result shape. Internally it fetches vector candidates from FAISS, ranks all chunks for lexical term matches, then uses RRF to combine the two lists. This improves exact terminology matches without removing semantic retrieval.

The workspace now uses POST /api/chat/stream. It returns application/x-ndjson with status, delta, done, and error events. The UI renders each delta as it arrives and reports when the bounded Agent is calling an MCP tool. POST /api/chat remains available for existing non-streaming clients.

Optional Qwen3.5-4B tutor fine-tuning (deferred)

This release intentionally stops before adapter training, GGUF conversion, and four-model quality comparison. The checked-in corpus, isolated indexes, trajectory compiler, dataset validator, and CPU preflight are preparation artifacts only; no local-model quality improvement is claimed. DeepSeek remains the default Tutor and the local Ollama option stays release-gated.

The current app-scope release audit is:

.venv\Scripts\python.exe .\tools\audit_release.py --scope app

The local tutor fine-tuning path is pinned to the official Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, which matches the qwen35 4.7B architecture used by the local Ollama qwen3.5:4b model. Ollama's quantized GGUF file is for inference; training uses the original Safetensors checkpoint.

Install the optional training dependencies:

.venv\Scripts\python.exe -m pip install -e ".[train,dev]"

Formal data uses complete system/user/assistant.tool_calls/tool/assistant trajectories and the frozen data/finetune/tutor_tools.v1.json schema. Only assistant spans contribute to loss. System prompts, user questions, and tool responses are masked. The checked-in eight-row file remains a response-only smoke sample and cannot pass the formal 300-train/60-evaluation gate.

.venv\Scripts\python.exe .\tools\fetch_public_corpus.py --inspect-rights
.venv\Scripts\python.exe .\tools\fetch_public_corpus.py --accepted-policy .\data\corpus\accepted-rights.v1.json
.venv\Scripts\python.exe .\tools\build_corpus_indexes.py
.venv\Scripts\python.exe .\tools\create_tutor_review_worklist.py --split train --output .\data\finetune\tutor_review.train.jsonl
.venv\Scripts\python.exe .\tools\create_tutor_review_worklist.py --split eval --output .\data\finetune\tutor_review.eval.jsonl
.venv\Scripts\python.exe .\tools\compile_tutor_trajectories.py --worklist .\data\finetune\tutor_review.train.jsonl --index-dir .\data\indexes\wikimedia_cs_v1\train --output .\data\finetune\tutor_sft.train.jsonl
.venv\Scripts\python.exe .\tools\compile_tutor_trajectories.py --worklist .\data\finetune\tutor_review.eval.jsonl --index-dir .\data\indexes\wikimedia_cs_v1\eval --output .\data\finetune\tutor_sft.eval.jsonl
.venv\Scripts\python.exe .\tools\validate_tutor_dataset.py --train .\data\finetune\tutor_sft.train.jsonl --eval .\data\finetune\tutor_sft.eval.jsonl --report .\data\finetune\tutor_dataset.validation.json
# Formal QLoRA training is deferred in this release.

The 8 GB profile uses a 768-token maximum, NF4 double quantization, rank-8 LoRA, batch size 1, gradient accumulation, BF16 compute, and gradient checkpointing. LoRA targets are discovered only under model.language_model.layers; the visual tower stays frozen. The current validation report records exactly 300/60 rows, semantic cross-split validation, and review_mode=ai_assisted_authorized. Formal training remains deferred; no adapter or loss result is part of this release.

After a future training run, merge on CPU, convert to GGUF Q4_K_M, and run the same holdout through base, adapter, DeepSeek, and deployed Ollama candidates only when the recorded release gates pass:

.venv\Scripts\python.exe .\tools\merge_tutor_lora.py
.venv\Scripts\python.exe .\tools\convert_tutor_to_gguf.py
.venv\Scripts\python.exe .\tools\run_tutor_evaluation.py --cases .\data\finetune\tutor_sft.eval.jsonl --output .\data\eval\tutor\runs.jsonl --candidate qwen3.5-4b-base=transformers --candidate studypilot-tutor-4b-adapter=peft:data/models/studypilot-tutor-4b-lora --candidate deepseek=deepseek --candidate studypilot-tutor-4b-q4_k_m=ollama:studypilot-tutor:4b
.venv\Scripts\python.exe .\tools\score_tutor_evaluation.py --cases .\data\finetune\tutor_sft.eval.jsonl --runs .\data\eval\tutor\runs.jsonl --output .\data\eval\tutor\report.json
.venv\Scripts\python.exe .\tools\create_ollama_tutor.py --gguf .\data\models\studypilot-tutor-4b-q4_k_m.gguf --evaluation-report .\data\eval\tutor\report.json --name studypilot-tutor:4b
.venv\Scripts\python.exe .\tools\audit_release.py --artifact .\data\models\studypilot-tutor-4b-q4_k_m.gguf

The Web API accepts tutor_provider=deepseek|ollama, and the page stores the selection per browser conversation. Planner always uses DeepSeek. The local choice is disabled until the verified artifact hash, matching deployment manifest, and OLLAMA_RELEASE_GATES_PASSED=1 are configured; ENABLE_EXPERIMENTAL_LOCAL_TUTOR=1 is an explicit non-release bypass. See fine-tuning and deployment for data contracts, scoring, failure evidence, and exact promotion rules.