repo-rag

A chat agent over my own GitHub profile — ask it "has Aman used Kotlin?" or "is CareerSpire finished?" and it answers in synthesized prose with citations, backed by a tool-calling loop that decides for itself what to look up.

The interesting decision

Most personal-profile RAG demos batch-embed every repo upfront, then do plain vector search over the embeddings. This doesn't do that. At the scale of one person's GitHub account, pre-embedding everything is wasted work — most repos never get asked about, and the ones that do only need a few files read, not a full-repo index.

Instead, the LLM is given the responsibility of deciding what to explore:

• Tier 1 (always available, no embeddings needed): every repo's name, language, and a short description, loaded once per session.
• Tier 2 (on demand): the agent calls search_index first (checks a Postgres pgvector cache — free, no API cost). If the cache is empty or thin, it falls back to live GitHub calls (list_repo_files / read_repo_file). Whatever gets read live is chunked, embedded, and written into the cache as a deterministic side effect — not something the LLM decides to do — so the next question about that repo is cheaper.

This is a cache-augmented, lazy-indexing agent, not classic batch RAG and not pure live browsing either. Full reasoning: docs/LEARNINGS.md (§8-9), diagram in docs/ARCHITECTURE.md.

Try it

CLI (streaming, human-in-the-loop confirmations, model routing all live here):

docker-compose up -d                          # Postgres + pgvector
uv sync
GROQ_MODEL=openai/gpt-oss-120b uv run repo-rag  # interactive session

Renders live as it streams: tool calls as they happen (with args/latency), the answer's tokens as they generate, and citations — not a wall of JSON. Any live GitHub API call (list_repo_files/read_repo_file) pauses for a y/n/always confirmation first — a human-in-the-loop gate; search_index/get_repo_status never pause, they're cache-only. Structured logs go to repo-rag.log, not the terminal.

Web: docker-compose up --build runs the FastAPI app too, serving a minimal chat UI at http://localhost:8000 (POST /ask non-streaming, POST /ask/stream for SSE — the frontend uses the streaming one). No HITL here — pausing a live HTTP request for out-of-band approval needs a different mechanism than the CLI's synchronous prompt; accepted gap, not solved.

MCP: uv run repo-rag-mcp exposes the same four tools over the Model Context Protocol (stdio transport) — usable from Claude Desktop, Claude Code, or any other MCP client, independent of the Groq-specific agent loop above.

Does it actually work?

docs/eval/results.md — a golden set of real questions run against the live agent, scored for correctness and groundedness (not vibes). Methodology and the deliberate choice not to use an LLM judge: docs/eval/README.md.

What actually broke building this

docs/INCIDENTS.md — the real failures hit getting this running: a Postgres type-cast bug, a Groq client rejecting a parameter it shouldn't have, a model that deterministically mis-formats tool calls (not a flake — three different fix attempts produced the identical failure before the real cause was found: the model, not the prompt), and a token-budget collision that took several iterations to actually solve. One of these — an error class the graceful-degradation handler didn't catch — was found by running the eval harness, not by manual testing.

What's built, with a named limit

• Cache invalidation is SHA-based, not proactive. Every cached chunk stores the GitHub blob SHA it was fetched at. list_repo_files reuses the tree listing it already has to compare live SHAs against cached ones and drops anything stale — no extra API calls, since GitHub returns blob SHAs for free. The limit: this only fires when the agent decides to list a repo's files again, not on every cached search_index hit. A repo that's re-committed but never gets list_repo_files called on it again keeps serving stale cached chunks. Renamed/deleted files also aren't caught — only SHA mismatches on paths that still exist. docs/INCIDENTS.md, PgVectorStore.invalidate_stale.
• Rate-limit handling is tiered by wait length, not a queue. Groq returns a real retry-after header. A short wait (its per-minute limit, resets in seconds) gets one bounded automatic retry. A long wait (its per-day quota, resets in minutes to hours — see docs/INCIDENTS.md #10) fails fast with the actual wait time surfaced to the user, instead of either hanging the CLI for 20 minutes or pretending "try rephrasing" would help. What's still missing: no persistent retry queue across restarts.
• Multi-provider fallback is an abstraction, not a wired second provider. FallbackLLMClient (infrastructure/llm/fallback_client.py) tries a list of clients in order, falling through only on the long-wait RateLimitError case above — no second provider's API key was available to actually wire one in, so it's ready-to-plug-in, tested against fakes, not proven against a real second provider.
• Model routing is a two-tier cascade, not a learned router. The first turn of every question uses a cheap/fast model (llama-3.1-8b-instant); any turn after that (meaning a tool call was actually needed) escalates to the capable model. A real question mix showed a good fraction resolve with zero tool calls, i.e. cheaply. No complexity classifier, no cost tracking dashboard — just this one cascade point.
• Human-in-the-loop only exists on the CLI. Live GitHub tool calls (list_repo_files/read_repo_file) pause for a synchronous y/n/always prompt there. The streaming HTTP route has no equivalent — pausing a request mid-stream for an out-of-band approval is a different, harder mechanism that wasn't built.
• Streaming doesn't retry. GroqLLMClient.chat_stream() skips the tool-use-failed/short-rate-limit retry logic the non-streaming chat() has, since retrying would mean discarding partially-streamed text. Accepted gap.

What's deliberately not built

• No hybrid search or reranking. Flat cosine similarity only. Fine at single-profile scale; would need revisiting at more repos or more ambiguous queries.
• No LLM-as-judge eval. Deterministic fact-presence scoring instead, to keep eval runs cheap and reproducible on a free-tier account. Trade-off: can't catch a confidently wrong answer that happens to contain the right keywords.

Docs

• docs/PRD.md / docs/TRD.md — product and technical requirements
• docs/ARCHITECTURE.md — module map, data flow
• docs/LEARNINGS.md — design reasoning for the cache-augmented approach
• docs/INCIDENTS.md — real bugs, symptom → diagnosis → fix
• docs/eval/ — eval harness, methodology, results
• docs/AI_ENGINEER_ROADMAP.md — what this project covers against the AI Engineer curriculum, and what's genuinely left
• docs/STATUS_AND_NEXT_STEPS.md — what's working (verified, not assumed), what's left to close the zero-to-AI-engineer roadmap, what comes after