AgentOS Knowledge Platform

A production-minded hybrid RAG system for grounded, cited answers over unstructured pharma documents.

Part of a larger 5-project Agentic AI platform (see roadmap). This project is the foundational knowledge service — every other agent in the platform reaches it through MCP.

---

Why this project

This is a from-scratch reference implementation of the same category of system I built professionally: an unstructured-data pipeline making pharma documents (SOWs, clinical protocols, safety reports, regulatory correspondence) queryable in natural language, with citation-grounded answers a business or clinical user can trust. No proprietary data is used — the corpus is synthetic, mixed with real, public FDA guidance documents for genuine content diversity.

The goal wasn't a tutorial-grade demo. Every major design decision, bug, and fix documented here happened during real development — including a citation-grounding bug that was root-caused and fixed with before/after evidence (see the case study).

What it does

Ask a natural-language question, get a grounded answer with source citations and a confidence score:

POST /ask
{
  "question": "What is the primary endpoint for the NVX-204 Phase II study?",
  "session_id": "demo",
  "top_k": 5
}

{
  "answer": "The primary endpoint for the NVX-204 Phase II study is the change from baseline in DAS28-CRP score at Week 24.",
  "citations": [
    { "chunk_id": "protocol_summary_nvx204.md_recursive_1", "source_file": "protocol_summary_nvx204.md" }
  ],
  "confidence": 1.0,
  "context_used": true
}

Features

• Hybrid retrieval — dense (pgvector) + sparse (BM25) fusion via Reciprocal Rank Fusion, with cross-encoder reranking
• Grounded generation — structured-output answers with per-claim citations and a confidence score, explicitly instructed to decline when context is insufficient
• Multi-format ingestion — Markdown, plain text, PDF (with OCR fallback for scanned pages), DOCX, HTML, and CSV/XLSX, all through one dispatcher
• Deduplication — cosine-similarity dedup at ingest, idempotent across repeated ingestion runs (deterministic chunk IDs)
• Conversational memory — LangGraph-orchestrated, Redis-checkpointed short-term memory (session-scoped) plus Postgres-backed long-term conversation summaries, with query rewriting so follow-up questions resolve correctly
• MCP server — exposes search_documents, retrieve_context, and index_documents as MCP tools for other agents in the platform
• Live ingestion API — POST /ingest with automatic index refresh, no restart required
• Tested — unit + integration tests (isolated test database, mocked external calls), including regression tests tied to real bugs found during development
• Evaluated — RAGAS-based quality scoring (faithfulness, context precision/recall, answer relevancy) against a golden dataset
• Visualized — UMAP projection of the embedding space, confirming real semantic clustering by document category

Architecture

flowchart TB
    Client(["Client / Swagger UI / MCP client"])

    Client -->|"POST /ingest"| Ingest["Ingestion"]
    Ingest --> Loader["Multi-format loader<br/>md, txt, pdf, docx, html, csv"]
    Loader --> Chunker["Chunker<br/>fixed-size / recursive (heading-aware)"]
    Chunker --> Store["pgvector store<br/>+ dedup + embeddings"]
    Store --> BM25["In-memory BM25 index"]

    Client -->|"POST /ask"| Graph["LangGraph pipeline"]
    Graph --> Rewrite["Query rewriting<br/>(uses chat history)"]
    Rewrite --> Hybrid["Hybrid search<br/>dense + sparse + RRF + rerank"]
    Hybrid --> Store
    Hybrid --> BM25
    Hybrid --> Gen["Grounded generation<br/>citations + confidence"]
    Gen --> Client

    Graph <-->|"checkpointed state"| Redis[("Redis Stack<br/>short-term memory")]
    Graph -->|"on-demand summarize"| PG[("Postgres<br/>long-term memory")]

    MCP(["MCP clients<br/>(future: Project 2 agents)"]) --> MCPServer["MCP server<br/>search_documents / retrieve_context / index_documents"]
    MCPServer --> Hybrid
    MCPServer --> Ingest

Tech stack

FastAPI · Pydantic v2 · SQLAlchemy (async) · PostgreSQL + pgvector · Redis Stack (RediSearch) · LangGraph · OpenAI (embeddings + generation) · rank-bm25 · sentence-transformers (cross-encoder reranking) · FastMCP · Docker / Docker Compose · pytest · RAGAS · UMAP

Quickstart

git clone <repo-url>
cd AgentOS-Knowledge
cp .env.example .env   # add your OPENAI_API_KEY

docker compose up -d
python scratch_test.py   # ingests data/raw_docs/ and runs a sanity-check query

uvicorn app.main:app --reload

Open http://127.0.0.1:8000/docs for the interactive API.

API

Endpoint	Purpose
GET /health	Liveness check
POST /ask	Ask a question, get a grounded, cited, session-aware answer
POST /ingest	Upload a document, index it, immediately searchable
POST /conversations/{session_id}/summarize	Persist a long-term summary of a conversation
GET /conversations/recent-summaries	Retrieve recent cross-session summaries

MCP server

mcp dev app/mcp_server.py

Exposes search_documents, retrieve_context, and index_documents for any MCP-compatible agent client.

Testing & evaluation

pytest -v                    # unit + integration tests
python -m eval.run_eval      # RAGAS evaluation against the golden dataset
python -m eval.visualize_embeddings   # UMAP embedding visualization

Results: faithfulness 1.0, context recall 1.0, context precision 0.93, answer relevancy 0.82. Full breakdown, embedding visualizations, and a worked debugging case study: eval/evaluation_report.md.

Engineering challenges & fixes

Real production-level issues found and resolved during development — kept here in detail because they're a stronger signal of engineering depth than a feature list.

1. Citation-grounding bug (retrieval quality)

/ask returned a factually correct answer but cited the wrong source document — a regulatory correspondence log, instead of the actual protocol document where the fact came from.

Root cause: the markdown chunking library split section headings into metadata only, invisible to embeddings, BM25, and the reranker. The protocol's "Primary Endpoint" chunk was stored as just "Change from baseline in DAS28-CRP score at Week 24" — the words "primary endpoint" never appeared in its searchable text, while a competing chunk from another document happened to contain that exact phrase in a fluent sentence.

Fix: contextual chunk enrichment — prepending document title and section heading directly into each chunk's indexed content. Verified with before/after reranker scores: the correct chunk went from absent in the top 20 results to a confident second place, and the citation corrected accordingly.

Residual limitation: fluency bias is reduced, not eliminated — a longer, more fluently-written passage can still occasionally outrank a short, factually authoritative chunk. Would need a stronger reranker or explicit source-type weighting to close further.

2. Idempotency bug (data integrity)

Re-running ingestion on unchanged documents created duplicate rows on every run, instead of recognizing the content already existed.

Root cause: chunk_id was generated with uuid4() — random on every run — defeating the ON CONFLICT DO NOTHING upsert logic, which relies on a stable, repeatable ID to detect "this row already exists."

Fix: deterministic chunk IDs derived from stable properties (source_file + chunking strategy + chunk_index). Verified by running ingestion twice in a row and confirming stored chunk count didn't grow.

3. Stale in-memory index after live ingestion (operational correctness)

Documents uploaded via POST /ingest weren't searchable until the server was manually restarted.

Root cause: BM25 is an in-memory index, rebuilt once per process at startup. Any new data written to Postgres wasn't reflected in the BM25 index until it was explicitly rebuilt — a gap that showed up repeatedly across different entry points (the API server, a RAGAS eval script, and the MCP server each needed their own fix).

Fix: /ingest (and the MCP index_documents tool) now call rebuild_bm25_index() immediately after storing new chunks. Verified by uploading a new document and successfully querying it with no restart in between.

Known limitation, by design: the in-memory approach is a deliberate dev-scale tradeoff — production deployment would move to OpenSearch (natively persistent, incrementally indexable), rather than solving persistence for the current BM25 library.

4. Missing infrastructure capability (Redis + LangGraph memory)

Wiring LangGraph's checkpointer to Redis failed with unknown command 'FT._LIST'.

Root cause: langgraph-checkpoint-redis requires RediSearch (a Redis module for indexed lookups), which isn't present in vanilla open-source Redis.

Fix: switched the Redis image to redis/redis-stack-server, which bundles RediSearch.

Deployment implication, documented ahead of time: AWS ElastiCache's standard managed Redis does not include RediSearch — production deployment will need a self-managed Redis Stack instance or an alternative managed service.

5. Dependency-ecosystem instability (RAGAS and MCP)

Two separate multi-step incidents, both root-caused via direct evidence (inspecting installed package versions and declared dependencies) rather than trial-and-error:

• RAGAS eagerly imports every optional LLM-provider integration at module load time; an installed langchain_community version had removed a submodule RAGAS's code depended on. Worked around with a stub module; flagged as fragile and a candidate to revisit (e.g. deepeval).
• MCP/FastMCP: the official SDK's API had been substantially restructured between versions, and the companion fastmcp package's loose dependency constraint (mcp>=1.24.0, no upper bound) allowed installing an incompatible newer major version. Fixed with an explicit compatible pin (mcp>=1.24.0,<2.0).

Both are documented as a real characteristic of building on current- generation LLM/agent tooling — young, fast-moving ecosystems where companion-library version compatibility isn't always guaranteed by default resolution.

6. CSV parsing — a deliberate strictness tradeoff

A hand-authored CSV with an unquoted comma inside a text field caused pandas to reject the entire file rather than misparse it.

Design decision, not just a fix: rather than switching to lenient row-skipping parsing, strict parsing was kept deliberately — in a compliance-adjacent domain, silently dropping a row of batch-release data is worse than failing loudly and requiring the source file to be corrected.

Known limitations

• Fluency bias in reranking (see #1 above) — reduced, not eliminated
• No image/diagram captioning — embedded figures in PDFs/DOCX aren't extracted or described; a page with a diagram and no surrounding text yields no searchable content for that diagram
• Tabular (CSV/XLSX) chunking isn't row-boundary-aware — rows are converted to paragraphs but chunked with the same generic logic as prose
• BM25 index is in-memory/per-process (see #3 above)
• Corpus has real class imbalance — 2 real FDA documents account for 92% of stored chunks, visible directly in the embedding visualization
• CSV parsing fails the whole file on a malformed row, by design (see #6)

Platform roadmap

This is Project 1 of 5 in a larger interconnected Agentic AI platform:

#	Project	Status
1	AgentOS Knowledge Platform (this repo)	✅ Complete
2	Multi-Agent Enterprise Assistant	Not started
3	AgentOps Platform	Not started
4	AI Safety & Guardrails Platform	Not started
5	Continuous Learning Platform	Not started

AWS deployment (ECS/EKS, OpenSearch, Terraform, CI/CD) is deliberately deferred until all 5 projects have working local versions.