Policy Document Q&A Assistant — RAG + MCP Server (LangChain, Azure AI Foundry, PostgreSQL/pgvector, Docker, Langfuse)

A question-answering assistant over real Australian public policy documents (Queensland Government financial/ICT policy and ATO federal tax guidance), built as a Retrieval-Augmented Generation (RAG) pipeline and exposed as an MCP (Model Context Protocol) server — so it plugs into Claude Desktop or any other MCP-compatible client as a callable tool, not just a one-off script. The whole pipeline, from PDF ingestion through to the tool itself, runs containerized with Docker and is instrumented end-to-end with Langfuse tracing.

---

OBJECTIVE

Built to close a specific set of AI Engineer skill gaps identified across recent job applications (Queensland Treasury, Warren and Mahoney, McCosker, CI&T, Datacom) — agent tooling via MCP, LangChain-based orchestration, a real vector database (not an embedded/toy store), containerization, a named cloud-native AI platform (Azure AI Foundry specifically, not just "an OpenAI API key"), and LLMOps observability. Rather than collecting certificates, the goal was one real, working, end-to-end system that demonstrates all six hands-on — the same approach used for the sibling lead-scoring-databricks-pipeline project.

The application itself mirrors a realistic enterprise use case: "let staff ask natural-language questions against our own policy documents, with every answer traceable back to a real source page" — thematically relevant to government-adjacent roles specifically.

---

DATASET USED

8 real PDF documents (162 pages total), sourced directly from official .gov.au domains — no third-party mirrors:

• FAH_Volume_1_Introduction_2025.pdf — Queensland Financial Accountability Handbook, Vol. 1
• Overview-of-Queensland-Financial-Accountability-Framework-as-at-Jan-2020.pdf
• ict-as-a-service_decision_framework-overview_v1_0_0.pdf
• paf-policy-overview.pdf / paf-supporting-guidelines.pdf — Queensland Project Assurance Framework (incl. public-private partnerships)
• n75057 [DE-81739] - 2026 Tax Time toolkit for small business_DIGITAL.pdf — ATO
• n75127 [DE-73758] - Residency for tax purposes - factsheet_DIGITAL.pdf — ATO
• tr2023-001.pdf — ATO Taxation Ruling 2023/1

The corpus is deliberately split across two clusters — Queensland state policy and Australian federal tax guidance — so retrieval has to genuinely discriminate between state vs. federal sources, and between a formal ruling and a plain-language summary of a related topic, rather than trivially matching on a single obvious keyword. An initial set of 11 documents (344 pages) was trimmed down to these 8 (162 pages) to keep the corpus large enough to be a real retrieval problem while staying fast to iterate on.

---

ARCHITECTURE & PIPELINE

flowchart LR
    subgraph Ingestion["Ingestion (run once, re-run on doc changes)"]
        A[Policy PDFs] --> B[Chunking\nLangChain RecursiveCharacterTextSplitter]
        B --> C[Embed\nAzure text-embedding-3-small]
        C --> D[(PostgreSQL + pgvector\nvia Docker)]
    end

    subgraph Query["Query time"]
        E[MCP Client\ne.g. Claude Desktop] --> F[MCP Server\nPython, containerized]
        F --> G[LangChain RAG chain]
        G --> D
        G --> H[Azure AI Foundry\ngpt-5-mini]
        H --> G --> F --> E
    end

    G -. traces .-> I[Langfuse\nobservability]

Ingestion: Each PDF is loaded page-by-page with LangChain's PyPDFLoader (preserving source filename + page number as metadata for later citations), then split with RecursiveCharacterTextSplitter at 1000 characters / 150 overlap — chosen over a smaller 500/50 split after comparing both directly on the real corpus (950 vs. 521 chunks); the larger chunk size suits these procedural, structurally dense policy documents better, giving the model more surrounding context per retrieved piece. Each of the 521 resulting chunks is embedded via Azure's text-embedding-3-small and written into a PostgreSQL database running in Docker, with the pgvector extension enabling vector similarity search directly inside the relational database rather than a separate dedicated vector store — a more production-credible pattern, since many real organizations add vector search to a Postgres instance they already run.

Query time: A question is embedded with the same model, PostgreSQL returns the top-4 most similar chunks by vector similarity, and those chunks are "stuffed" into a prompt that instructs gpt-5-mini to answer only from that retrieved context — or say "I don't know" rather than fall back on its own general knowledge. The answer is returned together with the exact source document and page number for every chunk it drew from, so every answer is independently verifiable.

Interface: The RAG chain is wrapped as an MCP server using the official Python MCP SDK, exposing two tools — query_policy_docs (the core Q&A capability) and list_indexed_documents (lets a client show what's actually covered before asking). The RAG logic and the MCP protocol layer are kept in separate files on purpose, so the underlying capability could be reused behind a different interface later without touching it. The server is containerized with Docker; because MCP's stdio transport requires the server to be attached directly to its client's stdin/stdout, it runs as a short-lived container spawned on demand per connection (docker compose run --rm -i), unlike the always-on PostgreSQL container.

Observability: Every query is traced end-to-end with Langfuse — retrieval and generation nested under one connected trace per question, showing the exact prompt, retrieved chunks, latency, and token cost.

---

MODEL USED

Both models are deployed via Azure AI Foundry (chosen deliberately over the classic "Azure OpenAI" resource type, since Foundry is named explicitly in several target job postings):

• gpt-5-mini — generation, temperature=0 for faithful, literal answers grounded in retrieved text rather than creative ones.
• text-embedding-3-small — embeddings, for both the document corpus and incoming questions.

This Foundry resource uses Azure's newer unified v1 API surface (.../openai/v1), so the integration uses LangChain's plain ChatOpenAI / OpenAIEmbeddings classes with base_url + api_key + model, rather than the Azure-specific AzureChatOpenAI / AzureOpenAIEmbeddings classes, which target the older dated-api-version endpoint scheme and 404 against this resource.

---

RESULTS & INSIGHTS

Evaluated manually with 5 real test questions run through the live system:

1. "What is the Financial Accountability Handbook designed to help agencies do?" → correct, near-verbatim grounded answer with accurate citations.
2. "What does the ICT-as-a-service decision framework help agencies decide?" → correct, captured the real nuance (deployment/service models, risk assessment).
3. "What is the capital of France?" (deliberately out-of-scope) → correctly answered "I don't know" instead of using the model's own general knowledge — the key anti-hallucination test.
4. "What is a public private partnership under Queensland's PPP policy?" → answered "I don't know." Verified independently by directly inspecting the source PDF's extracted text: the document genuinely never defines the term in plain language anywhere in its 23 pages — a real, defensible corpus gap correctly identified rather than a retrieval failure or a fabricated answer.
5. "What record-keeping obligations does the small business Tax Time toolkit mention?" → correct, specific detail, accurate citation.

5/5 sensible behavior — 3 correct grounded answers and 2 correct refusals (one genuinely out-of-scope, one a real corpus gap correctly recognized rather than hallucinated). Distinguishing "retrieval missed the right chunk" from "the corpus genuinely doesn't contain the answer" is a real RAG evaluation skill, and this round demonstrated both failure modes being handled correctly — not just an eyeballed pass/fail.

Verified against a real MCP client (Claude Desktop) end-to-end, both running locally via .venv and fully containerized via Docker, with every query traced in Langfuse.

---

SETUP & USAGE

Requirements: Docker Desktop, Python 3.13, an Azure AI Foundry resource with gpt-5-mini and text-embedding-3-small deployed, and (optionally) a free Langfuse Cloud account for tracing.

1. Copy .env.example to .env and fill in your own Azure, Postgres, and (optionally) Langfuse credentials.
2. Start the vector store: docker compose up -d postgres
3. Ingest the documents (one-off, or whenever the document set changes):

   pip install -r requirements-ingest.txt
   python embed_and_store.py
   

4. Run the assistant directly from the CLI: python rag_chain.py
5. Or connect it as an MCP tool: build the server image (docker compose build mcp-server), then add it to your MCP client's config (e.g. Claude Desktop's claude_desktop_config.json):

   "policy-rag": {
     "command": "docker",
     "args": ["compose", "-f", "<path-to-repo>/docker-compose.yml", "run", "--rm", "-i", "mcp-server"]
   }
   

---

CONCLUSIONS

The pipeline is fully working end to end: real government PDFs are chunked, embedded, and stored in a real vector database; retrieval and generation are grounded and correctly refuse to answer beyond the corpus; the whole capability is exposed as a standards-compliant MCP tool, verified live against a real MCP client; the server is containerized and portable; and every query is traced with cost, latency, and full context for observability. Every layer named as a target skill gap — MCP, LangChain, a real vector database, Docker, Azure AI Foundry, and LLMOps tracing — is demonstrated with working, tested code rather than a tutorial-only implementation.

---

AUTHOR INFORMATION

Varun Vikas Jaiswal 2026

---

KEYWORDS

RAG, Retrieval-Augmented Generation, LangChain, MCP, Model Context Protocol, Azure AI Foundry, Azure OpenAI, pgvector, PostgreSQL, vector database, vector similarity search, Docker, Docker Compose, Langfuse, LLMOps, observability, Python, AI Engineer