AI Due Diligence Copilot

AI Due Diligence Copilot is an end-to-end financial document analysis system that ingests corporate filings (10-K / 10-Q), extracts structured financial information, retrieves supporting evidence, and answers analyst questions using the most reliable source available.

Unlike many AI applications that rely entirely on an LLM, this system intelligently routes each question to either:

• Structured financial data stored in PostgreSQL
• Retrieved filing evidence through a Retrieval-Augmented Generation (RAG) pipeline
• A combination of both

To improve trustworthiness, generated answers can be evaluated using a PyTorch-based groundedness classifier that checks whether claims are supported by retrieved evidence.

---

Why This Project?

Financial analysts frequently need answers that are:

• Factually accurate
• Explainable
• Traceable to source documents

Traditional LLM-based assistants can hallucinate numbers or provide unsupported claims.

This project addresses that problem by:

• Using SQL for quantitative reasoning
• Using RAG for qualitative document understanding
• Using groundedness evaluation to assess evidence support
• Supporting optional Claude-based answer synthesis while remaining fully functional without any paid API

For deeper implementation details and design decisions, see:

ARCHITECTURE.md

---

Features

Document Ingestion

• Upload financial filings (.txt, .pdf, .md)
• Automatic document cleaning and chunking
• Filing metadata tracking
• Persistent PostgreSQL storage

Structured Financial Metric Extraction

Currently extracts and stores:

• Revenue
• Operating Margin
• Net Income
• R&D Expense
• Cash & Cash Equivalents

These metrics are stored in PostgreSQL and can be queried directly for quantitative analysis.

Intelligent Query Routing

The system automatically determines whether a question should be answered through:

• Structured SQL retrieval
• Document retrieval (RAG)
• SQL + RAG

Examples:

Question	Route
What was the change in operating margin?	SQL
What supplier risks does the company face?	RAG
How did margins change and why?	SQL + RAG

MCP Tool Integration

Implements Model Context Protocol (MCP) tools:

• Company Lookup
• Financial Ratio Calculator
• Document Search

Groundedness Evaluation

A lightweight PyTorch classifier evaluates whether generated claims are supported by retrieved evidence.

Outputs include:

• Groundedness scores
• Claim-level support classification
• Confidence indicators

Evaluation Harness

Built-in evaluation framework measuring:

• Routing accuracy
• Retrieval relevance
• Groundedness
• Latency

Interactive Dashboard

Web interface for:

• Filing uploads
• Question answering
• Viewing routing decisions
• Viewing tool usage
• Viewing groundedness scores

---

System Architecture

                    Financial Filing
                           │
                           ▼
                  Document Ingestion
                           │
                           ▼
                      Chunking
                           │
                           ▼
       Metric Extraction + Vectorization
                 (TF-IDF + SVD)
                 │              │
                 ▼              ▼
          PostgreSQL      Vector Store
                 ▲              ▲
                 │              │
User Query ───► Query Router ───┘
                 │
      ┌──────────┴──────────┐
      ▼                     ▼

 SQL Financial Route     RAG Retrieval

      ▼                     ▼

 Financial Metrics     Evidence Search

      └──────────┬──────────┘
                 ▼

         Answer Generation

                 ▼

      Groundedness Evaluation

                 ▼

          Final Response

---

Answer Generation Modes

1. Offline / Local Mode (Default)

No API keys required.

The system answers questions using:

• PostgreSQL financial data
• RAG retrieval
• Extractive answer generation
• PyTorch groundedness evaluation

This mode was used during development and testing.

---

2. Claude-Assisted Mode (Optional)

If an Anthropic API key is provided:

ANTHROPIC_API_KEY=your_api_key_here

retrieved evidence can be passed to Claude for natural-language answer generation.

Pipeline:

User Query
    ↓
Retrieval / SQL
    ↓
Claude
    ↓
Groundedness Evaluation
    ↓
Final Response

Claude is only used for answer synthesis.

The system does not depend on Claude for:

• Retrieval
• Query routing
• Financial calculations
• Groundedness scoring

If no API key is present, the application automatically falls back to the local extractive pipeline.

---

Tech Stack

Backend

Technology	Purpose
FastAPI	API Framework
Uvicorn	ASGI Server
Pydantic	Data Validation

Database

Technology	Purpose
PostgreSQL	Primary Database
SQLAlchemy	ORM

Machine Learning

Technology	Purpose
PyTorch	Groundedness Classifier
Scikit-Learn	Retrieval Pipeline
NumPy	Numerical Operations

Retrieval

Component	Purpose
TF-IDF	Document Vectorization
Truncated SVD	Dense Semantic Representation
Cosine Similarity	Retrieval Ranking

AI Tooling

Component	Purpose
MCP Tools	Structured Tool Access
Query Router	Route Selection
Groundedness Evaluator	Evidence Validation

---

Project Structure

diligence-copilot/
│
├── app/
│   ├── db/
│   ├── ingestion/
│   ├── ml/
│   ├── rag/
│   ├── mcp_tools/
│   └── main.py
│
├── data/
│
├── scripts/
│   └── setup_postgres.sql
│
├── tests/
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── ARCHITECTURE.md
└── README.md

---

Installation

Prerequisites

• Python 3.11+
• PostgreSQL 16+ (tested on PostgreSQL 17.11)
• Git

---

1. Clone Repository

git clone <repository-url>
cd diligence-copilot

---

2. Create Virtual Environment

Windows

python -m venv venv
.\venv\Scripts\Activate.ps1

Linux / macOS

python -m venv venv
source venv/bin/activate

---

3. Install Dependencies

Install CPU-only PyTorch:

pip install torch --index-url https://download.pytorch.org/whl/cpu

Install project requirements:

pip install -r requirements.txt

---

4. Configure PostgreSQL

Run:

psql -U postgres -f scripts/setup_postgres.sql

This creates:

• diligence_copilot database
• diligence_app user
• Required permissions

---

5. Build Groundedness Dataset

python -m app.ml.build_dataset

---

6. Train Groundedness Classifier

python -m app.ml.groundedness

---

7. Start Application

uvicorn app.main:app --reload

Application:

http://localhost:8000

API Documentation:

http://localhost:8000/docs

---

Docker

docker-compose up --build

---

Verification Walkthrough

Create Test Filing

Total revenue for fiscal year 2024 was $500 million.

Operating margin for fiscal year 2024 was 12.5%, compared to 10.1% in fiscal year 2023.

The company faces significant risk from a single supplier located in Vietnam.

---

Upload Filing

Use:

• Ticker: TEST
• Company: Test Company
• Fiscal Period: FY2024

Example output:

Done: 1 chunks, 3 financial metrics extracted.

---

Test Financial Reasoning

Question:

What was the change in operating margin?

Example output:

operating_margin moved from 10.1 (FY2023)
to 12.5 (FY2024), a change of 23.76%.

Route:

SQL

Tool:

financial_ratio_calculator

---

Test Retrieval

Question:

What supplier risks does the company face?

Example output:

The company faces significant risk from a single supplier located in Vietnam.

Note: For very small filings that fit into a single chunk, retrieval may return the entire chunk rather than a single sentence.

Route:

RAG

---

Run Evaluation

python -m app.eval.run_eval

---

Results

Validated end-to-end on:

• Document ingestion
• Financial metric extraction
• PostgreSQL persistence
• Query routing
• SQL-based financial reasoning
• Retrieval-based risk analysis
• Groundedness evaluation
• FastAPI deployment

Sample Evaluation Results

Metric	Value
Route Accuracy	1.00
Average Retrieval Relevance	0.67
Average Latency	~30ms

Groundedness Classifier

Metric	Value
Accuracy	0.70 – 0.90
Recall	0.75 – 1.00

Results vary slightly because the evaluation dataset is intentionally small.

---

Limitations

• Groundedness classifier trained on 38 labeled examples
• Financial metric extraction currently uses rule-based patterns
• Retrieval uses TF-IDF + SVD instead of transformer embeddings
• Vector index is in-memory and optimized for demonstration-scale workloads
• Claude integration is optional and not required for core functionality

---

Future Improvements

• Transformer-based embeddings
• Hybrid search (keyword + vector)
• Larger groundedness datasets
• LLM-based extraction fallback
• Multi-step agent workflows
• pgvector integration
• Multi-document comparison
• Automated analyst report generation