Aegis — Agent-Ready Knowledge Compiler

Aegis is a durable, evidence-grounded knowledge compiler that transforms instructional video content (starting with YouTube playlists in v1) into versioned, structured, and validated knowledge packages exposed to AI agents via the Model Context Protocol (MCP 2026-07-28).

---

Key Capabilities

• Deterministic 10-Stage Pipeline: Discover $\rightarrow$ Acquire $\rightarrow$ Evidence Extraction $\rightarrow$ Segmentation $\rightarrow$ Structured Extraction $\rightarrow$ Evidence Binding $\rightarrow$ Enhancement $\rightarrow$ Validation $\rightarrow$ Vector Indexing $\rightarrow$ Atomic Publication.
• PostgreSQL 18 + pgvector Single Source of Truth: All stage executions, worker leases, idempotent operation identities, evidence citations, and knowledge items reside in PostgreSQL.
• Content-Addressed Immutable Artifacts: All intermediate outputs and raw transcripts are hashed (SHA256) and stored in S3/MinIO.
• Hybrid Semantic & Lexical Retrieval: Combines dense text-embedding-3-small vector similarity with PostgreSQL full-text search and evidence-support boosting.
• Read-Only MCP 2026-07-28 Server: 5 read-only tools exposed over stdio and Streamable HTTP for seamless integration with Claude Desktop, Cursor, and custom agent SDKs.
• Operator Console: Next.js 16 + React 19 web dashboard for ingestion, run tracking, evidence timeline exploration, and human-in-the-loop review queue resolution.

---

Monorepo Layout

aegis/
├── apps/
│   ├── api/          # FastAPI 0.139 Control REST API (:8000)
│   ├── worker/       # Background processing daemon & reconciler
│   ├── mcp/          # Read-only MCP 2026-07-28 Server (:8001)
│   └── web/          # Next.js 16 + React 19 Operator Console (:3000)
├── packages/
│   ├── domain/       # Pure domain models (zero framework dependencies)
│   ├── schemas/      # Pydantic v2 validation models & extraction schemas
│   ├── database/     # SQLAlchemy 2.0 async models (16 tables) & repositories
│   ├── pipeline/     # Idempotency engine, lease manager, 10-stage graph
│   ├── providers/    # External AI (OpenAI GPT-5.6) & YouTube (yt-dlp) adapters
│   ├── retrieval/    # Hybrid vector + lexical search engine
│   └── storage/      # Content-addressed S3/MinIO storage backend
├── migrations/       # Alembic database migrations
├── infra/
│   ├── compose/      # Docker Compose definition & overrides
│   ├── docker/       # Production multi-stage Dockerfiles
│   └── scripts/      # Backup, restore, migration, and healthcheck utilities
├── tests/            # Unit, integration, contract, evaluation, and crash recovery tests
└── docs/             # Architecture, REST API, operations runbook, and MCP guide

---

Technology Stack

Layer	Technology
Backend Runtime	Python 3.13
Dependency Manager	uv (Python) + pnpm (Node.js)
Control API	FastAPI 0.139, Pydantic v2
Orchestration & State	LangGraph 1.2, SQLAlchemy 2.0 Async, Alembic
Database & Search	PostgreSQL 18 + pgvector 0.8.6
Object Storage	AWS S3 / MinIO (Content-Addressed)
LLM & Embeddings	OpenAI Responses API (GPT-5.6) / text-embedding-3-small
Media Extraction	yt-dlp 2026.7.4
Protocol Integration	Model Context Protocol (MCP 2026-07-28 Python SDK v2)
Operator Console	Next.js 16, React 19, TypeScript, Tailwind CSS
Quality & Linters	pytest, pytest-asyncio, Ruff

---

Quickstart

Option A: One-Liner Production Installation (curl)

Bootstrap dependencies, virtual environment, and configuration with a single command:

# Automated installer (fetches uv, clones/syncs repo, initializes .env with secure keys, checks Docker)
curl -fsSL https://raw.githubusercontent.com/Demi8-patch/aegis/main/install.sh | bash

Or execute locally from the repository root:

./install.sh

---

Option B: Quickstart with uv

# 1. Clone and Bootstrap Environment
cp .env.example .env
uv sync
pnpm install

# 2. Verify System Health & Diagnostics
uv run aegis doctor

# 3. Start PostgreSQL 18 (with pgvector) and MinIO
docker compose -f infra/compose/docker-compose.yml up -d

# 4. Apply Database Migrations
uv run aegis migrate

---

Unified Aegis CLI Reference

Aegis provides a comprehensive command-line tool aegis accessible via uv run aegis <command> (or directly as aegis when installed):

Command	Description	Example
aegis doctor	Probes environment, database, pgvector, S3 storage, and OpenAI API keys.	uv run aegis doctor
aegis init	Safely creates .env with a cryptographically secure 256-bit SECRET_KEY.	uv run aegis init
aegis ingest <url>	Compiles a YouTube playlist or video end-to-end through the 10-stage pipeline.	uv run aegis ingest "https://youtube.com/playlist?list=..."
aegis mcp	Starts the read-only Model Context Protocol (MCP 2026-07-28) server.	uv run aegis mcp --port 8001 or uv run aegis mcp --stdio
aegis api	Launches the FastAPI Control REST API server with Swagger docs.	uv run aegis api --port 8000
aegis worker	Runs the durable background worker daemon and periodic lease reconciler.	uv run aegis worker
aegis eval	Runs continuous retrieval evaluation benchmarks or records regression cases.	uv run aegis eval --list-cases
aegis migrate	Applies latest Alembic database migrations.	uv run aegis migrate
aegis backup	Creates a compressed PostgreSQL database dump and syncs S3 artifacts.	uv run aegis backup --backup-dir ./backups
aegis restore	Restores database schema and object storage artifacts from a backup archive.	uv run aegis restore backups/aegis_db_*.sql.gz

---

Launching Development Services

# Terminal 1: FastAPI Control API (:8000)
uv run aegis api --port 8000 --reload

# Terminal 2: Aegis Background Worker & Reconciler
uv run aegis worker

# Terminal 3: Read-Only MCP Server (:8001 / stdio)
uv run aegis mcp

# Terminal 4: Operator Web Console (:3000)
pnpm --filter aegis-web dev

The Operator Console will be live at http://localhost:3000, the Control API at http://localhost:8000, and the MCP endpoint at http://localhost:8001/mcp.

---

MCP Tools Reference

The Aegis MCP server exposes 5 read-only tools conforming to the MCP 2026-07-28 specification:

MCP Tool	Description
search_knowledge	Hybrid vector + lexical search across published knowledge items with evidence citations.
get_knowledge	Fetches a structured knowledge item by UUID with bound citations and semantic relationships.
get_evidence	Retrieves the raw transcript segment, start/end video seconds, and confidence score.
search_sources	Searches imported playlists and videos by title, URL, channel, or video ID.
get_source	Fetches source metadata, video durations, and published package versions.

For client integration setup (Claude Desktop, Cursor, Windsurf, custom agents), see the MCP Client Integration Guide.

---

Testing & Quality Assurance

Aegis includes a comprehensive test suite with 82+ tests covering unit logic, integration flows, crash recovery, prompt injection defense, and retrieval quality.

# Run full pytest suite (82 passed)
uv run pytest tests/

# Run Ruff linter and style checks
uv run ruff check .
uv run ruff format --check .

# Build Wheel & Source Distribution
uv build

# Build Next.js Operator Web Console
pnpm --filter aegis-web build

---

Production Operations & Disaster Recovery

• Database & Artifact Backup:

  ./infra/scripts/backup.sh
  

• Database & Artifact Restore:

  ./infra/scripts/restore.sh backups/aegis_db_YYYYMMDD_HHMMSS.sql.gz backups/aegis_artifacts_YYYYMMDD_HHMMSS.tar.gz
  

• Comprehensive Documentation:
  • Architecture Specification
  • REST API Reference
  • Operations Runbook
  • MCP Client Guide

---

License

Apache-2.0