English | 日本語

Hybrid RAG — Agent Long-Term Memory System

A hybrid RAG system combining dense and sparse retrieval, with a tag-based memory mechanism (importance, staleness rate per knowledge_type, and access frequency) built into its reranking. For design rationale, see hybrid_rag_agent_spec.en.md.

Run it as an MCP server and agents such as Claude Code can use it directly as "long-term memory."

How this mechanism works

Following the spec, processing is split into two kinds.

Class	Content	Implementation
① Model-dependent (reasoning)	Importance tagging, query expansion / sufficiency judgment, orchestration	Agent side (LLM judgment)
② Structure-dependent (deterministic processing)	Chunking, embedding generation, hybrid search, staged reranking, forgetting/archival	RAG side (this library / MCP server)

"Importance" and "staleness rate per knowledge_type" are treated as separate axes; rather than a simple linear combination, they are applied in stages: ① cutoff by importance → ② time decay by knowledge_type → ③ boost by access frequency (see spec section 2.3 for details).

principle  : no decay (MBSE design principles, math/algorithms)
paper      : re-evaluated roughly every half year (papers, technical articles)
news       : decays significantly over weeks to months (news, model-release info)
experiment : decays according to project duration (experiment logs, run records)

knowledge_type is designed to be determined deterministically from the ingestion source rather than judged by an LLM from chunk content (e.g., a design document explicitly registered by a human → principle; an arXiv paper/technical article → paper; news/web search results → news; an execution log → experiment).

Note: principle (no decay) does not guarantee that a chunk will "never be forgotten by run_forgetting_batch." Because staged reranking applies the ① importance cutoff first, a knowledge_type=principle chunk can still become an archival target if its importance is set low and falls below importance_threshold (confirmed by tests/test_archival.py). "No decay" applies only to the ② time-decay stage — it is not a "never forgotten" guarantee across the full ①②③ pipeline.

Note: The memory mechanism (knowledge_type/importance/staged reranking/forgetting batch/MCP server) is implemented only for the FAISS backend (HybridRAGSystem). The Qdrant/Chroma/PostgreSQL versions are available only as a plain hybrid-search library.

Installation

pip install -r requirements.txt

For development/testing:

pip install -r requirements-dev.txt

Usage ① As an MCP server (recommended)

Starting the server

python mcp_server/server.py

The storage locations can be set via environment variables (defaults: hybrid_rag.db / indices).

HYBRID_RAG_DB_PATH=my_memory.db HYBRID_RAG_INDEX_PATH=my_indices python mcp_server/server.py

Registering with Claude Code

.mcp.json at the project root is already set up as follows. Claude Code picks it up automatically when it opens this repository.

{
  "mcpServers": {
    "hybrid-rag-memory": {
      "type": "stdio",
      "command": "python",
      "args": ["mcp_server/server.py"],
      "env": {
        "HYBRID_RAG_DB_PATH": "hybrid_rag.db",
        "HYBRID_RAG_INDEX_PATH": "indices"
      }
    }
  }
}

If you use a virtual environment, rewrite command to the absolute path of the Python interpreter inside your venv (e.g., "command": "./.venv/Scripts/python.exe").

Tools provided

In addition to the minimal 3 tools (①–③) called for in spec section 5, this server provides 10 more tools (④–⑬, spec extensions) for data ingestion, tagging, duplicate prevention, forgetting batches, and health checks.

#	Tool	Description
①	embed(text)	Vectorizes text with a fixed embedding model (deterministic processing)
②	hybrid_search(query, tags?, filters?, top_k?, include_stats?)	Hybrid vector+BM25 search. Returns chunks that have already gone through relevance reranking (Cross-encoder). With include_stats=True, the return shape becomes {"chunks": [...], "stats": {...}}, adding orphan_index_entries (entries dropped as index debris) and duplicate_contents (entries collapsed for identical body text) alongside timing info [include_stats is a spec extension, added 2026-08-08]
③	rerank(chunks, time_weight?, freq_weight?, importance_threshold?)	Staged reranking: importance cutoff → per-knowledge_type time decay → access-frequency boost
④	ingest(file_paths, metadata?, rebuild_index?)	Ingests documents. rebuild_index=True (default) runs a lightweight incremental update (update_index) internally [spec extension]
⑤	set_chunk_tags(doc_id, chunk_index, importance?, knowledge_type?, tags?)	Assigns importance tags / re-tags knowledge_type (intended for a human gate) [spec extension]
⑥	find_by_tag(tag)	Exact-match tag lookup (bypasses semantic search). Used to check whether a document from the same source has already been ingested [spec extension]
⑦	get_document_chunks(doc_id, chunk_index?, window?)	Fetches neighboring chunks within the same document (a direct lookup that bypasses semantic search). Compensates for context lost at chunk boundaries [spec extension]
⑧	delete_document(doc_id, rebuild_index?)	Deletes a document and all of its chunks. Used in the "replace" flow when re-ingesting [spec extension]
⑨	update_index()	A lightweight index update that incrementally incorporates only the chunks added since the last index update [spec extension, added 2026-07-30]
⑩	rebuild_index()	Does a full rebuild of the FAISS/BM25 indexes from every chunk in the DB. Required after deletions (⑧ or run_forgetting_batch), since incremental updates cannot handle them [spec extension]
⑪	run_forgetting_batch(time_weight?, freq_weight?, importance_threshold?, score_threshold?, dry_run?)	The forgetting/archival batch job. Intended for infrequent execution only [spec extension]
⑫	index_health()	Checks consistency between the index and the DB and reports it (makes no changes). Detects "in the index but not in the DB" debris (from forgetting to call rebuild_index after a deletion) and "in the DB but not in the index" gaps (from forgetting to call update_index after ingest(rebuild_index=False)) [spec extension, added 2026-08-08]
⑬	get_system_stats()	Reports the DB's memory-mechanism tagging coverage (makes no changes). Where index_health looks at structural consistency between the index and the DB, this looks at "how well the memory axis can actually function" — counts per knowledge_type, the importance-set rate, the tag-coverage rate, etc. [spec extension, added 2026-08]

Without ④–⑬, the ①–③ tools alone can neither ingest data, finalize importance tags, nor avoid duplicate registration from the same source — making the system impractical, which is why they were added.

A note on ingesting many files back-to-back (important)

Background (a past issue fixed on 2026-07-30): ingest used to default to "re-embed every chunk in the DB and rebuild the index on every call," so the cost of a single call grew linearly with corpus size, and ingesting files one at a time in a row would time out. ingest(rebuild_index=True) (the default) now calls update_index() internally — an incremental approach that embeds only the newly added chunks since the last update and .add()s them to the FAISS index — so it is now fast regardless of overall corpus size (the BM25 side still does a lightweight full rebuild every time, because its IDF statistics depend on the whole corpus, but this is cheap since it involves no neural embedding).

That said, running an incremental update on every single file is still wasted overhead, so when ingesting many files back-to-back it's better to pass rebuild_index=False to each ingest call and call update_index() once at the end of the batch to reconcile everything at once. .claude/agents/doc-to-memory.md and .claude/agents/session-to-memory.md are already implemented with this pattern. Checking the DB via find_by_tag (for duplicate prevention / progress verification) queries SQLite directly, so it works without waiting for the index to catch up.

When a full rebuild_index() is required: whenever the batch includes even a single delete_document call or an archival pass from run_forgetting_batch (i.e., any vector deletion). Incremental addition (update_index) only supports adding to FAISS, not removing from it, so any batch that includes deletions must end with a full rebuild_index(). A batch consisting purely of new additions is fine with update_index().

Preventing duplicates when re-registering the same source

Because ingest derives doc_id from a hash of the file's contents, re-ingesting byte-identical content is automatically skipped (a diff-based update). However, in cases where the same source (e.g., the same session) is re-summarized by an LLM and re-ingested every time, slight variations in the summary text each time will cause it to be treated as a different document — creating duplicates.

To avoid this, ingest with a unique identifier tag (e.g., session_id:xxx) and an updated-at tag (e.g., session_last_activity:2026-07-28T15:59:49Z), and on subsequent runs:

1. Check whether the document already exists via find_by_tag("session_id:xxx")
2. If the existing updated-at tag matches the current value, skip — do nothing
3. Only if it differs (the source has changed), remove the old one with delete_document(doc_id, rebuild_index=False) before ingest-ing the new content

Implementing this "skip if unchanged, replace if changed" pattern is recommended. .claude/agents/session-to-memory.md is a reference implementation of this pattern.

Usage example (conceptual)

1. ingest(["design_doc.md"], metadata={"knowledge_type": "principle", "tags": ["mbse"]})
2. hybrid_search("about consistency between requirements and architecture", top_k=5)
   -> [{"doc_id": ..., "chunk_index": ..., "content": ..., "knowledge_type": "principle",
        "importance": null, "access_count": 0, "score": 0.87}, ...]
3. set_chunk_tags(doc_id, chunk_index, importance=0.9)
4. rerank(chunks, time_weight=0.5, freq_weight=0.1, importance_threshold=0.3)
   -> chunks reordered along the memory axis (staleness, frequency, importance)

Usage ② As a Claude Code agent

.claude/agents/rag-memory.md provides a sub-agent definition responsible for the "agent side (class ①)" of this memory mechanism. Once .mcp.json is registered, you can invoke it from Claude Code like:

Use the rag-memory agent to look into past design decisions

The operating rules for actions that need human intent confirmation — importance tagging, knowledge_type re-tagging, deciding when to run the forgetting batch — are also written into this agent definition.

Additionally, .claude/agents/session-to-memory.md is a dedicated agent that summarizes past Claude Code sessions (chat transcripts) and ingests them into long-term memory as knowledge_type="experiment". It runs on a Haiku model to keep costs down, and when reprocessing the same session, it compares against the existing entry via the session_id/updated-at tags — skipping if unchanged, replacing if changed (see the previous section). The caller must explicitly specify which sessions to target; it never targets all sessions without limit.

Usage ③ Directly as a Python library

You can also call it directly from Python code without going through the MCP server.

from hybrid_rag import HybridRAGSystem

rag = HybridRAGSystem(db_path="hybrid_rag.db", index_path="indices")

rag.ingest_documents(
    ["design_doc.md"],
    metadata={"knowledge_type": "principle", "importance": 0.9, "tags": ["mbse"]},
)

result = rag.query(
    "about consistency between requirements and architecture",
    top_k=5,
    enable_memory_rerank=True,   # enable the memory mechanism's staged reranking
    memory_time_weight=0.5,
    memory_freq_weight=0.1,
    memory_importance_threshold=0.3,
)
print(result["context"])

# assign an importance tag after the fact (no vector rebuild needed)
rag.set_chunk_tags(doc_id="design_doc_xxxx", chunk_index=0, importance=0.9)

# forgetting/archival batch (normally run infrequently)
report = rag.run_forgetting_batch(score_threshold=0.05, dry_run=True)

Running the forgetting batch from the CLI

A script intended for infrequent batch execution — e.g., on a 3-month cycle or when a new model is released (it is never run automatically inside the server).

python scripts/run_forgetting_batch.py --dry-run
python scripts/run_forgetting_batch.py --score-threshold 0.1 --time-weight 0.8

Main options: --db-path --index-path --archive-path --time-weight --freq-weight --importance-threshold --score-threshold --dry-run

Archived chunks are evacuated to archive/chunks_archive.jsonl (raw text + metadata + score + deletion reason + deletion timestamp), and their vector representations are discarded.

Automatically evaluating retrieval accuracy from the CLI

A script that measures retrieval accuracy against a golden query set (Precision@k/Recall@k/MRR/NDCG@k/Hit Rate@k, authority-document rank, noise rate) in a reproducible way, instead of relying on manual queries and eyeballing results via Cursor/Claude Code.

cp eval/golden_queries.example.yaml eval/golden_queries.yaml  # once, at first use — rewrite the doc_ids for your own corpus
python scripts/run_evaluation.py --db-path mcp_server/hybrid_rag.db --index-path mcp_server/hybrid_rag_indices

Main options: --db-path --index-path --golden-set (default eval/golden_queries.yaml) --k-values (default 1,3,5,10) --authority-window (default 20) --output

Auditing near-duplicate ingests

ingest's duplicate detection cannot catch cases where identical content enters via a different file (a different path/filename — see "Preventing duplicates when re-registering the same source" above). This script just lists near-duplicates that have already made it into an existing corpus. It never deletes anything.

python scripts/find_near_duplicates.py --db-path mcp_server/hybrid_rag.db
python scripts/find_near_duplicates.py --db-path mcp_server/hybrid_rag.db --output eval/duplicates_report.json

It groups documents whose normalized-content hash (documents.content_hash) matches. Deciding which one to keep — and whether to delete anything — is left to the user; call delete_document(doc_id, rebuild_index=False) manually (and be sure to call rebuild_index at the end of the batch).

eval/golden_queries.yaml is .gitignored, since it is personal data containing doc_ids specific to your actual corpus. Reports are written to eval/eval_report_<date>.md (plus a same-named .json), which are likewise .gitignored (they remain on your machine for ongoing tracking).

Memory-mechanism fields

Fields carried in the metadata of ingest/the Python API, or per chunk:

Field	Type	Description
knowledge_type	str	principle / paper / news / experiment. Determined deterministically from the ingestion source
importance	float (0.0–1.0)	Importance assigned after the fact by the agent. Unset (None) always passes the cutoff
tags	list[str]	Arbitrary tags. Used to narrow results via hybrid_search's tags argument
access_count	int	Access frequency. Auto-incremented each time a chunk is actually returned by a query
last_accessed_at / created_at	str	Last-accessed / created timestamps. Serve as the basis for time decay

Tests

pytest tests/ -v

• test_metadata_pipeline.py: a regression test that knowledge_type/importance/tags survive the ingest → build_index → query pipeline
• test_memory_scoring.py: unit tests for staged reranking (cutoff, decay, frequency boost)
• test_archival.py: unit tests for the forgetting/archival batch
• test_index_health.py: unit tests for index_health (index/DB consistency checking)

See File layout for the full list and role of the other test files.

Base-library functionality (common across backends)

The base RAG functionality — dense/sparse hybrid search, RRF, Cross-encoder reranking, MMR diversity selection, query expansion, caching, etc. — is common to every backend (FAISS/Qdrant/Chroma/PostgreSQL).

from hybrid_rag import create_rag_system

rag = create_rag_system(backend="faiss")   # "qdrant" / "chroma" / "postgres" are also available
rag.ingest_documents(["document1.pdf", "document2.md"])
result = rag.query("What is machine learning?", top_k=5)

Aspect	FAISS	Qdrant	ChromaDB	PostgreSQL
Filtered search	post-processing	fast (single stage)	post-processing	post-processing
Server required	no	no	no	yes
Scale	up to ~20M	up to ~50M	mid-size	large-scale
Memory mechanism (this README)	✓	✗	✗	✗

Optional installs: this repository has no pyproject.toml/setup.py, so it is not distributed in the pip install hybrid-rag[...] form. To use the Qdrant/Chroma/PostgreSQL versions, install the corresponding client library directly (pip install qdrant-client / pip install chromadb / pip install "psycopg[binary]" pgvector — all of these are already listed in requirements.txt, so pip install -r requirements.txt alone covers them).

Key additional settings (a subset of the FAISS version's HybridRAGSystem constructor arguments):

rag = HybridRAGSystem(
    dense_model="paraphrase-multilingual-MiniLM-L12-v2",
    rerank_model="BAAI/bge-reranker-v2-m3",
    max_chunk_size=512,
    index_type="hnsw",           # "flat" / "ivf" / "hnsw"
    enable_mmr=True, mmr_lambda=0.6,
    enable_cache=True, cache_ttl_seconds=3600,
    query_expander=None,          # pass a QueryExpander instance for LLM-based query expansion
    memory_half_life_overrides=None,  # override the half-life (days) per knowledge_type
    enable_guaranteed_candidates=True,  # always add principle/high-importance chunks to the candidate pool (default True)
    guaranteed_knowledge_types=None,    # defaults to ["principle"]
    guaranteed_importance_threshold=0.7,
    guaranteed_candidates_limit=50,
)

enable_guaranteed_candidates (default True) addresses a problem where knowledge_type=principle chunks (or chunks with importance>=0.7) never made it into the search candidate pool in the first place, and staged reranking could not rescue them (the "principle documents getting buried" issue reported in RAG_EVALUATION_REPORT_2026-07-30.md/RAG_精度テスト_2026-07-31.md). It works by always adding matching chunks to the candidate pool right after retrieval and letting the Cross-encoder score their relevance — it does not force them to the top. Calls to query()/hybrid_search that pass metadata_filters (filters) skip this merge.

Documentation (Sphinx) / diagrams (PlantUML)

pip install sphinx sphinx-rtd-theme
python -m sphinx -b html docs/source docs/build

docs/uml/ is intended to hold PlantUML sources for class diagrams, sequence diagrams, and state-machine diagrams (not yet populated as of this writing).

File layout

hybrid_rag_agent_spec.md   # design spec for the memory mechanism
.mcp.json                  # MCP server registration for Claude Code
.claude/agents/rag-memory.md  # sub-agent definition for Claude Code

mcp_server/
└── server.py              # the MCP server itself (13 tools, see the table above)

scripts/
├── run_forgetting_batch.py       # CLI for the forgetting/archival batch
├── run_evaluation.py             # CLI that automatically evaluates retrieval accuracy against a golden query set
├── find_near_duplicates.py       # CLI that audits near-duplicate ingests in the existing corpus (report-only, never deletes)
├── backfill_source_date.py       # bulk-backfills source_date on existing chunks
├── list_md_files.py              # lists candidate Markdown files for ingestion
├── manage_ingest_status.py       # tracks ingest progress against list_md_files.py's listing
├── manage_conv_ingest_status.py  # tracks ingest progress against convert_conversations.py's output
└── convert_conversations.py      # converts a Claude.ai export (JSON) into Markdown

hybrid_rag/
├── __init__.py
├── ingestion.py            # document processing
├── chunking.py             # semantic chunking
├── indexing.py             # dense & sparse index (FAISS)
├── indexing_bm25.py        # BM25 index
├── indexing_sparse_tfidf.py  # TF-IDF sparse index (shared by the Chroma/Postgres/Qdrant backends)
├── indexing_qdrant.py / indexing_chroma.py / indexing_postgres.py
├── retrieval.py            # RRF search
├── reranking.py            # Cross-encoder reranking (relevance axis)
├── memory_scoring.py        # staged reranking (memory axis: importance/decay/frequency)
├── archival.py              # forgetting/archival batch processing
├── index_health.py          # index/DB consistency checking (backs the ⑫ index_health tool)
├── caching.py / embedding_cache.py
├── context.py / diversity.py / evaluation.py
├── storage.py               # SQLite database (including memory-mechanism fields)
├── query_expansion.py
├── rag_system.py            # main orchestrator (FAISS version, implements the memory mechanism)
├── _rag_system_indexing.py  # ^ ingest/build/incremental-update/load (mixin)
├── _rag_system_query.py     # ^ query pipeline (mixin)
├── _rag_system_memory.py    # ^ tags/neighboring chunks/forgetting batch (mixin)
├── _rag_system_stats.py     # ^ stats & cache management (mixin)
├── _rag_system_docops.py    # ^ embedding/delete/lightweight search (mixin)
├── rag_system_base.py       # base class shared by the Chroma/Postgres/Qdrant backends
├── rag_system_qdrant.py / rag_system_chroma.py / rag_system_postgres.py
└── rag_system_factory.py

tests/
├── test_metadata_pipeline.py   # metadata regression test across ingest → build_index → query
├── test_memory_scoring.py      # unit tests for staged reranking
├── test_archival.py            # unit tests for the forgetting/archival batch
├── test_incremental_index.py   # unit/integration tests for update_index (incremental updates)
├── test_index_health.py        # unit tests for index_health (index/DB consistency check)
├── test_result_dedup.py        # unit tests for RRF fusion-key stability and search-result dedup
├── test_diversity.py           # unit tests for MMR diversity selection
├── test_reranking.py           # unit tests for Cross-encoder reranking stats
├── test_retriever_shutdown.py  # tests for RRFRetriever resource cleanup (thread leaks)
├── test_indexing_bm25.py       # unit tests for the BM25 index
├── test_storage_concurrency.py # unit tests for concurrent SQLite writes
├── test_source_date.py         # unit tests for source_date derivation (time-decay reference point)
├── test_document_chunks.py     # unit tests for get_document_chunks (fetching neighboring chunks)
├── test_evaluation.py          # unit tests for RAGEvaluator (Precision@k, etc.)
├── test_database_stats.py      # unit tests for get_database_stats / duplicate-ingest detection
├── test_guaranteed_candidates.py  # unit tests for guaranteed candidate-pool merging (the fix for principle burial)
├── test_rag_system_factory.py  # unit tests for create_rag_system (backend switching)
└── conftest.py                 # shared pytest configuration

License

MIT License