arXiv Research-Concept Companion

An AI/ML study-companion agent (KSE Agentic Lab assignment). It reads a concept-summary note from your Obsidian vault, finds related arXiv papers, scores each on topical relevance and age-adjusted citation impact, finds the well-established papers a surviving candidate builds on, and writes the findings back into the vault.

• Existing MCP server (Part A): Obsidian Local REST API MCP.
• Custom MCP server (Part B): custom_server/ — FastMCP app, 3 tools over the public arXiv and OpenAlex APIs (no auth).
• Agent: agent/ — a PydanticAI Agent (OpenRouter-backed) holding both MCP connections as toolsets, orchestrated by a LangGraph state machine.

Prerequisites

• Python 3.12+, uv.
• An OpenRouter API key.
• Obsidian with the Local REST API community plugin installed and running, and an MCP server that speaks to it (any Obsidian Local REST API MCP implementation — the launch command is configurable, see below).

Install

uv sync
cp .env.example .env

Fill in .env:

Variable	Meaning
OPENROUTER_API_KEY	OpenRouter key — used by the agent and by score_paper_relevance.
OPENROUTER_MODEL	Model slug, e.g. openai/gpt-4o-mini.
OBSIDIAN_API_KEY / OBSIDIAN_BASE_URL	Local REST API plugin credentials.
OBSIDIAN_MCP_COMMAND	Space-separated argv to launch your Obsidian MCP server, e.g. npx -y <obsidian-mcp-package>.
RELEVANCE_PASS_THRESHOLD	Minimum relevance score (0–1) to survive the filter. Default 0.5.
CITATIONS_PER_YEAR_THRESHOLD	Minimum citations/year to pass the impact check. Default 5.
NEW_PAPER_AGE_EXEMPT_YEARS	Papers younger than this are exempt from the impact check. Default 1.

Running

Two independent processes, sharing one uv project:

# process 1 — the custom MCP server (arXiv + OpenAlex)
uv run python -m custom_server.server

# process 2 — the agent (connects to both MCP servers), driven by a free-text prompt
uv run python -m agent.graph "Find papers related to my 'Transformers Concept Note'"

agent/graph.py spawns custom_server/server.py itself as a stdio subprocess, so process 2 does not need process 1 already running — the two commands above just demonstrate that each is independently startable.

The prompt is not a literal note title — the agent's first step (parse_prompt) uses an LLM call to identify which Obsidian note the prompt refers to. If it can't identify one, the run halts immediately and prints "Not enough information: no Obsidian note or page was named in the prompt." without touching Obsidian. If the note it finds doesn't yield enough concept keywords (fewer than min_keywords, default 2), the run halts after reading it and prints a similar "not enough information" message instead of searching arXiv.

Offline / replay mode

The custom server calls three live network APIs (arXiv, OpenAlex, OpenRouter). Setting CUSTOM_SERVER_OFFLINE=1 serves its tools from recorded fixtures in custom_server/fixtures/ instead — no network access or OPENROUTER_API_KEY required. Useful for a demo/defence without reliable network, or for fast iteration.

CUSTOM_SERVER_OFFLINE=1 uv run python -m custom_server.server

What's covered: search_arxiv_papers (one recorded search feed, served for any query — see limitation below), and score_paper_relevance / find_foundational_citations for two recorded papers, GPT-3 (2005.14165) and ResNet (1512.03385).

Known limitations:

• search_arxiv_papers is query-agnostic in offline mode — it always returns the same recorded feed regardless of the query text.
• score_paper_relevance and find_foundational_citations only recognize the two recorded papers above. An unrecorded arxiv_id raises PaperNotFoundError (the same error a real OpenAlex miss would produce); an unrecorded paper title passed to score_paper_relevance raises FixtureNotFoundError — distinguishable, not a silent wrong answer.

To regenerate or extend the fixtures: uv run python -m custom_server.fixtures.record re-fetches the recorded arXiv/OpenAlex responses (both public, unauthenticated APIs) and overwrites the JSON/XML files in custom_server/fixtures/. To add a new paper, add its two httpx.get calls to record.py and a matching entry to relevance_scores.json (hand-authored — not real OpenRouter output, since recording its raw chat-completion response isn't worth the wire-format fragility; the structured {relevance, novelty, rationale} fields are replayed directly through a PydanticAI FunctionModel).

Tests

uv run pytest custom_server/tests agent/tests

All network calls (arXiv, OpenAlex, OpenRouter) are mocked; no live traffic during tests.

Tool contracts (Part C)

search_arxiv_papers (custom)

Purpose	Primary data-source tool: search arXiv for candidate papers on a topic.
Model-facing description	"Search arXiv for papers on a topic, optionally restricted to categories and a minimum submission date. Use this to find candidate papers before evaluating them individually with score_paper_relevance. A valid query that matches nothing returns an empty list — that is a normal result, not an error."
Input	query: str, categories: list[str] = [cs.LG, cs.AI, cs.CL, stat.ML], since_date: str | None (YYYY-MM-DD), max_results: int = 10 (1–50)
Output	list[{arxiv_id, title, abstract, authors: list[str], published_date, categories: list[str]}]
Error conditions	ValueError on an invalid category code, a malformed since_date, or max_results out of [1, 50] — raised before any network call. Upstream HTTP failure raises via raise_for_status(). Zero matches is a valid empty list, not an error.
Side effects	None — read-only HTTP GET to export.arxiv.org.
Example	search_arxiv_papers(query="transformer attention", max_results=5) → 5 candidate papers with abstracts.

score_paper_relevance (custom)

Purpose	Evaluative tool: judge one candidate's topical fit and whether its citation record clears an age-adjusted bar.
Model-facing description	"Score how relevant and novel a paper is to a concept summary, and check whether its citation impact clears a minimum bar (citations per year, exempting papers younger than one year). Use this on each candidate from search_arxiv_papers to decide whether it belongs in a reading list. Raises if the paper has no OpenAlex record, or if the underlying relevance-scoring model call fails."
Input	concept_summary: str, paper: {arxiv_id, title, abstract}
Output	{relevance: float, novelty: float, citation_count: int, publication_year: int, citations_per_year: float, impact_pass: bool, rationale: str}
Error conditions	PaperNotFoundError (from custom_server.openalex) if OpenAlex has no record for the paper's arXiv DOI — distinct from a found-but-uncited paper, which is a valid citation_count: 0. UnexpectedModelBehavior if the OpenRouter call's structured output fails schema validation after retries.
Side effects	Read-only: one OpenAlex GET, one OpenRouter chat-completion call.
Example	score_paper_relevance(concept_summary="attention mechanisms in NLP", paper={...}) → {relevance: 0.92, novelty: 0.6, citation_count: 84331, impact_pass: True, ...}

find_foundational_citations (custom)

Purpose	Citation-graph analysis: given one paper, rank its own references by citation count to surface the well-established work it builds on. Distinct from search_arxiv_papers — it analyzes a specific paper's reference list, not a keyword search.
Model-facing description	"Given one paper's arXiv ID, return its most-cited references — the well-established prior work it builds on. Use this after selecting a paper to read, to surface the background literature behind it. A paper with no recorded references returns an empty list — that is a normal result, not an error."
Input	arxiv_id: str, max_results: int = 3 (1–3)
Output	list[{openalex_id, title, cited_by_count, publication_year}], sorted by cited_by_count descending, top max_results
Error conditions	ValueError if max_results outside [1, 3]. PaperNotFoundError if OpenAlex has no record for the arXiv ID. A paper with zero references returns [] — valid, not an error.
Side effects	Read-only: one OpenAlex paper lookup + one or more batched OpenAlex works lookups (chunked at 50 IDs per request).
Example	find_foundational_citations(arxiv_id="2005.14165", max_results=3) → the 3 most-cited papers GPT-3 references.

Obsidian Local REST API MCP (existing, Part A)

Used via the PydanticAI agent's natural-language tool calls (not a fixed wrapper function) for two operations in the flow:

Reference resolution	Before any Obsidian call, parse_prompt asks the PydanticAI agent (plain LLM reasoning, not an MCP call) to identify the note title implied by the user's free-text prompt. If none is identifiable, the flow halts with an "insufficient information" status and never calls Obsidian.
Read	The agent is prompted to read the note titled note_title (from parse_prompt) and return its plain-text content — feeds concept_text, the input to keyword extraction and relevance scoring.
Write	The agent is prompted to create/overwrite a note titled "{note_title} — Related Papers" with the markdown produced by compose_note_content — the observable effect that closes the loop between both MCP servers.
Error conditions	Stopped plugin, invalid API key, or a missing note surface as a distinguishable tool-call failure from the MCP server, not a silent empty result.

Design rationale

• Why Obsidian: the assignment needs an existing MCP server the agent both reads from and writes to. A student's own concept notes are a natural "what do I already know" input, and writing survivors back closes the loop visibly in the vault.
• Why arXiv + OpenAlex instead of a login-walled site: the originally considered KSE schedule/Moodle sources both require personal login, which the assignment's public-API rule rules out. arXiv and OpenAlex are public, unauthenticated, and directly support the "relevance + impact" domain.
• Why relevance is LLM-judged, not embeddings: OpenRouter has no embeddings endpoint (verified against its live model catalog), so score_paper_relevance uses a PydanticAI structured-output call instead of vector similarity — reusing the one model credential the project already needs.
• Why find_foundational_citations isn't "search again with OpenAlex": it takes one specific paper's reference list and ranks it by citation impact, the same kind of controlled indicator-comparison the assignment's own examples use — distinct responsibility and processing from the keyword-driven search_arxiv_papers.
• Filtering is plain Python, not a 4th tool: the relevance-threshold + impact_pass filter in agent/graph.py's filter_candidates_node is deterministic post-processing over already-scored data, not new domain logic — a tool would just be indirection around an if.
• Trade-offs / limitations: the custom server's offline/replay mode (see "Offline / replay mode" above) covers two recorded papers and a query-agnostic arXiv search — not a general record/replay of arbitrary queries. agent/'s own Obsidian and OpenRouter calls are unaffected by it and still require live access. Impact/relevance thresholds are .env values, not runtime-tunable per request.

Deferred (flagged, not dropped)

• Exposing the hardcoded thresholds as richer runtime config beyond .env.

Demo / defence checklist

• [ ] uv run python -m custom_server.server starts standalone; a raw MCP client's list_tools shows all 3 tools.
• [ ] uv run pytest custom_server/tests agent/tests — all green, network mocked.
• [ ] CUSTOM_SERVER_OFFLINE=1 uv run python -m custom_server.server starts and serves all 3 tool calls with no live network or API keys required (see "Offline / replay mode").
• [ ] Seed a demo vault note with a concept summary (e.g. "attention mechanisms"), titled e.g. "Transformers Concept Note".
• [ ] uv run python -m agent.graph "Find papers related to my 'Transformers Concept Note'" — full live run: resolves the note reference, reads the note, searches arXiv, scores candidates, filters, finds foundational citations, writes "<note> — Related Papers" back to the vault.
• [ ] Show both MCP connections feeding the final output: the write-back note cites both arXiv/OpenAlex data (custom server) and the original concept note content (Obsidian).
• [ ] Insufficient-information demo: run with a prompt that names no note (e.g. "What's a transformer?") — show the agent halts and prints "Not enough information..." without calling Obsidian. Then run against a note with near-empty content — show it halts after reading the note, before calling arXiv.
• [ ] Failure demo, Obsidian: stop the Local REST API plugin (or use a bad OBSIDIAN_API_KEY / a nonexistent note title) — show the agent surfaces a distinguishable error, not a silent empty result.
• [ ] Failure demo, custom server: call search_arxiv_papers with an invalid category, or find_foundational_citations with an arXiv ID absent from OpenAlex — show ValueError / PaperNotFoundError respectively, distinct from a valid empty result.