MarketDocs India

MarketDocs India is an MIT-licensed, agent-first metadata index for NSE/BSE company documents. It indexes a rolling five years of concall transcripts, annual reports, and investor presentations, deduplicates companies by ISIN, and exposes provenance through server-rendered HTML, REST, and a stateless Streamable HTTP MCP server. This edition never stores or republishes source documents.

The Worker is deployed as a D1-only starter index. Its initial seed contains three verified company identities and two source-only records; it makes no claim of complete market coverage. Expanding to the requested ~5,000-company / ~70,000-document five-year index requires a licensed exchange dataset, issuer-approved feeds, or a user-supplied registry—none of those records are fabricated by this project.

What is implemented

• Cloudflare Worker serving SSR HTML, REST v1, OpenAPI, MCP, llms.txt, llms-full.txt, robots, and partitionable sitemaps.
• D1 schema for companies, aliases, source health, documents, revisions, tombstones, and takedown audits.
• D1-only metadata records with official source URLs, provenance, quality, and rights state.
• SQLite Durable Object daily anonymous rate limiter with tighter content/MCP budgets.
• ISIN-first company identity, document classification, fiscal-period parsing, checksum deduplication, revisions, and rolling five-year retention.
• Issuer RSS, sitemap, and investor-relations adapters with conditional HTTP support. Exchange collection is hard-disabled; licensed NSE/BSE interfaces are explicit extension points.
• Public GitHub Actions boundary for future source discovery and validation; the live edition writes metadata only to D1.
• Deployment guard that refuses to run unless the supplied Cloudflare account ID exactly matches the separately supplied expected account ID.
• Provenance-first, no-JavaScript UI with explicit partial coverage, quality, rights, source, checksum, extraction, and freshness.

Local development

Requirements: Node.js 22+ and Python 3.11+.

npm install
npm run db:migrate:local
npm run db:seed:local
npm run dev

Then open http://localhost:8787. The seed is metadata-only: it does not republish issuer documents.

Run all Worker/domain tests:

npm run check

Run ingestion tests without installing Docling:

python -m venv ingest/.venv
ingest/.venv/bin/python -m pip install -e './ingest[test]'
ingest/.venv/bin/python -m pytest ingest/tests

Docling conversion is installed in public CI with ./ingest[conversion,test]. Source PDFs live only in a temporary directory during validation/conversion.

Public surfaces

Surface	Route
Search/latest	/
Company	/companies/:slug
Document provenance	/companies/:slug/documents/:id
Canonical Markdown	/documents/:id.md
Coverage/freshness	/coverage
Rights/takedowns	/rights
REST	/api/v1/*
OpenAPI/docs	/openapi.json, /api/docs
Remote MCP	/mcp
Agent guides	/llms.txt, /llms-full.txt

MCP tools are search_companies, find_documents, get_document, and get_coverage. get_document returns bounded sections with continuation cursors; resource templates expose company and document provenance records.

Source and rights policy

Do not scrape NSE or BSE websites. Their public sites are not production inputs. Add only reviewed issuer-controlled feeds/pages to registry/sources.json, or implement the LicensedExchangeAdapter boundary after obtaining suitable rights.

Full text can become public only when rights_status is cleared and quality is provisional or verified. Other records are source-only. Revocation requires immediate R2 deletion, a source_only transition, and a takedown audit entry.

The software is MIT-licensed. Source documents remain subject to their owners’ terms. See /rights in the app.

Real-time ingestion

“Real-time” begins when an issuer publishes a document. Hourly Cloudflare Cron dispatches the public GitHub workflow; GitHub’s hourly schedule is the fallback. The workflow reads the reviewed issuer registry, discovers supported links, validates and hashes PDFs, publishes metadata for all rights states, and publishes Markdown only for cleared records. Configured live sources target p90 provisional availability within 60 minutes; /coverage reports actual freshness.

The workflow promotes provisional text immediately after the fast text-layer pass, then lets Docling replace it with verified output after layout/OCR checks. Protected GitHub production environments keep both publication steps behind the separate-account secrets.

Separate Cloudflare account deployment

Verify the active Wrangler account before deployment. Create the D1 resource in the separate account, replace placeholder IDs/names in wrangler.jsonc, configure GitHub’s protected production environment, and use a narrowly scoped API token for automation.

export CLOUDFLARE_ACCOUNT_ID="separate-account-id"
export EXPECTED_CLOUDFLARE_ACCOUNT_ID="separate-account-id"
export CLOUDFLARE_API_TOKEN="scoped-token"
npm run deploy

The guard fails closed if any value is absent or the IDs differ. Initially retain the configurable PUBLIC_ORIGIN workers.dev URL. After buying a domain, update PUBLIC_ORIGIN; canonical links, OpenAPI, MCP examples, sitemaps, and agent files derive from it.

Free-tier budgets

• Warn at 350 MB D1, below 500 MB per database.
• Store metadata and source URLs only; do not retain source files.
• Bound API/MCP pages and apply daily hashed-IP rate limits.
• Keep the public source registry and coverage denominator honest as issuer permissions expand.

Relevant primary documentation: Workers limits, R2 pricing, D1 limits, Cloudflare remote MCP, GitHub Actions billing, NSE terms, and Docling.