remove-paywall-mcp

MCP server that removes article paywalls by searching internet archives. Give it a URL, get back the article text.

How it works

1. You give it a paywalled article URL
2. Tracking params are stripped and the URL is normalized
3. It tries multiple approaches in parallel:
   • Direct fetch with Googlebot user-agent (FT, WSJ, and many others serve full content to crawlers)
   • 12ft.io proxy for hard paywalls
   • iitty textise for plain-text rendering
   • Wayback Machine CDX API, archive.ph/is mirrors, and Wayback Availability API
4. It extracts the article body with readability-lxml, stripping navigation, ads, and sidebar cruft
5. Post-extraction check: if the result still contains paywall text (e.g., the snapshot captured the paywall itself), it retries with the next archive
6. It returns clean text with the title and snapshot URL

It also learns from every attempt — success rates per domain per archive source are tracked in a local SQLite database, and archive search order is re-ranked automatically using Laplace smoothing.

Install

# zero-install (recommended — works everywhere uvx is available)
uvx remove-paywall-mcp

# from PyPI
pip install remove-paywall-mcp

# from source
pip install git+https://github.com/jasval/remove-paywall-mcp.git

# Docker
docker run -i --rm remove-paywall-mcp
docker compose up -d  # HTTP mode on port 8000

Platform configs

Once installed, add this to your MCP client config:

OpenCode

{
  "mcp": {
    "remove-paywall": {
      "type": "local",
      "command": ["uvx", "remove-paywall-mcp"],
      "enabled": true
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "remove-paywall": {
      "command": "uvx",
      "args": ["remove-paywall-mcp"]
    }
  }
}

LiteLLM

mcp_tools:
  remove_paywall:
    type: "stdio"
    command: "uvx"
    args: ["remove-paywall-mcp"]

Docker (any client)

{"command": "docker", "args": ["run", "-i", "--rm", "remove-paywall-mcp"]}

Tools

remove_paywall

Main tool. Removes a paywall from an article URL and returns clean article text.

Parameter	Type	Description
url	string	The paywalled article URL

search_archives

Search all archive sources for snapshots without extracting content. Useful to see what's available.

Parameter	Type	Description
url	string	The article URL to search for

get_from_archive

Fetch from a specific archive source.

Parameter	Type	Description
url	string	The article URL
source	string	googlebot, 12ft, iitty, wayback, archive_is, or wayback_available

domain_info

Look up a domain in the knowledge base — paywall status, notes, and per-archive success rates.

Parameter	Type	Description
domain	string	Domain name (e.g. nytimes.com)

add_domain

Register a domain in the knowledge base. Mark paywalled domains so archives are searched first, or non-paywalled domains so the live page is fetched directly.

Parameter	Type	Description
domain	string	Domain name
has_paywall	boolean	true if the site has a paywall
notes	string?	Optional description

Prompts

The server provides 3 prompt templates for LLMs to use the tools effectively.

remove_paywall_prompt

Full instruct for bypassing a specific URL. Tells the assistant to use remove_paywall, fall back to search_archives, and check domain_info.

Parameter	Type	Description
url	string	The paywalled article URL

bypass_paywall

Short alias — just tells the assistant to call remove_paywall on the URL.

Parameter	Type	Description
url	string	The paywalled article URL

handle_paywalls

System prompt fragment. No arguments — returns instructions for the assistant to automatically call remove_paywall whenever it encounters a paywall, login wall, or metered content. Paste this into your system prompt or load it as a prompt at session start.

Domain knowledge base

Seeded with 33 well-known paywalled domains (NYT, WSJ, Bloomberg, Medium, etc.), stored in SQLite at ~/.remove-paywall-mcp/domains.db. Tracks every archive success/failure per domain and re-ranks archive search order automatically — domains where archive.is consistently fails won't waste time on it.

Env vars

Variable	Default	Description
MCP_TRANSPORT	stdio	stdio or streamable-http
MCP_HOST	0.0.0.0	Bind address (HTTP mode)
MCP_PORT	8000	Port (HTTP mode)
MCP_DB_DIR	~/.remove-paywall-mcp	Database directory

Archive sources

Source	Priority	Notes
Googlebot direct	1	Fetches directly with Googlebot/2.1 UA — many sites (FT, WSJ) serve full content to crawlers
12ft.io	2	Proxy at 12ft.io/proxy?q=<url> — reliable for most hard paywalls
iitty	3	textise.iitty.com — plain-text rendering, works well on FT
Wayback Machine	4	CDX API, newest-first (limit=-5), dedup via collapse=digest, HTML-only
archive.is/ph mirrors	5	Tries newest/oldest across archive.ph, archive.is, archive.today, archive.md
Wayback Availability	6	archive.org/wayback/available — single closest snapshot as fast fallback

Priority is dynamically re-ranked per domain based on historical success rates recorded in the knowledge base.

Architecture

MCP client (Claude/OpenCode/LiteLLM)
       │  stdio or HTTP
       ▼
┌─────────────────┐
│    server.py     │  MCPServer with 5 tools
│  +3 prompts       │
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐ ┌──────────┐
│archives│ │ extractors│
│  .py   │ │   .py     │
│        │ │           │
│googlebot│ │readability│
│ 12ft   │ │Beautiful  │
│ iitty  │ │Soup       │
│ wayback│ │           │
│ archive│ │           │
│ .is/ph │ │           │
│ wayback│ │           │
│ avail  │ │           │
└───┬────┘ └──────────┘
    │
    ▼
┌──────────────┐
│domain_store  │
│   .py        │
│              │
│ SQLite knows │
│ which domains│
│ have paywalls│
│ and which    │
│ archives work│
│ best (Laplace│
│ smoothed)    │
└──────────────┘

License

MIT