mcp-web-tools-server

A custom Model Context Protocol (MCP) server that gives an AI agent real, working tools for fetching web pages and extracting content from them: clean readable article text, structured data by CSS selector, and a robots.txt permission check.

What is MCP?

MCP is an open protocol, originally published by Anthropic, that standardizes how AI applications (like Claude Desktop or Claude Code) connect to external tools and data sources. Instead of every AI app inventing its own plugin format, an MCP server exposes a fixed set of tools (and optionally resources and prompts) over a simple JSON-RPC interface, and any MCP-compatible client can discover and call them the same way. This repo is one such server: it runs as a small local process and speaks MCP over stdio, so any MCP client can list its tools and call them without knowing anything about httpx, selectolax, or trafilatura underneath.

Why this is useful

Out of the box, an LLM cannot fetch a live web page. This server plugs that gap with a small, tested, well-scoped toolset: an agent can pull the readable text out of an article, pull specific fields out of a page by CSS selector (price, title, tags, whatever the page structure is), and check whether a site's robots.txt allows the request before making it. It is deliberately narrow rather than a general-purpose scraping framework, on the theory that a few tools that work correctly and fail predictably are more useful to an agent than a large surface area that sometimes doesn't.

Tools

fetch_and_extract(url: str) -> str

Fetches a URL and returns clean, readable main-content text: scripts, styles, nav, ads, and footers stripped out. Uses trafilatura for article extraction, with a paragraph-density heuristic (built on selectolax) as a fallback for pages trafilatura doesn't confidently handle.

extract_structured(url: str, css_selectors: dict) -> dict

Fetches a URL and extracts fields by CSS selector, e.g.:

{"title": "h1", "price": ".price", "tags": ".tag-list a"}

returns:

{"title": "Trail Blazer 29 Mountain Bike", "price": "$1,249.00", "tags": ["mountain", "hardtail", "29er"]}

A selector matching one element returns its text, matching several returns a list of their texts, matching none returns null. Parsing is done with selectolax.

check_robots_txt(url: str) -> dict

Fetches the target site's robots.txt and reports whether this server's user agent is allowed to request the given URL, using Python's standard urllib.robotparser. If no robots.txt is found, it reports that explicitly (robots_txt_found: false) rather than silently assuming permission was actually granted.

This exists because scraping etiquette should be a first-class concern, not an afterthought: an agent (or the person driving it) should be able to check permission before fetching, not just when things go wrong.

Design principles

• Honest identification. Requests use a real User-Agent string identifying this tool and linking back to this repo, not a spoofed browser UA.
• Bounded requests. Every fetch has a fixed timeout (10 seconds by default) so a slow or hanging server can't stall the whole session.
• robots.txt is a tool, not silently enforced. check_robots_txt is provided so an agent (or the person driving it) can check permission before scraping, but it does not currently block fetch_and_extract or extract_structured automatically. See Limitations below.
• No crashing on bad input. Network failures, timeouts, and invalid URLs are caught and returned as clean error text or an {"error": ...} dict, never an unhandled exception that kills the server process.

Project layout

mcp_web_tools/
  server.py        MCP server definition and the three tool entry points
  extractors.py     Pure HTML-parsing logic (no network), used for readable-text and CSS-selector extraction
  robots.py         robots.txt fetching and permission checking
  http_client.py    Shared httpx fetch helper: user agent, timeout, error handling
tests/
  test_extractors.py    Unit tests against local HTML fixtures, no network
  test_robots.py        Unit tests with the network call mocked out
  test_http_client.py   Unit tests for URL validation
  test_integration.py   Integration tests against live public sites, marked and run separately
  fixtures/              Static HTML used by the unit tests
scripts/
  test_client.py    Standalone script that launches the server and talks real MCP protocol to it

Running the server

python -m venv venv
# Windows
venv\Scripts\activate
# macOS/Linux
source venv/bin/activate

pip install -r requirements.txt
python -m mcp_web_tools.server

The server communicates over stdio using the MCP protocol; running it directly from a terminal will just sit there waiting for a client to connect. Use it through an MCP client (see below) or the included test client script.

Configuring it as an MCP server in Claude Desktop or Claude Code

Add an entry to your MCP client's server config, pointing command at the venv's Python interpreter and args at the module. For Claude Desktop, this goes in claude_desktop_config.json:

{
  "mcpServers": {
    "web-tools": {
      "command": "C:\\path\\to\\mcp-web-tools-server\\venv\\Scripts\\python.exe",
      "args": ["-m", "mcp_web_tools.server"],
      "cwd": "C:\\path\\to\\mcp-web-tools-server"
    }
  }
}

On macOS/Linux, command would be /path/to/mcp-web-tools-server/venv/bin/python.

For Claude Code, run:

claude mcp add web-tools -- /path/to/mcp-web-tools-server/venv/bin/python -m mcp_web_tools.server

(substitute the Windows venv path if applicable), or add the equivalent entry to your project's .mcp.json.

Running the tests

Unit tests run against local HTML fixtures and do not touch the network:

pytest

Integration tests hit real, stable, public test sites (example.com and books.toscrape.com, standard public scraping test/demo targets) and are excluded from the default run. Run them explicitly when you have network access:

pytest -m integration

There is also a standalone script that launches the server as a real subprocess and drives it through the actual MCP client/server protocol, rather than calling the Python functions directly:

python scripts/test_client.py

Limitations and what I'd add next

• No JavaScript rendering. This server fetches raw HTML with httpx. Pages that render their content client-side (heavy React/Vue SPAs) will return little or nothing useful. A fourth tool wrapping Playwright for a headless-browser fetch would be the natural next addition, at the cost of being much heavier to run.
• No rate limiting. Each tool call makes one request when called. There is no built-in per-domain throttling or request queue if an agent calls the tools in a tight loop against the same host. check_robots_txt does surface crawl_delay_seconds when a site publishes one, but nothing currently enforces it.
• robots.txt is advisory, not enforced. fetch_and_extract and extract_structured do not automatically consult check_robots_txt before fetching. That's a deliberate scope decision for this version (an agent should call check_robots_txt itself first), but a stricter mode that refuses disallowed fetches automatically would be a reasonable addition.
• Readability heuristic is basic. The selectolax fallback used when trafilatura doesn't produce a confident result is a simple paragraph-density scorer. It is good enough for typical article and blog layouts, but will do worse on unusual page structures than a purpose-built readability library.
• No caching. Every call re-fetches, even for the same URL moments apart. Fine for a demo/portfolio server, not ideal for heavier use.
• Single transport in practice. The server is set up for stdio, which is what Claude Desktop and Claude Code use. The mcp SDK also supports SSE and streamable-HTTP transports; wiring one of those up would be needed to run this as a hosted service rather than a local subprocess.