Yet another web scraper MCP

A minimal MCP server that lets an LLM drive a real Chromium browser to scrape websites, collect structured rows, and export them to CSV.

Setup

uv sync
uv run playwright install chromium

Running

uv run main.py

This starts the MCP server over stdio. Point your MCP client (e.g. Claude Desktop / Claude Code) at uv run --directory <this folder> main.py.

Installing via npm (recommended for end users)

No need to clone this repo. The npm package still runs the Python implementation under the hood via uv (which it will tell you how to install if it's missing), but gives a familiar one-line install/run command that works the same on Windows, macOS, and Linux:

npx -y yet-another-web-scraper-mcp

Add it to your MCP client config, e.g. Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}

On first run, npm install triggers a postinstall step that runs uv sync and downloads the Chromium browser for Playwright. This can take a minute the very first time.

Requires uv and Python 3.13+ to be resolvable on the machine; npm alone cannot provide the Python runtime.

Adding it to CLI coding agents

Claude Code

claude mcp add web-scraper -- npx -y yet-another-web-scraper-mcp

Codex CLI: add to ~/.codex/config.toml:

[mcp_servers.web-scraper]
command = "npx"
args = ["-y", "yet-another-web-scraper-mcp"]

Gemini CLI: add to ~/.gemini/settings.json (or a project's .gemini/settings.json):

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}

Cursor: add to .cursor/mcp.json (project-level) or ~/.cursor/mcp.json (global):

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}

Windsurf: same mcpServers JSON shape as Cursor above, in Windsurf's MCP config panel or ~/.codeium/windsurf/mcp_config.json.

Any other MCP-compatible client that takes a raw command/args pair (stdio transport) can point at the same npx -y yet-another-web-scraper-mcp command; the JSON snippets above are all equivalent to that one line.

Tools

• open_browser(headless=True) / close_browser(): start/stop the browser.
• navigate(url): go to a URL, returns cleaned page content.
• click(selector): click a CSS-selected element (e.g. a "next page" button), returns the resulting cleaned content.
• get_page_content(max_length): re-read the current page's HTML, with <script> and <svg> tags stripped out since they're noise for an LLM.
• get_page_links(): list {text, href} for links on the page.
• add_record(record): append one scraped row (dict of column -> value).
• get_records(): review everything collected so far.
• export_csv(filename): write the collected rows to a CSV via pandas.

Typical flow: open_browser -> navigate -> get_page_content / get_page_links -> click to page through results -> add_record per item -> export_csv when done -> close_browser.