fetch-url-raw

Short description: a stateless MCP server that exposes one tool, fetch_url_raw — a lightweight programmable HTTP client for LLM agents. Send arbitrary methods, headers, and bodies; control timeouts, redirects, response size, TLS verification, and DNS overrides. No cookies or session state between calls.

Requirements

• Python 3.12+
• Network access from the host that runs the server

Install

python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

For tests:

pip install -e '.[dev]'

Deployment

The server speaks MCP over stdio. Deploy it as a local process that your MCP client launches; it does not open a public HTTP port by default.

1. Install into a venv (recommended)

cd /path/to/fetch-url-raw
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Confirm the entry point (stdio by default; waits on stdin for MCP):

fetch-url-raw
# Stop with Ctrl+C.

Or:

python -m fetch_url_raw

HTTP mode is off by default. To enable it, see section 3 below.

2. Wire into an MCP client

Point the client at the venv interpreter (or the fetch-url-raw script) so dependencies resolve correctly.

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "fetch-url-raw": {
      "command": "/path/to/fetch-url-raw/.venv/bin/python",
      "args": ["-m", "fetch_url_raw"],
      "cwd": "/path/to/fetch-url-raw"
    }
  }
}

Generic MCP host / Cursor-style config:

{
  "mcpServers": {
    "fetch-url-raw": {
      "command": "/path/to/fetch-url-raw/.venv/bin/fetch-url-raw"
    }
  }
}

Codex / other stdio MCP runners: use the same command + args pattern; no extra env is required for basic use.

3. Optional HTTP server mode (not enabled by default)

Default transport is stdio. To expose MCP over HTTP instead, pass --transport explicitly and set listen address/port:

# Streamable HTTP (recommended HTTP transport)
fetch-url-raw --transport streamable-http --host 127.0.0.1 --port 8000

# SSE transport
fetch-url-raw --transport sse --host 127.0.0.1 --port 8000

# Bind all interfaces (trusted networks only)
fetch-url-raw --transport streamable-http --host 0.0.0.0 --port 9000 --allow-remote

Flag	Default	Description
--transport	stdio	stdio, streamable-http, or sse
--host	127.0.0.1	Listen IP (HTTP transports only)
--port	8000	Listen port (HTTP transports only)
--path	/mcp or /sse	Endpoint path (streamable-http → /mcp, sse → /sse)
--stateless-http	off	FastMCP stateless HTTP mode (streamable-http only)
--allow-remote	off	Relax Host/Origin DNS-rebinding checks for non-local clients
--allow-private-network	off	Allow resolved private/local destination IPs (see below)
--log-level	INFO	Uvicorn/server log level

Endpoints:

• streamable-http: http://<host>:<port>/mcp (or custom --path)
• sse: http://<host>:<port>/sse (messages under the default FastMCP message path)

Example MCP client config against a local HTTP server (client-specific; streamable HTTP):

{
  "mcpServers": {
    "fetch-url-raw": {
      "url": "http://127.0.0.1:8000/mcp"
    }
  }
}

HTTP mode is optional. Prefer stdio for desktop/local agent integrations. Only bind 0.0.0.0 or enable --allow-remote on trusted networks; the tool can initiate arbitrary outbound HTTP.

Private / local destination blocking

By default the server blocks connections by resolved destination IP (not by DNS name):

• DNS lookup is always allowed
• After resolve (or dns_override / literal IP URL), the TCP destination must not fall in private/local ranges
• Blocked by default: RFC1918 (10/8, 172.16/12, 192.168/16), loopback, link-local (169.254/16, fe80::/10), CGNAT (100.64/10), IPv6 ULA (fc00::/7)
• Always blocked: multicast / unspecified special-use ranges
• IPv4-mapped IPv6 addresses are checked as their IPv4 form

Opt in when you intentionally need LAN/metadata/loopback targets:

fetch-url-raw --allow-private-network
# with HTTP mode:
fetch-url-raw --transport streamable-http --host 127.0.0.1 --port 8000 --allow-private-network

Blocked attempts return:

{
  "success": false,
  "error": {
    "type": "DESTINATION_BLOCKED",
    "message": "destination IP 192.168.1.1 is private/local (...)"
  }
}

4. Operational notes

Topic	Guidance
State	Stateless — safe to restart anytime; no DB or disk cache
Network	Outbound HTTP/HTTPS only; needs reachability to targets you fetch
Security	Tool can hit arbitrary URLs — run only for trusted clients; consider host firewall / network policy
Resources	Returned body is capped (max_response_bytes, default 1 MiB); up to 16 MiB may be buffered to decode large text
Proxies	System proxy env is ignored (trust_env=False) for predictable behavior
Logs	Server logs go to stderr; keep stdin/stdout for MCP framing in stdio mode
HTTP listen	Not started unless --transport streamable-http or --transport sse is set

5. Optional: install from wheel

pip install dist/fetch_url_raw-0.1.0-py3-none-any.whl
fetch-url-raw

Usage

Once the MCP server is connected, call the fetch_url_raw tool from the client.

Parameters

Parameter	Type	Default	Description
url	string	required	Absolute http or https URL
method	string	GET	HTTP method (normalized to uppercase)
headers	object	—	Request headers (string → string)
body	string | object | array | number | bool	—	Request body: raw string as-is, or JSON value (object/array/number/bool) which is serialized and gets Content-Type: application/json when unset
timeout	number	30	Timeout in seconds
follow_redirect	bool	true	Follow redirects
max_response_bytes	int	1048576	Max body bytes returned to the LLM. Internally buffers up to 16 MiB so large text can still be decoded, then truncates the returned body/body_base64
dns_override	object	—	Hostname → IP map (like curl --resolve)
verify_tls	bool	true	Verify TLS certificates
include_tls	bool	false	Include TLS metadata + peer cert(s) in the result (HTTPS)
tls_only	bool	false	Handshake only (no HTTP). Requires include_tls=true and https

Example tool calls

GET

{
  "url": "https://example.com/"
}

POST JSON (LLM-friendly: pass a JSON object directly; string body still works)

{
  "url": "https://httpbin.org/post",
  "method": "POST",
  "headers": {
    "Authorization": "Bearer token"
  },
  "body": {"hello": "world"},
  "timeout": 15
}

Raw string body (no auto Content-Type):

{
  "url": "https://httpbin.org/post",
  "method": "POST",
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  },
  "body": "hello=world&a=1"
}

No redirects + small body cap

{
  "url": "https://example.com/redirect",
  "follow_redirect": false,
  "max_response_bytes": 4096
}

DNS override (connect to 1.2.3.4 while keeping Host/SNI as api.example.com)

{
  "url": "https://api.example.com/health",
  "dns_override": {
    "api.example.com": "1.2.3.4"
  },
  "verify_tls": true
}

Inspect TLS cert only (no HTTP request)

{
  "url": "https://example.com/",
  "include_tls": true,
  "tls_only": true
}

Fetch and include TLS info

{
  "url": "https://example.com/",
  "include_tls": true,
  "max_response_bytes": 4096
}

Success response

{
  "success": true,
  "status": 200,
  "reason": "OK",
  "http_version": "HTTP/1.1",
  "headers": { "...": "..." },
  "body": "...",
  "body_json": null,
  "body_base64": null,
  "content_type": "text/html",
  "encoding": "utf-8",
  "elapsed_ms": 123,
  "redirected": false,
  "final_url": "https://example.com",
  "truncated": false,
  "received_bytes": 12345,
  "content_length": 12345
}

• Text-like content types (text/*, application/json, etc.) fill body (string still kept).
• body_json is set only when the body is valid JSON (object/array/etc.); otherwise null.
• Other non-text types set body to null and put Base64 data in body_base64.
• Internally the client may buffer up to 16 MiB so large text (e.g. JS bundles) can be decoded as text even when max_response_bytes is smaller. The tool result is then truncated to max_response_bytes.
• If the returned body is truncated, truncated is true and only the first N bytes (or text whose UTF-8 size is N) are returned.
• received_bytes is how many body bytes were actually returned to the LLM (after truncation).
• Optional tls object (when include_tls or on many TLS_ERRORs): version, cipher, ALPN, SNI, peer IP, leaf cert PEM/fingerprint/SAN/dates, and chain.
• content_length is the full response body size when known: from the Content-Length header if present, otherwise the full size if the body fit in the 16 MiB prefetch buffer, otherwise null if the stream was cut at the prefetch ceiling without a header. Use this so agents know the real size (e.g. 1 MiB JS) while reading only the first 64 KiB of text.

Error response

Failures return a structured object instead of raising:

{
  "success": false,
  "error": {
    "type": "TIMEOUT",
    "message": "Operation timed out"
  }
}

error.type	Meaning
INVALID_URL	Missing/unsupported scheme or host
INVALID_PARAMETER	Bad method, headers, timeout, etc.
DNS_ERROR	Hostname could not be resolved
TIMEOUT	Connect/read/write/pool timed out (message says which phase)
TLS_ERROR	Certificate or TLS failure (message distinguishes expired, self-signed/untrusted, hostname/SAN mismatch, incomplete chain, outdated/unsupported protocol, malformed TLS)
CONNECT_ERROR	TCP/connect failure (message distinguishes refused, reset/RST, network/host unreachable, broken pipe, etc.)
DESTINATION_BLOCKED	Resolved destination IP denied by private-network policy
PROTOCOL_ERROR	HTTP protocol / too many redirects
HTTP_ERROR / ERROR	Other HTTP or unexpected failure

Example message values:

• TIMEOUT: Connection timed out while establishing TCP/TLS, Timed out while reading the response
• CONNECT_ERROR: Connection refused (no service listening or port closed), Connection reset by peer (TCP RST during connect or request), Network is unreachable (no route to destination network), No route to host (destination host unreachable)
• TLS_ERROR: TLS certificate has expired, TLS certificate is self-signed and not trusted (...), TLS certificate hostname mismatch: ... (wrong CNAME/SAN), TLS certificate chain incomplete or untrusted: unable to get local issuer certificate (...), TLS protocol version mismatch or unsupported/outdated TLS (...), TLS handshake failed: malformed or unexpected TLS data (...)

Features

• Methods: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, TRACE
• Custom headers and body (raw string or JSON value for LLM-friendly POSTs)
• Timeout, redirect control, response size limit
• DNS override (SNI and Host header preserved)
• TLS verification toggle
• Stateless: no cookies, session cache, or filesystem writes
• Default block of private/local destination IPs (post-resolve); opt-in with --allow-private-network
• Structured errors suitable for LLM tool loops

Development

pip install -e '.[dev]'
pytest
FETCH_URL_RAW_LIVE=1 pytest tests/test_live.py   # optional live network tests

Design

See design.md for architecture, DNS override details, body decoding rules, and security considerations.

License

MIT