<div align="center">

mcp-open-webresearch

</div>

Proxy-aware Model Context Protocol (MCP) server for web searching and content extraction.

Designed to be robust and compatible with various network environments, including those using SOCKS and HTTP proxies.

Features

• Dynamic Engine Discovery: Engines are loaded dynamically from the src/infrastructure/search/ directory. Adding a new engine requires only a new folder and file, without modifying core logic.
• Multi-Engine Search: Aggregates results from Bing, DuckDuckGo, and Brave.
• Deep Research (search_deep): Recursive research agent that performs multi-round searching, citation extraction, and answer synthesis.
• Ephemeral Downloads: In-memory storage for Deep Search reports using a 100MB bounded LRU cache with 10-minute auto-expiration.
• Centralized Throttling: Rate limit management (search and pagination cooldowns) across prioritized engines.
• Smart Fetch: Configurable fetching utility (impit) with two operational profiles:
  • Browser Mode: Includes modern browser headers (User-Agent, Client Hints) for compatibility with sites requiring browser-standard requests.
  • Standard Mode: Uses a minimal HTTP client profile for environments where browser-like identification is not required.
• Result Sampling: Optional LLM-based filtering to assess result relevance.
• Content Extraction: Webpage visiting and markdown extraction tool (visit_webpage) using a headless browser.
• Proxy Support: Full support for SOCKS5, HTTPS, and HTTP proxies.
• Configuration: Configurable via environment variables and CLI arguments.
• Deployment: Docker images available for production and testing.

---

Credits

This project includes work from the following contributors:

• Manav Kundra: Initial implementation of the server.
• Aasee: Added multiple search engines and Docker support.
• mzxrai: Core logic for the visit_page tool.

---

Installation & Quick Start

Docker (Recommended)

Latest Stable Release:

docker pull ghcr.io/rinaldowouterson/mcp-open-webresearch:latest
docker run -p 3000:3000 ghcr.io/rinaldowouterson/mcp-open-webresearch:latest

Test/Debug Image:

docker pull ghcr.io/rinaldowouterson/mcp-open-webresearch:test

Local Installation

To run the server locally (e.g., in Claude Desktop or Cline):

[!NOTE] Replace /absolute/path/to/project with your actual project path.

Configuration (mcp_config.json):

{
  "mcpServers": {
    "open-webresearch": {
      "command": "npm",
      "args": [
        "run",
        "start:sampling",
        "--silent",
        "--prefix",
        "/absolute/path/to/project"
      ],
      "headers": {},
      "disabled": false
    }
  }
}

Remote Server (Streamable HTTP)

Endpoint: http://localhost:3000/mcp

Configuration:

{
  "mcpServers": {
    "open-webresearch": {
      "serverUrl": "http://localhost:3000/mcp",
      "headers": {}
    }
  }
}

---

Client Configuration & Timeouts

Deep Search processes can take several minutes to complete. Some MCP clients (like Cline and RooCode) have a default timeout of 60 seconds, which will cause the operation to fail.

You MUST configure a higher timeout in your client settings.

Cline (cline_mcp_settings.json)

Add the "timeout" parameter (in seconds). Recommended: 1800 (30 minutes).

{
  "mcpServers": {
    "open-webresearch": {
      "disabled": false,
      "timeout": 1800,
      "type": "stdio",
      "command": "npm",
      "args": [
        "run",
        "start:sampling",
        "--silent",
        "--prefix",
        "/absolute/path/to/mcp-open-webresearch"
      ],
      "autoApprove": []
    }
  }
}

RooCode (mcp_settings.json)

RooCode also respects the timeout parameter.

{
  "mcpServers": {
    "open-webresearch": {
      "disabled": false,
      "timeout": 1800,
      "command": "npm",
      "args": [
        "run",
        "start:sampling",
        "--silent",
        "--prefix",
        "/absolute/path/to/mcp-open-webresearch"
      ],
      "alwaysAllow": []
    }
  }
}

Antigravity / Windsurf (mcp_config.json)

Antigravity / Windsurf handles long-running tools natively, but if they let you configure a timeout, it is best practice to do so.

{
  "mcpServers": {
    "open-webresearch": {
      "command": "npm",
      "args": [
        "run",
        "start:sampling",
        "--silent",
        "--prefix",
        "/absolute/path/to/mcp-open-webresearch"
      ],
      "disabled": false
    }
  }
}

---

Developer Guide: Adding New Engines

To add a new search engine:

1. Create Directory: src/infrastructure/search/{engine_name}/

2. Implement Logic: Create {engine_name}.ts with the fetching/parsing logic.

3. Export Interface: Create index.ts exporting the SearchEngine interface:

   import type { SearchEngine } from "../../../types/search.js";
   import { searchMyEngine } from "./my_engine.js";
   import { isThrottled } from "../../throttle.js"; // Optional
   
   export const engine: SearchEngine = {
     name: "my_engine",
     search: searchMyEngine,
     isRateLimited: () => isThrottled("my_engine"),
   };
   

4. Restart: The server will automatically discover and load the new engine.

---

Build and Run

Locally

# 1. Clone
git clone https://github.com/rinaldowouterson/mcp-open-webresearch.git
cd mcp-open-webresearch

# 2. Install
npm install

# 3. Build & Start
npm run build
npm start

Docker

# Production
docker build -t mcp-websearch .
docker run -p 3000:3000 mcp-websearch

# Testing
npm run test:docker

---

Testing

Unit & E2E Tests

Uses Vitest for testing. Includes dynamic contract tests for all discovered engines.

npm test

Compliance Tests

Verifies the "Smart Fetch" behavior (User-Agent headers) usage using a local mock server.

npm run test .test/engines/smart_fetch_mode.test.ts

Infrastructure Validation

Validates Docker image builds and basic functionality.

npm run test:infrastructure

---

Available Scripts

Command	Description
npm run build	Compiles TypeScript to build/ folder.
npm run watch	Recompiles on file changes.
npm run inspector	Launches MCP inspector UI.
npm start	Runs the compiled server.
npm test	Runs local tests.
npm run test:docker	Runs tests in Docker container.
npm run test:infrastructure	Validates docker images.
npm run generate-certs	Generates self-signed certificates for testing.

---

Configuration

Configuration is managed via Environment Variables or CLI arguments.

Variable	Default	Description
PORT	3000	Server port.
PUBLIC_URL	http://localhost:port	Public URL for download links.
ENABLE_CORS	false	Enable CORS.
CORS_ORIGIN	*	Allowed CORS origin.
DEFAULT_SEARCH_ENGINES	bing,duckduckgo,brave	Default engines list.
ENABLE_PROXY	false	Enable proxy support.
HTTP_PROXY	-	HTTP Proxy URL.
HTTPS_PROXY	-	HTTPS Proxy URL.
SOCKS5_PROXY	-	SOCKS5 Proxy URL (Highest Priority).
SAMPLING	false	Enable result sampling.
SKIP_IDE_SAMPLING	false	Prefer external API over IDE.
LLM_BASE_URL	-	External LLM API base URL.
LLM_API_KEY	-	External LLM API key.
LLM_NAME	-	External LLM model name.
LLM_TIMEOUT_MS	30000	Timeout for external LLM calls.
DEEP_SEARCH_MAX_LOOPS	20	Max research iterations.
DEEP_SEARCH_RESULTS_PER_ENGINE	5	Results per engine per round.
DEEP_SEARCH_SATURATION_THRESHOLD	0.6	Threshold to stop research early.
DEEP_SEARCH_MAX_CITATION_URLS	10	Max URLs to visit for citations.
DEEP_SEARCH_REPORT_RETENTION_MINUTES	10	Download expiration time (minutes).
WRITE_DEBUG_TERMINAL	false	Log debug output to stdout.
WRITE_DEBUG_FILE	false	Log debug output to file.

CLI Arguments

CLI arguments override environment variables.

Argument	Description
--port <number>	Port to listen on.
--debug	Enable debug logging (stdout).
--debug-file	Enable debug logging (file).
--cors	Enable CORS.
--proxy <url>	Proxy URL (http, https, socks5).
--engines <items>	Comma-separated list of engines.
--sampling	Enable sampling.
--no-sampling	Disable sampling.

---

Search Pipeline & Scoring

The server uses a multi-stage pipeline to aggregate and refine search results:

1. Multi-Engine Retrieval

Concurrent requests are dispatched to all configured engines (Bing, Brave, DuckDuckGo). Raw results are collected into a single pool.

2. Consensus Scoring & Deduplication

Results are grouped by their canonical URL (protocol/www-agnostic hash).

• Deduplication: Multiple entries for the same URL are merged.
• Scoring: A consensusScore is calculated for each unique URL:
  • Inverted Rank Sum: Sum of inverted ranks ($1/rank$) across engines. Higher placement results in a higher score.
  • Engine Boost: Multiplies the sum by the number of unique engines that identified the URL. This prioritizes multi-provider agreement.
• Sorting: The final list is sorted by the calculated consensusScore in descending order.

3. LLM Sampling (Optional)

If SAMPLING=true, the top-ranked results are sent to an LLM to evaluate semantic relevance to the query.

• Filtering: Sampling acts as a binary filter. It removes results identified as irrelevant (spam, off-topic).
• Final Set: The original consensus scores are preserved. Only the composition of the list changes.

---

LLM Sampling Strategy

When sampling is enabled, the server follows a tiered resolution logic to select which LLM to use:

SKIP_IDE_SAMPLING	IDE Available	API Configured	Resolution
false (default)	✅	✅	IDE Sampling
true	✅	❌	IDE Sampling
false	❌	✅	External API
true	✅ OR ❌	✅	External API
false OR true	❌	❌	No Sampling

[!TIP] You can use a model without API key, the LLM_API_KEY value is optional.

[!IMPORTANT] Deep Search Compatibility: The search_deep tool strictly requires LLM capability (either via IDE or API). If neither is available, the tool will appear in the MCP list but will throw an error upon execution.

---

Tools Documentation

search_deep

Recursive research agent for deep investigation. Searches multiple sources, extracts citations, and synthesizes a comprehensive answer.

Requires LLM Sampling capability.

Input:

{
  "objective": "Deep research goal",
  "max_loops": 3,
  "results_per_engine": 5,
  "max_citation_urls": 10,
  "engines": ["bing", "brave"],
  "attach_context": false
}

Output: A structured Markdown report including a reference list. If configured, a Download URL at the top of the output permits downloading the results as a file.

search_web

Performs a search across configured engines.

Input:

{
  "query": "search query",
  "max_results": 10,
  "engines": ["bing", "brave"],
  "sampling": true
}

visit_webpage

Visits a URL and returns markdown content.

Input:

{
  "url": "https://example.com/article",
  "capture_screenshot": false
}

set_engines

Updates default search engines.

Input:

{
  "engines": ["duckduckgo", "brave"]
}

get_engines

Returns configured search engines.

set_sampling

Enables or disables result sampling.

Input:

{
  "enabled": true
}

get_sampling

Returns current sampling status.

---

📥 Ephemeral Downloads

Deep Search results are served via an in-memory buffer cache.

• Storage: Reports are stored as Buffer objects in the C++ heap to avoid V8 string memory limits.
• Expiration: Each individual entry expires exactly 10 minutes after creation. Access operations (get) do not extend the time-to-live (TTL).
• Memory Safety: The cache is bounded by a 100MB ceiling. When the limit is reached, a Least Recently Used (LRU) eviction policy removes the oldest entries.
• URL Configuration: Link generation depends on the PUBLIC_URL variable to ensure accessible download endpoints in proxied environments.

---

Roadmap

• [x] Deep Search: Recursive research and synthesis engine.
• [ ] Keyless GitHub Adapter: Implement adapter for GitHub content access.

---

License

Apache License 2.0. See LICENSE.