Googlebot Simulator MCP

An MCP server that simulates Googlebot crawling pages and captures analytics events. Use this to verify your site's bot detection and analytics tracking work correctly.

Features

• Authentic Googlebot User-Agents: Uses real Googlebot desktop and mobile user-agent strings
• Idle Detection Simulation: Mimics Googlebot's requestIdleCallback-based idle detection
• Event Interception: Captures analytics/tracking requests matching your pattern
• Detailed Reporting: Returns timing info, captured payloads, and screenshots

Installation

Add to Claude Code

claude mcp add googlebot-sim -- npx googlebot-simulator-mcp

Or add to .claude.json

{
  "mcpServers": {
    "googlebot-sim": {
      "type": "stdio",
      "command": "npx",
      "args": ["googlebot-simulator-mcp"]
    }
  }
}

From source

git clone https://github.com/RichardDillman/googlebot-simulator-mcp.git
cd googlebot-simulator-mcp
npm install
npm run build

Usage

Once added to Claude, you can use natural language:

"Test https://example.com/jobs as Googlebot, watching for requests containing 'page_viewed'"

Or be more specific:

"Simulate mobile Googlebot on https://talent.com/jobs, capture events to **/serp-event-producer/send"

Tool: simulate_googlebot

Input

{
  // Required: URL(s) to test
  urls: string | string[];

  // Required: Pattern to match analytics events
  eventPattern: {
    urlPattern?: string;    // e.g., "**/analytics/**"
    bodyContains?: string;  // e.g., "page_viewed"
  };

  // Optional configuration
  options?: {
    userAgent?: "desktop" | "mobile";  // Default: "desktop"
    idleTimeout?: number;              // Default: 5000ms
    maxWaitTime?: number;              // Default: 10000ms
    captureScreenshot?: boolean;       // Default: true
    screenshotDir?: string;            // Default: cwd
  };
}

Output

{
  results: Array<{
    url: string;
    success: boolean;              // true if matching event was captured
    eventsFired: Array<{
      timestamp: number;
      url: string;
      method: string;
      payload: object;
      matchedPattern: boolean;
      responseStatus?: number;
    }>;
    timing: {
      navigationStart: number;
      domContentLoaded: number;
      idleDetected: number;
      totalTime: number;
      firstIdleCallback?: number;
    };
    screenshot?: string;           // Path to screenshot file
    errors: string[];
    idleCallbacksObserved: number;
  }>;
  summary: {
    totalUrls: number;
    passed: number;
    failed: number;
    totalEventsCaptured: number;
  };
}

Example Output

{
  "results": [
    {
      "url": "https://talent.com/jobs",
      "success": true,
      "eventsFired": [
        {
          "timestamp": 1704825600000,
          "url": "https://events.talent.com/serp-event-producer/send",
          "method": "POST",
          "payload": {
            "event_type": "page_viewed_bot",
            "bot_type": "google",
            "page_name": "serp"
          },
          "matchedPattern": true,
          "responseStatus": 200
        }
      ],
      "timing": {
        "navigationStart": 1704825595000,
        "domContentLoaded": 1704825596500,
        "idleDetected": 1704825600000,
        "totalTime": 5000,
        "firstIdleCallback": 3200
      },
      "screenshot": "/tmp/googlebot_desktop_jobs_1704825600000.png",
      "errors": [],
      "idleCallbacksObserved": 3
    }
  ],
  "summary": {
    "totalUrls": 1,
    "passed": 1,
    "failed": 0,
    "totalEventsCaptured": 1
  }
}

How It Works

1. Launches headless Chromium with Googlebot user-agent and viewport settings
2. Injects tracking script to monitor requestIdleCallback usage
3. Sets up network interception to capture matching requests
4. Navigates to URL and waits for page to reach idle state
5. Captures screenshot at the "idle" moment (what Googlebot sees)
6. Returns detailed report with all captured events and timing

Use Cases

• Verify bot detection: Ensure your analytics correctly identifies Googlebot
• Test event timing: Confirm events fire before Googlebot's idle timeout
• Debug rendering: See what content is visible when Googlebot considers the page "done"
• Compare desktop vs mobile: Test both Googlebot variants

License

MIT