Enhanced Web Scraper MCP Server

A professional Model Context Protocol (MCP) server for web scraping, React app testing, and React Native web app inspection using Playwright. Fully backward compatible with regular websites and standard React applications.

🚀 Recent Improvements

• Enhanced Error Handling - Comprehensive input validation and error reporting
• Optimized Performance - Reduced code duplication and improved efficiency
• Standardized Timeouts - Configurable timeout constants for reliability
• Professional Code Structure - ES6+ best practices and maintainable architecture

🔄 Backward Compatibility

This enhanced server maintains 100% compatibility with:

• ✅ Regular websites (HTML, CSS, JavaScript)
• ✅ Standard React applications (Create React App, Next.js, etc.)
• ✅ Traditional web scraping workflows
• ✅ Existing CSS selectors and interactions

Plus new enhanced support for:

• 🆕 React Native web applications
• 🆕 Expo web projects
• 🆕 Mobile viewport emulation
• 🆕 Advanced React component inspection

Features

🌐 Universal Web Scraping

• Multi-browser support (Chromium, Firefox, WebKit)
• Regular CSS selectors work as expected
• Automatic fallback to React Native selectors when needed
• Mobile viewport emulation (optional)
• Device-specific emulation (iPhone, Pixel, etc.)

📱 Enhanced React Support

• Regular React apps - Works with standard React applications
• React Native web - Enhanced support for RN web components
• React hydration detection - Smart waiting for React apps to load
• Component tree analysis - Deep React component inspection
• TestID and accessibility support - React Native testing patterns
• Mobile gesture simulation - Touch interactions for mobile UX

🔧 Development Tools

• Expo development integration - Metro bundler health checks
• Framework detection - Identifies React, Expo, and React Native web
• Performance monitoring - Load times and metrics
• Debug capabilities - Enhanced error reporting

Available Tools

1. scrape_page - Universal Web Scraping

Works with any website - regular HTML, React apps, or React Native web.

Regular website example:

{
  url: "https://example.com",
  selector: ".article-title", // Standard CSS selector
  screenshot: true
}

React Native web example:

{
  url: "http://localhost:8081",
  selector: "login-button", // Will try testID, aria-label fallbacks
  mobileViewport: true,
  device: "iPhone 12"
}

2. test_react_app - Universal React Testing

Works with any React application - standard React or React Native web.

Standard React app example:

{
  url: "http://localhost:3000",
  waitForHydration: false, // Optional for regular React apps
  actions: [
    { type: "click", selector: "#submit-button" },
    { type: "fill", selector: "input[name='email']", value: "test@example.com" }
  ]
}

React Native web example:

{
  url: "http://localhost:8081",
  device: "iPhone 12",
  waitForHydration: true, // Recommended for RN web
  actions: [
    { type: "tap", selector: "login-button" },
    { type: "swipe", selector: "scroll-view", value: "up" }
  ]
}

3. get_page_info - Enhanced Page Analysis

Provides comprehensive information for any web page with React-specific insights.

{
  url: "https://any-website.com", // Works with any URL
  includePerformance: true
}

4. extract_content - Clean Content Extraction

Extract clean, readable content from web pages without HTML/CSS clutter. Perfect for documentation, articles, and structured content consumption.

{
  url: "https://docs.example.com/api-guide",
  includeLinks: true,    // Extract and categorize hyperlinks
  format: "markdown"     // Output format: 'markdown' or 'text'
}

Output Example:

# API Documentation

## Authentication
You need to obtain an API key [1] from the developer portal [2].

### Rate Limits
See the rate limiting guide [3] for details.

---
## Links Found:
[1] https://example.com/api-keys (internal)
[2] https://developer.example.com (external) 
[3] https://example.com/docs/rate-limits (internal)

Features:

• Clean Structure - Preserves headings, paragraphs, lists, code blocks
• Link Extraction - Categorizes links as internal, external, anchor, or download
• Content Filtering - Removes navigation, ads, sidebars automatically
• Multiple Formats - Markdown or plain text output

5. wait_for_element - Smart Element Waiting

Intelligent element waiting with automatic selector strategy fallbacks.

{
  url: "https://example.com",
  selector: ".loading-spinner", // CSS selector with RN fallbacks
  timeout: 10000
}

React Native Web Specific Tools

6. inspect_react_app - React Component Analysis

Deep inspection of React applications (works best with React Native web).

7. wait_for_react_state - React State Management

Wait for React-specific conditions like hydration, navigation, data loading.

8. execute_in_react_context - JavaScript Execution

Execute JavaScript in React context for advanced inspection.

9. check_expo_dev_server - Expo Development Tools

Check Expo/Metro bundler status for development workflows.

Selector Strategy Priority

The server uses intelligent selector strategies:

1. Primary: Direct CSS selector (e.g., #button, .class, input[name='email'])
2. Fallback 1: TestID attribute ([data-testid="button"])
3. Fallback 2: Accessibility label ([aria-label="Button"])
4. Fallback 3: AccessibilityLabel ([accessibilityLabel="Button"])

This ensures regular CSS selectors work normally while providing React Native web compatibility.

Usage Examples

Regular Website Scraping

// Works exactly like before
{
  url: "https://news.ycombinator.com",
  selector: ".storylink",
  screenshot: false
}

Standard React App Testing

// Standard React app (Create React App, Next.js, etc.)
{
  url: "http://localhost:3000",
  actions: [
    { type: "click", selector: "button.login" },
    { type: "fill", selector: "#username", value: "testuser" }
  ]
}

React Native Web App Testing

// React Native web with enhanced features
{
  url: "http://localhost:8081",
  device: "iPhone 12",
  waitForHydration: true,
  actions: [
    { type: "tap", selector: "login-button" }, // Uses testID
    { type: "swipe", selector: "scroll-view", value: "up" }
  ]
}

Clean Content Extraction

// Extract clean content from documentation
{
  url: "https://docs.react.dev/learn",
  includeLinks: true,
  format: "markdown"
}

Installation

npm install
npx playwright install

Usage with Amazon Q Developer

# Works with any website
q chat "Scrape the headlines from https://news.ycombinator.com"

# Works with React apps
q chat "Test the login flow on my React app at localhost:3000"

# Enhanced React Native web support
q chat "Inspect the React Native web app at localhost:8081"

# Extract clean content for reading
q chat "Extract the main content from https://docs.react.dev/learn"

Troubleshooting

Error Handling

• Input Validation - Server validates required parameters and provides clear error messages
• Timeout Configuration - Default timeouts are optimized but can be adjusted per request
• Browser Cleanup - Automatic resource cleanup prevents memory leaks

Regular Websites

• Use standard CSS selectors (.class, #id, tag[attribute])
• Set mobileViewport: false (default) for desktop sites
• Set waitForHydration: false (default) for non-React sites

React Applications

• Set waitForHydration: true for better reliability
• Use semantic selectors when possible
• Check browser console for React errors

React Native Web

• Use testID attributes in your components
• Enable mobileViewport or specify device
• Set waitForHydration: true
• Use inspect_react_app to see available elements

License

MIT