MCP 服务器

reelminner

Enables AI agents to scrape Instagram Reels and public profile data (metadata, engagement, owner info) without the official API, via tools for scraping, status checks, cookie import, stopping, and exporting results.

README

<h1 align="center">🎬 Reelminner</h1> Instagram Reel & Profile Scraper — Pro

Extract reel metadata, owner profiles, follower counts, music, likes / comments / views and more from any public Instagram Reel — through a desktop GUI, a one‑command CLI, a Python API, or an MCP server that AI agents can drive.

💡 Name note: This project's final public name is Reelminner. The Python engine class is Reelminner (see scraper.py), the CLI/GUI and MCP server are branded reelminner, and the GitHub repository is reelminner. The earlier working codename ReelSnipe has been fully retired. Other name ideas are listed in Name options.

📚 Table of Contents

What is Reelminner
✨ Features
🧠 How it works
🏗️ Project architecture
📦 Installation
🚀 Quick Start
💻 Usage
📊 Output format
⚙️ Configuration
🗂️ Project structure
🧪 Testing & QA
📦 Building a standalone EXE
⚠️ Legal & ethical disclaimer
🤝 Contributing
📄 License
🆘 Troubleshooting & FAQ
🏷️ Name

What is Reelminner

Reelminner is an open‑source toolkit that pulls structured data out of Instagram Reels and the profiles that posted them. It is built around a single, reusable engine (Reelminner) that is exposed four different ways:

Interface	File	Best for
🖥️ Desktop GUI	`gui.py`	Non‑technical users, one‑click scraping
⌨️ CLI	`scraper.py`	Power users, batch jobs, scripts
🤖 MCP server	`mcp_server.py`	AI agents / LLM workflows
🐍 Python API	import `scraper`	Embedding inside your own code

Everything shares the same parsing, session, and rate‑limit logic, so results are identical no matter which front‑end you use.

✨ Features

Multi‑source reel parsing — Reelminner reads data from several layers (embedded JSON, GraphQL responses, and a live DOM fallback) so it keeps working even when Instagram changes one of them.
Owner profile enrichment — for every reel it can auto‑fetch the poster's username, full_name, bio, followers, is_verified, and reels_count.
Follower count extraction — pulled via Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, with a DOM fallback and pagination (handles capped follower figures like “1.2M” by scrolling the profile).
Music metadata — reel audio music_title, music_artist, and music_id.
Engagement metrics — views, likes, comments, and the direct video_url / thumbnail.
Session & login management — interactive QR/login, cookie import from EditThisCookie exports, and a 24‑hour session refresh so you don't re‑login constantly.
Concurrent scraping — a thread pool (--workers, default 3) with polite inter‑request delays (--delay, default 2s) and adaptive back‑off when Instagram throws BLOCKED / RATE_LIMITED.
Resilient status tracking — every row carries a status code (OK, PARSED_PARTIAL, FAILED, NO_DATA, BLOCKED, RATE_LIMITED) so you know exactly what succeeded.
Multiple export formats — CSV (default), JSON, and Excel (.xlsx via openpyxl).
MCP server — five stable tools so an AI agent (Claude, Cursor, etc.) can scrape, check status, import cookies, stop, and export.
Desktop GUI — built‑in dark theme, paste‑URL box, live results table, right‑click copy URL / open reel, and one‑click export.
Tested — pytest suite + an end‑to‑end QA harness that enforces data‑quality gates.

🧠 How it works

┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐
│   GUI      │   │    CLI     │   │  MCP srv   │   │  Python    │
│  gui.py    │   │ scraper.py │   │mcp_server  │   │   import   │
└─────┬──────┘   └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
      └────────────────┴────────────────┴────────────────┘
                       ▼
              ┌───────────────────────┐
              │  Reelminner  │  ← the engine (scraper.py)
              │  • session / cookies   │
              │  • thread pool         │
              │  • adaptive back‑off   │
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │  parsers.py            │  ← pure extraction helpers
              │  parse_reel_page / json│
              │  parse_owner / music   │
              │  regex adapters         │
              └───────────────────────┘

Normalize the input URL (normalize_reel_url) so /reel/X/ and /reel/s/…/ both work.
Load session — apply saved cookies (sessionid, csrftoken, ds_user_id, ig_did, mid, rur) or log in.
Fetch & parse the reel page with a layered fallback:
- parse_reel_page → embedded window.__additionalData / sharedData HTML JSON
- parse_reel_json → raw GraphQL GQL response
- parse_graphql_reel → shortcodeMedia object
- DOM fallback → _extract_text_raw queries the live page for likes / comments / plays / followers via regex adapters.
Enrich owner (unless --no-profiles): fetch the profile and read followers, full_name, bio, is_verified, reels_count.
Respect limits: sleep delay between requests; if blocked, back off and retry.
Write rows to CSV / JSON / Excel with a status per row.

🏗️ Project architecture

Reelminner is a single‑engine, multi‑interface design. One core engine (Reelminner) does all the real work; the GUI, CLI, MCP server, and Python API are thin front‑ends that call into it. This keeps parsing, session handling, and rate‑limiting identical across every entry point.

                         ┌─────────────────────────────┐
        URL(s) in ──────▶│     Reelminner     │  scraper.py
                         │  ── engine / orchestrator ──  │
                         └───────┬───────────┬──────────┘
                  run scrapes    │           │  enrich owner
                                 ▼           ▼
                    ┌────────────────┐  ┌──────────────────┐
                    │   parsers.py    │  │ session + graphql│
                    │ pure extractors │  │ (followers/music)│
                    └───────┬────────┘  └─────────┬────────┘
                            └─────────┬────────────┘
                                      ▼
                            ReelData row + status
                                      ▼
                       CSV / JSON / Excel writers

Module responsibilities

File	Role	Key public symbols
`scraper.py`	Core engine + CLI. Owns the browser, session, thread pool, and writers.	`Reelminner`, `scrape()`, `login()`, `has_session()`, `save_cookies_from_file()`, `clear_session()`, `write_csv`, `export_json`, `export_excel`, `normalize_reel_url`, `csv_columns`, `ReelData`, `DEFAULT_STATE_FILE`
`parsers.py`	Pure extraction helpers — no browser, easy to unit‑test.	`parse_reel_page`, `parse_reel_json`, `parse_graphql_reel`, `parse_owner_username_from_html`, `parse_music`, `parse_count`, `parse_caption`, `parse_graphql_followers`, `parse_profile_card`
`gui.py`	Tkinter desktop app. Builds the window, menu, URL box, workers slider, results table, and export dialogs.	`ReelminnerGUI`, `build()`, `scrape()`, `export_*`, `copy_url()`, `open_reel()`
`theme.py`	GUI styling — applies the dark theme to `ttk` widgets.	`apply_dark_theme(root)`
`mcp_server.py`	MCP server — exposes the engine as 5 tools for AI agents over stdio.	`mcp` (FastMCP), `scrape_reels`, `get_status`, `import_cookies`, `stop_scrape`, `export_results`
`build_exe.py`	Packaging — PyInstaller one‑file build.	`EXE(...)`, `COLLECT`/`Analysis`
`run_qa.py`	QA harness — runs the engine over a corpus and enforces data‑quality gates.	`run_qa()`, gate checks, `qa_report.json`

Engine internals (`Reelminner`)

Session layer — _SESSION_COOKIE_NAMES (sessionid, csrftoken, ds_user_id, ig_did, mid, rur); _apply_cookies(), _refresh_if_needed() (24h), login() (interactive QR), clear_session().
Concurrency — scrape() spins up a ThreadPoolExecutor(max_workers=workers); each URL is handled by _worker → _scrape_url, which calls _gather_metadata (reel data) and optionally _gather_article (owner profile). A semaphore + _sleep() enforce politeness; status_code / retcode drive an adaptive retry/back‑off loop when Instagram returns BLOCKED / RATE_LIMITED.
Parsing pipeline (layered fallback) — inside _gather_metadata the engine tries, in order: parse_reel_page (embedded HTML JSON) → parse_reel_json (raw GraphQL GQL) → parse_graphql_reel (shortcodeMedia) → DOM fallback via the _extract_text_html / _extract_text_raw adapters and the _PATTERNS regex list (likes/comments/plays/followers).
Profile enrichment — get_follower_count() uses Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, falling back to the DOM and paginating followers (_fetch_followers with end_cursor) when counts are capped.
Output — rows are collected as ReelData dicts and written by write_csv (respecting csv_columns), export_json, or export_excel (needs openpyxl).

Why this layout

Testability — all parsing lives in parsers.py with no browser dependency, so tests/test_parsers.py can assert on saved HTML/JSON fixtures.
One source of truth — every interface shares the same Reelminner, so a fix in the engine benefits the GUI, CLI, and MCP server simultaneously.
Safe packaging — the GUI/CLI thin shells mean the PyInstaller EXE only bundles the engine + a minimal UI, keeping the binary small.

📦 Installation

Requirements: Python 3.10+ and the Playwright browser engine.

# 1. Clone
git clone https://github.com/ilovekushgola/reelminner.git
cd reelminner

# 2. (Recommended) create a virtual environment
python -m venv .venv
.venv\Scripts\activate        # Windows
# source .venv/bin/activate   # macOS / Linux

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install the Chromium browser for Playwright
playwright install chromium

GUI only: the desktop app uses tkinter, which ships with standard Python installs. No extra package needed. The GUI is most polished on Windows.

Optional dev/test tools:

pip install -r requirements-dev.txt   # pytest, coverage

💡 Before you start: Reelminner works best with a logged‑in Instagram session — some reels and all owner/follower data require authentication. Run python scraper.py --login once (interactive QR), or import cookies exported from the EditThisCookie browser extension with python scraper.py --import-cookies cookies.json. It only reads public content you're already allowed to view.

🚀 Quick Start

# Scrape a single reel from the command line
python scraper.py "https://www.instagram.com/reel/CxXYZ123/"

# …or many reels from a file (one URL per line)
python scraper.py -f urls.txt -o export.csv

# Launch the desktop GUI
python gui.py

💻 Usage

1. Desktop GUI

python gui.py

Click Login (optional but recommended — improves success rate).
Paste one reel URL per line into the box (or Ctrl+A to select all).
Drag the Workers slider, then click Scrape.
Watch results appear in the table.
Right‑click a row to Copy URL or Open Reel.
Export to CSV / Excel / JSON, or Open results folder.

The last results are auto‑saved to results/_last_results.json.

2. Command Line (CLI)

python scraper.py [URL ...] [options]

Flag	Default	Description
`urls`	—	One or more reel URLs (positional).
`-f`, `--file`	—	Text file with one reel URL per line.
`--login`	off	Open a browser to log in interactively (QR).
`--import-cookies FILE`	—	Import an EditThisCookie JSON export.
`--clear-session`	off	Delete the saved `storage_state.json`.
`--headless`	off	Run the browser without a window.
`-w`, `--workers`	`3`	Number of concurrent scrape threads.
`--delay`	`2.0`	Seconds to wait between requests.
`--state`	`storage_state.json`	Path for the saved session.
`-o`, `--output`	`reels_results.csv`	Output CSV path.
`--no-profiles`	off	Skip auto‑fetching owner follower data.

# Headless, 5 workers, 1s delay, no profile enrichment
python scraper.py -f reels.txt -w 5 --delay 1 --headless --no-profiles -o out.csv

3. MCP Server (for AI agents)

Reelminner ships an MCP (Model Context Protocol) server so an AI client can drive it.

python mcp_server.py            # stdio transport

Configure your MCP client (.mcp.json is included in the repo):

{
  "mcpServers": {
    "reelminner": {
      "command": "python",
      "args": ["mcp_server.py"],
      "cwd": ".",
      "env": { "RMIN_HEADLESS": "true" }
    }
  }
}

Tools exposed (5, stable):

Tool	Signature	Purpose
`scrape_reels`	`(urls, workers, delay, headless, with_profiles)`	Run a scrape job.
`get_status`	`()`	Current progress / last result summary.
`import_cookies`	`(json_path)`	Load cookies from an EditThisCookie file.
`stop_scrape`	`()`	Stop the running job.
`export_results`	`(path, fmt)`	Export to `csv` / `json` / `xlsx`.

Environment overrides: RMIN_HEADLESS, RMIN_WORKERS, RMIN_DELAY, RMIN_WITH_PROFILES.

4. Python API

from scraper import Reelminner, write_csv

scraper = Reelminner(workers=3, delay=2.0, headless=True)
rows, report = scraper.scrape(
    ["https://www.instagram.com/reel/CxXYZ123/"],
    with_profiles=True,
)
write_csv(rows, "out.csv")

for r in rows:
    print(r["username"], r["followers"], r["likes"], r["status"])

Key members of Reelminner:

scrape(urls, with_profiles=True) → (rows, report)
login() — interactive login
has_session() / save_cookies_from_file(path) / clear_session()
write_csv(rows, path), export_json(rows, path), export_excel(rows, path)
normalize_reel_url(url) — public helper
csv_columns — the ordered list of output fields
DEFAULT_STATE_FILE — default storage_state.json

📊 Output format

Each reel becomes one row. The full CSV schema (scraper.csv_columns):

Column	Description
`idx`	Row index.
`username`	Reel owner handle (e.g. `natgeo`).
`followers`	Owner follower count (may be `follower_min`–`follower_max`).
`full_name`	Owner display name.
`bio`	Owner biography text.
`is_verified`	`True` / `False`.
`reels_count`	Number of reels on the owner profile.
`profile_url`	Link to the owner profile.
`reel_url`	Canonical reel URL.
`reel_id`	Instagram reel shortcode / ID.
`caption`	Reel caption text.
`upload_date`	Post timestamp.
`views`	Play / view count.
`likes`	Like count.
`comments`	Comment count.
`video_url`	Direct video file URL.
`thumbnail`	Thumbnail image URL.
`music_title`	Audio track title.
`music_artist`	Audio artist.
`music_id`	Audio / music ID.
`scrape_ts`	When this row was scraped (ISO timestamp).
`status`	`OK` · `PARSED_PARTIAL` · `FAILED` · `NO_DATA` · `BLOCKED` · `RATE_LIMITED`.

⚙️ Configuration

Cookies / session

Log in with python scraper.py --login (saves storage_state.json).
Or export cookies from your browser via the EditThisCookie extension and run python scraper.py --import-cookies cookies.json.

Environment variables (used by MCP server & CLI defaults)

Variable	Effect
`RMIN_HEADLESS`	`true`/`false` — run browser headless.
`RMIN_WORKERS`	Default worker count.
`RMIN_DELAY`	Default delay between requests (seconds).
`RMIN_WITH_PROFILES`	`true`/`false` — auto‑enrich owner profiles.

A template is provided: copy mcp.env.example → mcp.env to override MCP defaults.

🗂️ Project structure

reelminner/
├── scraper.py          # Core engine: Reelminner + CLI
├── gui.py              # Tkinter desktop application
├── parsers.py          # Pure extraction helpers (HTML/JSON/music/regex)
├── mcp_server.py       # MCP server (5 tools for AI agents)
├── theme.py            # Dark‑theme styling for the GUI
├── build_exe.py        # PyInstaller build script
├── Reelminner.spec  # PyInstaller spec (one‑file EXE)
├── run_qa.py           # End‑to‑end QA harness with data‑quality gates
├── requirements.txt    # Runtime dependencies
├── requirements-dev.txt# Dev / test dependencies
├── mcp.env.example     # MCP env template
├── .mcp.json           # MCP client configuration
├── assets/             # Icons (icon.ico)
├── docs/               # SKILL.md, E2E test/fix plan
├── skills/             # Agent skill definition
├── tests/              # pytest suite + corpus.txt
└── results/            # Scrape outputs (git‑ignored)

🧪 Testing & QA

# Unit / integration tests
pytest -q

# End‑to‑end data‑quality run (uses your saved session)
python run_qa.py                 # full run over tests/corpus.txt
python run_qa.py --quick         # 1 URL, headless, fast iteration
python run_qa.py --url <reel>    # custom single URL
python run_qa.py --report-only   # show last qa_report.json

The QA harness enforces gates such as parsed‑rate, verified‑rate, non‑empty‑rate, blocked‑rate, and max runtime, and writes results/qa/qa_report.json + qa_results.csv.

📦 Building a standalone EXE

On Windows, produce a portable .exe (no Python needed by end users):

pip install pyinstaller
python build_exe.py

Output: dist/Reelminner.exe (one‑file build via Reelminner.spec).

⚠️ Legal & ethical disclaimer

Reelminner is provided for educational and authorized/personal use only.

Scraping Instagram may violate its Terms of Service. Use it only on content you own or are permitted to access.

Respect rate limits (--delay, fewer --workers) and do not use it for spam, harassment, or commercial bulk extraction.

You are responsible for how you use this tool and for complying with applicable laws (incl. GDPR / privacy regulations) in your jurisdiction.

The authors are not affiliated with Instagram/Meta and accept no liability.

🆘 Troubleshooting & FAQ

playwright says the browser isn't installed / pages won't open → Make sure you ran both pip install -r requirements.txt and playwright install chromium. Without the Chromium download nothing will launch.

Most fields are empty, or I get BLOCKED / RATE_LIMITED → Log in (python scraper.py --login) or import cookies, then slow down: --delay 4 and fewer workers (-w 1). Instagram throttles anonymous/unauthenticated traffic hardest, so an authenticated session is the single biggest success factor.

A reel returns NO_DATA → The post may be private, deleted, or region‑locked, or Instagram served a login wall. Try again with a logged‑in session.

The GUI window won't open or fonts look wrong → The GUI uses Python's built‑in tkinter. On Windows it's most polished. On Linux/macOS install the Tk package if the window fails to launch (e.g. sudo apt install python3-tk).

ModuleNotFoundError when I run a script → You're likely outside the repo or its virtual environment. cd into the project folder and activate the venv (.venv\Scripts\activate on Windows, source .venv/bin/activate on macOS/Linux) before running python scraper.py.

How do I scrape lots of reels at once? → Put one URL per line in a text file and run python scraper.py -f urls.txt -o out.csv.

Can an AI agent use this? → Yes — run python mcp_server.py and point any MCP client (Claude Desktop, Cursor, etc.) at the included .mcp.json. See MCP Server.

🤝 Contributing

Fork the repo and create a feature branch.
pip install -r requirements-dev.txt
Add/adjust tests in tests/; run pytest and python run_qa.py --quick.
Open a pull request describing the change and the QA result.

📄 License

Released under the MIT License — see LICENSE.

🏷️ Name

The project's final public name is Reelminner ("Reel miner"). Earlier internal codenames have been retired. If you fork it you can rename it to anything you like — just update the title in gui.py and this README.

Made with ❤️ for the open‑source scraping community.