InsightFace-MCP

A fully local, offline facial recognition search system for your photo library. It plugs into LM Studio via MCP (Model Context Protocol), so you can ask an LLM things like:

"Find all pictures of this person in my Events folder."

...and get back every matching photo, with confidence scores, without a single image or embedding ever leaving your machine.

LM Studio
   │  MCP tools
   ▼
Python MCP Server (server.py)
   ├── InsightFace   → detect faces, generate embeddings
   ├── FAISS         → store embeddings, fast similarity search
   └── Events/       → recursively scanned photo library

---

1. Project structure

InsightFace-MCP/
├── server.py          # MCP server (the entry point LM Studio launches)
├── face_engine.py      # shared helpers (model loading, I/O, FAISS, metadata)
├── index_faces.py      # scans Events/ and builds the face database
├── search_faces.py     # searches the database for a reference face
├── cluster_faces.py    # optional: group faces without a reference photo
├── config.py            # all settings live here
├── requirements.txt
├── README.md
├── database/
│   ├── faces.index       # FAISS vector index (created on first run)
│   ├── metadata.json      # per-face metadata (created on first run)
│   └── file_cache.json    # mtimes, for incremental re-indexing
├── known_faces/          # (optional) put reference photos here
└── Events/               # <- put your photo library here (or point config.py at it)

---

2. Installation (Windows)

2.1 Prerequisites

• Python 3.10 or 3.11 (64-bit). Check with python --version.
• (Optional, for GPU speed) An NVIDIA GPU with a recent driver, plus a matching CUDA runtime. CPU-only works fine too, just slower on large libraries.

2.2 Create a virtual environment

Open PowerShell in the InsightFace-MCP folder:

python -m venv venv
venv\Scripts\activate

2.3 Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Notes:

• If you do not have an NVIDIA GPU, edit requirements.txt first: comment out onnxruntime-gpu and uncomment onnxruntime. Then re-run the install command. The server auto-detects whether CUDA is available and falls back to CPU either way, but installing the plain CPU package avoids downloading unused CUDA binaries.
• The first time InsightFace runs, it will automatically download the buffalo_l model pack (~300MB) from its own model repository (this is a one-time model download, not a cloud inference call — after this, all face detection/embedding happens locally with no further network access).
• pillow-heif gives you HEIC/HEIF (iPhone photo) support. If installation fails on your system, everything else still works — HEIC files will just be skipped with a warning.

2.4 Point the server at your photo library

Open config.py and set EVENTS_FOLDER, e.g.:

EVENTS_FOLDER = Path(r"C:\Users\yourname\Pictures\Events")

Or set an environment variable instead of editing the file:

setx EVENTS_FOLDER "C:\Users\yourname\Pictures\Events"

The folder is scanned recursively — every nested subfolder, no matter how deep, is included. Supported formats: JPG, JPEG, PNG, WEBP, BMP, TIFF, and HEIC/HEIF (if pillow-heif installed successfully).

2.5 Build the initial database

You can do this once up front (recommended for large libraries), or just let LM Studio call index_events_folder() the first time you ask it to search.

python index_faces.py

For 10,000+ photos, expect this to take a while on first run (minutes to tens of minutes depending on CPU/GPU). Subsequent runs are incremental and only process new/changed files.

2.6 (Optional) Add known reference photos

Drop a clear, front-facing photo of each person into known_faces/, e.g. known_faces/alice.jpg. You can then just say "find Alice" in LM Studio and point it at that file — see usage examples below.

---

3. LM Studio MCP setup

LM Studio supports MCP servers via its mcp.json configuration (Program Files / Integrations, depending on your LM Studio version — check LM Studio's own docs for the exact menu, since this changes between versions).

Add an entry like this, adjusting paths to match your machine:

{
  "mcpServers": {
    "insightface-events-search": {
      "command": "C:\\path\\to\\InsightFace-MCP\\venv\\Scripts\\python.exe",
      "args": ["C:\\path\\to\\InsightFace-MCP\\server.py"],
      "env": {
        "EVENTS_FOLDER": "C:\\Users\\yourname\\Pictures\\Events"
      }
    }
  }
}

Key points:

• Use the full path to the venv's python.exe, not just python, so LM Studio uses the environment with insightface/faiss installed.
• args must point at the full path to server.py.
• The env block is optional if you already set EVENTS_FOLDER in config.py directly.

Restart LM Studio (or reload MCP servers) after editing the config. You should see insightface-events-search listed as a connected MCP server with four tools: find_person, index_events_folder, database_status, and list_clusters.

---

4. Usage examples (in LM Studio chat)

Once connected, you can just talk naturally:

• "Search my Events folder for this face" — attach/reference a photo, and the model will call find_person with that image path.
• "Show me every event where this person appears" — same tool, phrased differently; the LLM decides to call find_person.
• "Rebuild the face database" — triggers index_events_folder.
• "How many photos have you indexed so far?" — triggers database_status.
• "Who shows up most often in my photos?" — triggers list_clusters to explore recurring faces without needing a reference photo first.

Note: exactly how you supply the reference image path depends on your LM Studio version's file-attachment / tool-argument behavior. The simplest reliable approach: save a reference photo into known_faces/, e.g. known_faces/mom.jpg, then say:

"Find every photo of the person in known_faces/mom.jpg"

The find_person tool will resolve that relative path automatically.

---

5. How it works

1. Indexing (index_faces.py)

   • Recursively walks EVENTS_FOLDER with os.walk, which by design never skips subdirectories.
   • Loads each supported image (via Pillow, so HEIC/WEBP/TIFF all work consistently), converts to the BGR array format InsightFace/OpenCV expect.
   • Runs InsightFace's buffalo_l detector, which returns bounding boxes, detection confidence, and a 512-dimension ArcFace embedding per face.
   • Faces below MIN_DETECTION_SCORE (config.py) are discarded as low-confidence/noise.
   • Each embedding is L2-normalized, then added to a FAISS IndexFlatIP (inner product on normalized vectors = cosine similarity) wrapped in IndexIDMap2 so each vector has a stable integer ID.
   • Metadata per face (source file, filename, face index within that image, EXIF timestamp if present, bounding box, detection score) is stored in metadata.json, keyed by that same integer ID.
   • File modification times are cached in file_cache.json; on subsequent runs, unchanged files are skipped entirely (incremental indexing), and changed files have their old faces removed and re-added.

2. Searching (search_faces.py)

   • Loads your reference photo, detects the (largest, most prominent) face in it, and computes its normalized embedding the same way.
   • Queries the FAISS index for the most similar stored embeddings.
   • Groups results by source image (since one photo can contain multiple faces, and the same person might appear more than once in a group photo), keeping the best-matching face's score as that image's confidence.
   • Filters by threshold and returns results sorted by confidence, descending.

3. Serving (server.py)

   • Wraps the above in an MCP server using FastMCP, exposing find_person, index_events_folder, database_status, and list_clusters as callable tools with clear docstrings so the LLM in LM Studio understands when and how to call each one.
   • Runs over stdio, which is what LM Studio expects for locally-launched MCP servers.
   • No network calls are made at query time. The only network access ever used is the one-time InsightFace model download on first run.

---

6. Performance notes

• FAISS IndexFlatIP does an exact (not approximate) similarity search. For libraries up to roughly 100k-200k faces this is still fast (a few milliseconds to tens of milliseconds per query) since it's just a matrix multiply. If your library grows far beyond that, consider switching the index type in face_engine.new_empty_index() to an approximate index like IndexIVFFlat or IndexHNSWFlat for sub-linear search time.
• GPU vs CPU: face_engine.get_face_app() automatically tries CUDA via onnxruntime's CUDAExecutionProvider first, and transparently falls back to CPU if CUDA isn't available or fails to initialize. No configuration needed — check the server's stderr log line on startup to confirm which provider was used.
• Incremental indexing: re-running index_events_folder() after adding a handful of new photos to a 10,000-photo library only processes the new files, not the whole set.
• Batching: detection is done one image at a time (InsightFace's FaceAnalysis.get() is per-image), which is the standard approach for variable-resolution photo libraries and keeps memory use predictable even on large folders.

---

7. Troubleshooting

"No module named 'insightface'" or similar import errors Make sure you activated the venv (venv\Scripts\activate) before running anything, and that LM Studio's mcp.json points at the venv's python.exe, not your system Python.

Server doesn't show up in LM Studio

• Double-check mcp.json paths use double backslashes (\\) or forward slashes, and are absolute paths.
• Try running python server.py manually in your terminal first — if it crashes, LM Studio will fail silently. Fix any errors shown there first.

"CUDAExecutionProvider" not found / falls back to CPU unexpectedly

• Confirm onnxruntime-gpu installed successfully (pip show onnxruntime-gpu).
• Confirm your NVIDIA driver + CUDA toolkit versions are compatible with the installed onnxruntime-gpu version (check onnxruntime's release notes).
• This isn't fatal — CPU fallback still works, just slower.

HEIC files are being skipped

• Confirm pillow-heif installed: pip show pillow-heif.
• Some very new/exotic HEIC variants (e.g. certain burst-mode formats) may still fail to decode; check the server log for the specific error.

database_status shows database_healthy: false

• This means the FAISS index and metadata.json have drifted out of sync (e.g. a crash mid-write). Run python index_faces.py --full to rebuild cleanly from scratch.

Searches return no matches / too many false positives

• Too few matches: lower threshold (e.g. from 0.45 to 0.35) when calling find_person, or use a clearer, well-lit, front-facing reference photo.
• Too many false positives: raise threshold (e.g. to 0.55-0.6).
• Faces at extreme angles, heavy occlusion (masks, sunglasses), or very low resolution are inherently harder to match — this is a property of any face-recognition model, not specific to this setup.

Indexing is very slow

• Confirm you're actually using the GPU if you have one (check startup log).
• For very large libraries, consider running index_faces.py directly from the terminal once (rather than through LM Studio) so you can watch the tqdm progress bar and confirm throughput.

"No face detected in reference image"

• Use a photo where the face is reasonably large, front-facing, and not heavily obscured. Cropping the reference photo tighter around the face can help.

---

8. Privacy & security

• Everything — model inference, embedding generation, similarity search — runs on your machine.
• No images, embeddings, or metadata are uploaded anywhere.
• The only outbound network request in the entire system is InsightFace's one-time model weight download the first time it runs (standard for any local ML library); after that, the server works fully offline.