<!-- wl:readme.hero --> <p align="center"> <img src="assets/brand/banner.png" alt="ilinxa capture — video frames to LLM grid sheets" width="880"> </p>

ilinxa capture

Video frame extraction & composition service for AI vision pipelines.

<!-- /wl -->

<!-- wl:readme.about --> ilinxa capture extracts video frames and tiles them into 2×2 or 4×4 grid sheets a multimodal LLM can read in one image — self-hosted, via a REST API, a Web UI, or an MCP server. Instead of sending a model dozens of separate frames, you send one contact sheet, with optional frame-number and timestamp overlays so the model can still reason about time. One core engine backs all three interfaces, reachable however your pipeline prefers:

Interface	For	Entry point
REST API	Scripts, services, pipelines	/api/v1
Web UI	Interactive use	/ (3-step wizard)
MCP server	AI agents / MCP clients	stdio + Streamable HTTP
<!-- /wl -->

<!-- wl:readme.toc -->

Table of contents

• Demo
• Features
• Quick start
• Usage
  • REST API
  • Web UI
  • MCP server
• Configuration
• Architecture
• Development
• Testing
• Security model
• FAQ
• Contributing
• Support
• License <!-- /wl -->

<!-- wl:readme.demo -->

Demo

<p align="center"> <img src="assets/brand/demo.gif" alt="ilinxa capture demo — a video is extracted into frames and composed into a 4x4 grid sheet with per-frame timestamp overlays" width="880"> </p>

Video in, one grid sheet out — 16 frames tiled into a single 4×4 image with per-frame timestamp overlays, ready for a multimodal LLM. Watch the clip. <!-- /wl -->

<!-- wl:readme.features -->

Features

• Frame extraction at 1–30 FPS via FFmpeg
• Grid composition into 1×1, 2×2, or 4×4 sheets via Sharp, with optional frame-number / timestamp overlays
• Quality presets tuned for vision models: llm (1024 px, JPEG 80 %), high (original resolution, PNG), or fully custom
• Video download from YouTube, Vimeo, and 1,000+ sites via yt-dlp, with per-resolution presets or raw format selectors
• HLS support: parse master playlists, discover .m3u8 streams embedded in web pages, download variants (custom headers supported for protected streams)
• Sync or async jobs — async returns 202 + poll URL and supports webhook notifications on completion
• Streaming ZIP downloads of frames, sheets, or both — never buffered in memory
• Self-maintaining storage: TTL-based cleanup of finished jobs and orphaned temp files; job state recovers across restarts (no database) <!-- /wl -->

<!-- wl:readme.quickstart -->

Quick start

Docker (recommended)

docker build -t ilinxa-capture .
docker run -p 3000:3000 ilinxa-capture

Open http://localhost:3000 — the Web UI, REST API, and MCP HTTP endpoint are all served from the same port. FFmpeg and yt-dlp are included in the image.

Local

Requires Node 22+, FFmpeg (with ffprobe) and yt-dlp on PATH.

npm ci
npm run build
npm start                  # serves API + built UI on :3000

For development with hot reload, see Development. <!-- /wl -->

<!-- wl:readme.usage -->

Usage

REST API

Extract frames and compose a 2×2 sheet in one call:

curl -X POST http://localhost:3000/api/v1/extract-and-compose \
  -H "Content-Type: application/json" \
  -d '{
    "source": "https://example.com/video.mp4",
    "fps": 2,
    "mode": 4,
    "preset": "llm",
    "overlay_timestamp": true
  }'

Or upload a file (multipart):

curl -X POST http://localhost:3000/api/v1/extract \
  -F "file=@video.mp4" -F "fps=2" -F "preset=llm"

Add "async": true (or -F "async=true") to get an immediate 202 with a poll URL instead of waiting for the result. Pass "webhook_url" to be called back on completion.

Endpoints

Method	Path	Description
POST	/api/v1/metadata	Probe a video (file upload or {source} URL)
POST	/api/v1/extract	Extract frames
POST	/api/v1/compose	Compose frames into grid sheets
POST	/api/v1/extract-and-compose	Both steps in one call
POST	/api/v1/video/formats	List downloadable formats for a URL
POST	/api/v1/video/download	Download a video (preset or raw selector)
POST	/api/v1/hls/discover	Scan a web page for embedded HLS streams
GET	/api/v1/jobs/:id	Job status / result
DELETE	/api/v1/jobs/:id	Delete a job and its files
GET	/api/v1/jobs/:id/download?include=frames|sheets|all	Streaming ZIP
GET	/api/v1/jobs/:id/video	Download a downloaded video file
GET	/api/v1/files/:jobId/*	Serve an individual frame/sheet
GET	/api/v1/health	Health check

Errors are consistent JSON: { "error": { "code": "VALIDATION_ERROR", "message": "fps: expected number" } } with meaningful HTTP status codes (400, 404, 410, 413, 500).

Full endpoint documentation with request/response schemas: docs/GUIDE.md.

Web UI

A three-step wizard — Extract → Preview & Compose → Output — with file upload or URL input, live metadata preview, an HLS stream scanner, frame gallery, grid configuration, and ZIP downloads. Light and dark themes.

MCP server

ilinxa capture exposes its tools to any Model Context Protocol client.

Tools: capture_metadata, capture_extract, capture_compose, capture_extract_and_compose, capture_video_formats, capture_video_download, capture_hls_discover, capture_job_status.

Stdio (local clients — e.g. Claude Desktop, Cursor, VS Code):

{
  "mcpServers": {
    "ilinxa-capture": {
      "command": "node",
      "args": ["/absolute/path/to/ilinxa-capture/dist/mcp-entry.js"]
    }
  }
}

Streamable HTTP (remote agents): the running server exposes /mcp (POST/GET/DELETE) with session management and idle-session expiry (MCP_SESSION_TTL).

Setup walkthroughs for both transports: docs/GUIDE.md. <!-- /wl -->

<!-- wl:readme.config -->

Configuration

All configuration is via environment variables (validated at startup — the server refuses to boot on invalid config). Copy .env.example to .env to get started.

Variable	Type	Default	Description
PORT	int	3000	HTTP port
HOST	string	0.0.0.0	Bind address
NODE_ENV	enum	development	development / production / test
LOG_LEVEL	enum	info	Pino log level
STORAGE_MODE	enum	local	local or s3
LOCAL_OUTPUT_DIR	string	./data/jobs	Job output directory
LOCAL_TTL_SECONDS	int	3600	Finished-job retention before cleanup
MAX_VIDEO_DURATION	int	600	Max video length in seconds
MAX_UPLOAD_SIZE	int	524288000	Max upload size in bytes (500 MB)
MAX_CONCURRENT_JOBS	int	3	Processing concurrency limit
JOB_TIMEOUT	int	300	Per-job timeout in seconds
MCP_SESSION_TTL	int	1800	Idle MCP HTTP session lifetime in seconds
UI_DIR	string	./ui/dist	Built Web UI assets
S3_*	—	—	S3 credentials/bucket (required when STORAGE_MODE=s3)
<!-- /wl -->

<!-- wl:readme.architecture -->

Architecture

%%{init: {'theme':'neutral','themeVariables':{'primaryColor':'#CC785C','lineColor':'#6B7280'}}}%%
flowchart TD
    UI["Web UI · React 19"]
    MCPS["MCP · stdio"]
    HTTP["MCP HTTP · /mcp"]
    API["REST API · Fastify 5"]
    CORE["Core engine<br/>job queue · storage · cleanup"]
    FF["FFmpeg"]
    SH["Sharp"]
    YT["yt-dlp"]
    UI --> API
    MCPS --> API
    HTTP --> API
    API --> CORE
    CORE --> FF
    CORE --> SH
    CORE --> YT

One core engine sits behind three thin protocol adapters and shells out to FFmpeg, Sharp, and yt-dlp.

Design decisions worth knowing:

• No database. Job state lives in a job.json per job directory, schema-validated and reconstructed on startup. Stuck jobs are marked failed.
• Thin adapters. All business logic lives in src/core/; the REST and MCP layers only translate protocols.
• App factory. buildApp() creates the configured Fastify instance; index.ts just starts it — which is what makes the whole API testable.
• External binaries via execFile with argument arrays — no shell interpolation anywhere.
• Bounded resources. Concurrency-limited job queue, TTL cleanup for job dirs and orphaned temp files, idle-session expiry on the MCP HTTP transport.

src/
├── app.ts            # Fastify app factory
├── index.ts          # Server entry (graceful shutdown)
├── mcp-entry.ts      # MCP stdio entry
├── core/             # Engine: extractor, composer, metadata, downloader,
│                     #   hls, job-manager, storage, cleanup, presets
├── api/              # REST routes, handlers, schemas (+ api/lib helpers)
├── mcp/              # MCP server + tool registration
├── lib/env.ts        # Zod-validated environment
└── utils/            # Errors, logger, exec

ui/src/
├── app/              # Shell, providers, router
├── features/         # extraction / preview / output (components + api + types)
├── components/       # ui (shadcn) · common · layout
├── stores/           # Zustand (client state)
├── hooks/ lib/ types/ styles/

<!-- /wl -->

<!-- wl:readme.development -->

Development

# Backend (repo root)
npm ci
npm run dev            # tsx watch on :3000

# Web UI (separate terminal)
cd ui && npm ci
npm run dev            # Vite on :5173, proxies /api to :3000

Command	Where	Description
npm run dev	root / ui/	Dev server with hot reload
npm run build	root / ui/	Production build (tsup / Vite)
npm run typecheck	root / ui/	tsc --noEmit
npm test	root / ui/	Unit tests (Vitest)
npm run test:integration	root	End-to-end pipeline tests (real FFmpeg)
npm run test:coverage	root	Coverage report
npm run lint / format	ui/	ESLint / Prettier
npm run mcp:stdio	root	MCP server on stdio

Conventions: strict TypeScript everywhere (noUncheckedIndexedAccess), ESM only, Zod validation at every boundary, Pino logging (stdout is reserved for JSON-RPC on the MCP stdio transport). UI: named exports, Zustand for client state, TanStack Query for server state — never mixed. <!-- /wl -->

<!-- wl:readme.testing -->

Testing

Three tiers, all run in CI:

Tier	Count	What it proves
Backend unit	240 tests / 21 files	All logic, with FFmpeg/yt-dlp/Sharp/fs mocked — fast and hermetic
Backend integration	9 tests	The real pipeline: generates a video with FFmpeg, drives the live HTTP API with zero mocks — upload → extract → compose → ZIP → delete, asserting real frame counts and real sheet pixel dimensions, plus corrupt-input and path-traversal negative cases
UI	19 tests / 4 files	Store, API client, polling hook, wizard navigation guards (Vitest + Testing Library)

npm test -- --run                 # unit
npm run test:integration -- --run # integration (needs FFmpeg on PATH)
cd ui && npm test -- --run        # UI

<!-- /wl -->

<!-- wl:readme.security -->

Security model

ilinxa capture is designed for localhost / trusted-network use and ships with no authentication. By design it will fetch any URL it is given (yt-dlp, HLS discovery, direct HTTP), and the compose endpoint accepts explicit local frame paths — so it must never be exposed directly to the public internet or untrusted callers. File serving is confined to each job's own directory with path-traversal protection. For anything internet-facing, put it behind a gateway that provides authentication, rate limiting, and URL allow-listing.

To report a vulnerability, see SECURITY.md. <!-- /wl -->

<!-- wl:readme.faq -->

FAQ

How is this different from sending raw video frames to an LLM? A grid sheet packs many frames into one image, so the model reads a whole clip in a single request instead of dozens — fewer tokens, fewer round trips. Optional timestamp and frame-number overlays keep the model able to reason about when something happens.

Do I need FFmpeg and yt-dlp installed? For a local install, yes — both must be on your PATH (FFmpeg for extraction, yt-dlp for URL downloads). The Docker image bundles them, so docker run needs nothing else.

Can I use it without the Web UI — just the API or an MCP agent? Yes. The REST API, Web UI, and MCP server are three independent front doors to the same engine; run only the one you need.

Which video sources can it pull from? Local uploads, direct video URLs, HLS streams (including .m3u8 links discovered inside a web page), and any of the 1,000+ sites yt-dlp supports.

Is it safe to expose on the internet? Not as-is. It has no authentication and will fetch any URL it's given. Keep it on localhost or a trusted network, or put it behind a gateway — see the security model. <!-- /wl -->

<!-- wl:readme.contributing -->

Contributing

Contributions are welcome — see CONTRIBUTING.md for the development workflow, test requirements, and PR guidelines. By participating you agree to the Code of Conduct. <!-- /wl -->

<!-- wl:readme.support -->

Support

Questions and usage help go in GitHub Discussions; bugs and feature requests go in Issues. See SUPPORT.md for the full routing. <!-- /wl -->

<!-- wl:readme.license -->

License

Apache License 2.0 — see LICENSE and NOTICE. <!-- /wl -->