MCP 服务器

Cleya

An MCP server for creative generation and video production, letting AI agents author declarative animations, render deterministic frames, and measure output quality with perception instruments for brand and accessibility conformance.

README

<h1 align="center">Cleya</h1>

A creative-generation and video toolkit for AI agents. Works with any LLM to make beautiful creative, from statics to 2D animation, then checks its own work before handing it over.

Status: v0.6.0, actively maintained. 834 tests across 75 files, a documented scene grammar with a JSON Schema beside it, and a contract-tested starter library — a change that breaks a starter breaks the build. Issues and pull requests welcome.

What it is

67 tools on three surfaces — a CLI, an MCP server, and a TypeScript library — covering the whole path from "I have a brief" to "here is a measured, on-brand MP4":

Author. Describe an animation as declarative JSON (the scene DSL) instead of hand-writing HTML and timing code. Layers, entrances, idle motion, keyframe tracks, camera moves, tickers, data binding, multi-format output.
Render. Headless Chrome samples window.CLEYA.seekTo(t) once per frame, so every frame is a pure function of time — deterministic, seekable, resolution-independent — then FFmpeg encodes.
Measure. Perception instruments read the rendered pixels back: WCAG contrast per text box, optical alignment, dominant-colour palette, brand conformance by ΔE. The agent that made the creative can see whether the creative is actually any good.

That last step is the point. Generation is commodity; the loop is made → measured → corrected, and it runs without a human in it.

The loop, end to end

# 1. start from a guaranteed-valid skeleton
cleya starters --name promo-hero --copy-to ./promo

# 2. render it with your brand tokens
cleya render --scene promo/promo-hero.scene.json --brand brand.json --output promo.mp4

# 3. measure what came out
cleya contrast    --input promo.mp4 --time 3.2                    # WCAG per OCR'd text box
cleya align       --input promo.mp4 --time 3.2                    # near-miss edges, broken rhythm
cleya brand_check --input promo.mp4 --time 3.2 --brand brand.json # 0-100 ΔE conformance

# 4. correct deterministically — refinement as code, not deliberation
cleya edit --scene promo/promo-hero.scene.json --op '{"op":"center","layer":"headline"}' --verify

# 5. promote a good result into a reusable starter
cleya harvest --scene promo/promo-hero.scene.json --brand brand.json \
  --name book-flip-promo --when "Book-brand promo with flipping titles"

Or collapse steps 1–3 into one call:

cleya draft --starter promo-hero --brand brand.json \
  --copy '{"headline":"Ship it {accent:Friday}","tagline":"in one command","stat":42}' \
  --output draft.mp4
# → rendered + contrast-checked + brand-scored, ~15s
# (layer ids and knobs per starter: cleya starters --name promo-hero)

Quick start

Cleya is not published to npm. Install from source:

git clone https://github.com/severment/cleya.git
cd cleya && npm install && npm run build && npm link
cleya doctor          # check FFmpeg, ffprobe, Chrome, Whisper
cleya recipes         # goal → tool-sequence discovery

As an MCP server

{
  "mcpServers": {
    "cleya": { "command": "cleya-mcp" }
  }
}

Every tool is registered as cleya_<tool> with a typed schema. Point an agent at cleya_capabilities first — one call returns the whole machine-readable manifest (options, examples, categories, cost hints), which is how an agent learns the toolkit without guessing.

As a library

import { render, contrast, brandCheck, resolveConfig } from "cleya";

const config = resolveConfig();
const out = await render({ scene: "promo.json", brand: "brand.json", output: "promo.mp4" }, config);
const wcag = await contrast({ input: "promo.mp4", time: 3.2 }, config);

Every tool takes (params, config) and returns Promise<ToolResult<T>>.

Authoring — the scene DSL

A scene is data, not code. The compiler turns it into the deterministic seekTo runtime that the capture backends drive:

{
  "size": [1920, 1080], "duration": 10, "bg": "$palette.bg",
  "layers": [
    { "id": "h1", "type": "text", "text": "MAKE IT MOVE", "at": { "x": 118, "y": 208 },
      "style": { "size": 104, "weight": 900, "color": "$palette.ink", "uppercase": true },
      "in": { "at": 0.45, "dur": 0.75, "from": "rise", "ease": "outCubic" } },
    { "id": "cta", "type": "text", "text": "Get started", "anchor": "bottom-left", "safe": true,
      "style": { "bg": "$palette.accent", "pad": [18, 34], "radius": 999 },
      "in": { "at": 2.1, "dur": 0.5, "from": "pop" },
      "idle": { "kind": "breathe", "amp": 0.02 } }
  ]
}

--check validates without rendering (~0.05s) and warns on dead air.
--at <t> renders one frame; --preview writes a labelled contact sheet — this is how an agent "watches" a video without watching it.
--format og,square,story emits every size from one spec, with per-layer overrides.
--brand brand.json resolves $brand.* / $palette.* tokens and merges fonts.
--data / --data-each bind typed values and batch-render one spec into N named outputs.
cleya scrub opens a film in a self-contained scrub page — timeline with hover-scrub, frame-stepping, loop — with zero rendering, because it plays through the same seekTo contract. With --brand, it also carries a live palette bar: re-tint any brand role and the film updates instantly, then download the edited brand.json to render with. Works for raw --html films too, as long as their colours are authored as var(--cleya-<role>, <hex>).

type: "video" layers put real footage on the same timeline — the runtime seeks currentTime per frame rather than playing, so a cut is still a pure function of scene time.

Full grammar: docs/scene-spec.md · schema: docs/scene.schema.json · craft guidance for ambitious work: docs/craft.md

When the DSL genuinely isn't enough, render --html --runtime gsap is the escape hatch: it injects GSAP with every plugin registered plus window.cleya helpers, and you author a paused timeline that the same seekTo bridge drives.

Skills

skills/ ships agent-readable craft guidance, one directory per lens — read before authoring, not after:

Skill	For
edit-craft	The grammar of cuts: which cut, when to cut, ad pacing, and how each maps onto layer windows
marketing-producer	Campaign work — offer, proof, CTA
social-editor	Vertical, sound-off, first-three-seconds
documentary-editor	Long-form, interview-led
podcast-producer	Multi-speaker audio-first

Perception instruments

The part nothing else in this category ships. All of these read rendered pixels, not source specs.

They work on any video, not just one Cleya rendered. contrast, align, palette, brand_check, review and pacing take --input <file> and measure whatever is in it. Point them at output from another renderer, at footage from an editor, at a file someone sent you. The measuring half of this toolkit is not coupled to the authoring half, so you can adopt it without moving anything you already have. (scrub is the exception — it needs a seekTo film to drive.)

Tool	What it measures
contrast	WCAG ratio per OCR'd text box — strokes split from ground by Otsu, so it measures the text, not the box
align	Infers the frame's alignment systems, then flags only a lone element that almost joins one — two deliberate systems a few px apart are not a defect. Plus broken vertical rhythm
palette	Coverage-weighted dominant colours (ΔE-merged) + luminance stats
brand_check	0–100 brand conformance against a `BrandProfile` by ΔE, neutrals passing; optional scene font check
review	Render QA — black/empty/frozen frames, text collisions, unsafe margins, tiny text, audio dead air; `--platform tiktok\|reels\|shorts` adds UI-occlusion zones
claims	Lints copy against a claims policy — banned phrases, required disclaimer, line length, reading grade
pacing	The film's rhythm: holds, beats, segment lengths, end-card dwell, plus an activity strip. The only instrument that reads the film as a film rather than a frame, and the only way to check "the end card holds for three seconds" without watching it
diff	Where two frames differ (bounding box), by how much (per-channel max), PSNR, and a difference image amplified to the change it actually contains

harvest and edit --verify are instrument-gated: alignment defects block, contrast findings warn. A defective scene cannot silently enter your starter library.

The rest of the toolkit

Run cleya --help for all 67, or cleya capabilities for the machine-readable manifest.

Author — render, screenshot, card, title, caption, still, tts, capture (scripted real-app screen recording with synthetic cursor, auth state seeding, dev-overlay hiding)
Pipeline — draft, compose (multi-scene assembly with transitions, music bed, ducking, beat-snap, per-scene VO and content-hashed render cache), social_clip, autosub
Edit — trim, concat, speed, reformat, audio, overlay, pip, color, lut, effects, shake, upscale, gif, kenburns, layout, multiclip, sync, bg_remove
Scout — info, scenes, motion, loudness, beat_detect, cut_silence, thumbnails (saliency-ranked), transcribe, detect_text, image_similarity
Assets — cutout (Vision foreground segmentation to transparency), smart_crop, brand_extract, segment, lut_extract
Export — export_nle (FCPXML), translate_srt

Requirements

Requirement	Needed for	Notes
Node ≥ 18	everything
FFmpeg + ffprobe	everything	`brew install ffmpeg`. Invoked as a separate process; set `ffmpeg_path` to point elsewhere
Chrome / Chromium	`render`, `screenshot`, `capture`	Auto-detected; override with `CLEYA_CHROME`. Flags via `CLEYA_CHROME_FLAGS`
FFmpeg with libfreetype	`card`, `caption`, `autosub`	Default Homebrew formula lacks it: `brew tap homebrew-ffmpeg/ffmpeg && brew install homebrew-ffmpeg/ffmpeg/ffmpeg`
whisper.cpp	`transcribe`, `autosub`, `clean_speech`, `animated_captions`	Returns `WHISPER_NOT_FOUND` if absent
macOS + Xcode CLT	`contrast`, `align`, `detect_text`, `caption_extract`, `cutout`, `segment`, `bg_remove`, `animated_captions`, `image_similarity`	Vision-backed, so these nine need a Mac. Native helpers compile on first use and cache
`OPENAI_API_KEY`	`tts` only	Opt-in cloud; nothing else calls out

Cleya runs on macOS, Linux and Windows. 53 of the 67 tools are platform-independent, including the whole core loop: render, draft, starters, edit, harvest, scrub, compose, brand_check, palette, claims and diff. Nine are macOS-only (the Vision-backed row above), and five degrade rather than fail: review keeps its FFmpeg checks, smart_crop falls back to a centre crop, thumbnails loses saliency ranking, reformat loses crop:auto, autosub loses the animated style.

cleya doctor checks all of it at once.

Result contract

{ success: true, output: "/abs/path", tool, cleya_version, duration_ms, ...toolSpecificData }
{ success: false, error: "Human-readable message", code: "ERROR_CODE", suggestion?: "What to do" }

Codes are stable and actionable — VALIDATION_ERROR, FFMPEG_ERROR, WHISPER_NOT_FOUND, DRAWTEXT_NOT_FOUND, OUTPUT_EXISTS, NO_CANDIDATES, and more. Agents branch on the code; humans read the message.

Configuration

const config = resolveConfig({
  ffmpeg_path: "/opt/homebrew/bin/ffmpeg",
  temp_dir: "/tmp/cleya",
  ffmpeg_timeout: 600_000,
});

Key	Default	Purpose
`ffmpeg_path` / `ffprobe_path`	`"ffmpeg"` / `"ffprobe"`	Binary locations
`whisper_path`	`"whisper"`	whisper.cpp binary
`temp_dir`	`"/tmp/cleya"`	Working files
`retention_hours`	`24`	Temp cleanup
`output_dir`	`"."`	Default output directory
`ffmpeg_timeout` / `whisper_timeout`	`300000` / `600000`	Execution timeouts (ms)
`platform_presets`	see `presets.ts`	Per-platform size/codec/duration

Render tuning lives in the environment: CLEYA_RENDER_WORKERS (parallel frame capture), CLEYA_NO_GPU=1 (software raster), CLEYA_RENDER_ENGINE=chrome|webkit, CLEYA_STARTERS_DIR.

Development

npm run build        # tsc
npm test             # vitest — 834 tests across 75 files
npm run smoke        # end-to-end render smoke test

Starter scenes are contract-tested: every shipped starter must compile with zero errors and zero warnings against the neutral default brand, so a change that breaks a starter breaks the build.

License

Apache 2.0. See NOTICE for third-party attribution — in particular GSAP, which is used by the raw-HTML runtime, is licensed under GreenSock's standard "no charge" license (not Apache-2.0) and is installed from npm rather than redistributed here.