SceneCraft MCP — Text → Storyboard → Video (Test Version)

Author: snippetWizard

SceneCraft MCP turns plain text into a visual plan (storyboard) and—optionally—an assembled vertical video. It exposes the pipeline both as Python modules and as an MCP tool so agentic clients can orchestrate it.

• Text-to-Scene breakdown using an LLM (OpenAI, Ollama, or an offline Mock).
• Pydantic-backed models for Shots, Scenes, and Storyboards.
• Optional Stable Diffusion frame generation + MoviePy assembly into a vertical MP4.
• File-based storage of generated storyboards.
• MCP server (optional) so tools-savvy LLMs can call it directly.

This repository is a test version. More features are on the way, including cloud uploads and cross-platform social scheduling.

What’s MCP? The Model Context Protocol is a lightweight protocol that lets LLMs talk to local/remote tools securely and consistently. Here, we provide a simple MCP server exposing a “create_storyboard” tool.

Why this project? Planning shots is time-consuming. The idea came from wanting a zero-friction loop for solo creators: describe a scene → get a structured shot plan → (optionally) generate visuals → assemble a draft video. It’s a rapid prototyping lane for script-to-screen experiments.

Pipeline Overview (Files & Flow)

1. Input (your text/script)

   • Entry points:
     • examples/test_scene_local.py:1 — generates a storyboard only.
     • examples/script_to_video.py:1 — full storyboard → frames → video demo.

2. Parse Script → Scenes

   • scenecraft_mcp/engine/script_parser.py:1
     • parse_script(script) — minimal parser that treats the whole text as one scene (replaceable later with a real slugline parser).

3. Plan Shots (LLM)

   • scenecraft_mcp/engine/shot_planner.py:1
     • plan_shots(scene_text, scene_number, style_preset) → list[Shot]
     • _build_system_prompt(...), _build_user_prompt(...) craft prompt instructions.
     • Uses scenecraft_mcp/llm/factory.py:1 → get_llm_client() to select provider.
   • Providers (implement LLMClient):
     • scenecraft_mcp/llm/base.py:1 — abstract interface (complete_json, complete_text).
     • scenecraft_mcp/llm/openai_client.py:1 — calls OpenAI Chat Completions and parses strict JSON.
     • scenecraft_mcp/llm/ollama_client.py:1 — calls local Ollama /api/chat and parses JSON.
     • scenecraft_mcp/llm/mock_client.py:1 — offline, deterministic mock for local demos (default in .env).
   • Config:
     • scenecraft_mcp/config.py:1 — LLMProvider enum (openai, ollama, mock) and env-driven settings.

4. Data Models & Storage

   • scenecraft_mcp/models.py:1 — Pydantic models: Shot, Scene, Storyboard, enums (ShotType, CameraAngle, etc.).
   • scenecraft_mcp/storage/repository.py:1 — repo interface.
   • scenecraft_mcp/storage/file_repository.py:1 — JSON-based persistence under ~/.scenecraft_mcp/projects/.
   • scenecraft_mcp/utils/ids.py:1 — generate_project_id() like proj_ab12cd34.

5. Optional: Frame Generation (Stable Diffusion)

   • scenecraft_mcp/video/framegen_sd.py:1 — SDFrameGenerator.generate_frame(...)
     • Loads a Stable Diffusion pipeline (e.g., runwayml/stable-diffusion-v1-5) on CPU/GPU.
     • Produces stylized vertical 9:16 PNGs per shot.

6. Optional: Video Assembly (MoviePy)

   • scenecraft_mcp/video/assembler.py:1 — assemble_video(frames, output_path, fps=24)
     • Resizes and concatenates the per-shot images into a vertical MP4 (requires ffmpeg).

7. MCP Server (Optional Integration)

   • scenecraft_mcp/mcp_server.py:1 — build_server() with a create_storyboard(script, title=None) tool.
   • Expose storyboard creation to MCP-compatible hosts (e.g., Claude Desktop) once mcp is installed.

Installation

Core (storyboard + MCP):

pip install -r requirements.txt

Optional (frames + video):

• ffmpeg (required by MoviePy)
• Python packages:

pip install moviepy diffusers
# Then install PyTorch appropriate for your system:
# CPU-only example:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Tip: For NVIDIA GPUs, use the official PyTorch site to get the correct CUDA wheels: https://pytorch.org/get-started/locally/

Configuration

Copy .env (already included) and set a provider:

# LLM Provider (openai / ollama / mock)
LLM_PROVIDER=mock

# OpenAI
OPENAI_API_KEY=sk-your-key
OPENAI_MODEL=gpt-4.1-mini

# Ollama
OLLAMA_MODEL=llama3
OLLAMA_BASE_URL=http://127.0.0.1:11434

# Performance
LOW_LATENCY_MODE=true

• mock runs fully offline for predictable local demos.
• ollama requires ollama serve and a pulled model like llama3.
• openai requires a valid OPENAI_API_KEY.

Quickstart

Storyboard-only (offline by default):

python examples/test_scene_local.py

Storyboard → Frames → Video (requires optional deps):

python examples/script_to_video.py

Outputs:

• Storyboard JSON: ~/.scenecraft_mcp/projects/<project_id>.json
• Frames: outputs/frames/*.png
• Video: outputs/videos/<project_id>.mp4

Using the MCP Server (optional)

pip install mcp
python -m scenecraft_mcp.mcp_server

Connect this server to an MCP-compatible host (e.g., Claude Desktop). The tool create_storyboard accepts script and returns { project_id, scene_count, shot_count }.

What This Solves

• Fast ideation: turn a draft description into a structured shot plan.
• Consistent visual language: shot types, angles, durations captured as data, not prose.
• Extensible pipeline: swap LLMs, models, or frame generators without changing the flow.
• Automation-ready: MCP tool interface and file-based outputs are easy to orchestrate.

Roadmap (Test Version — Coming Soon)

• Cloud uploads after video generation (S3/GCS/Azure).
• Automation script to fetch by id/date and auto-post to YouTube and Instagram.
• Schedule entire months of content in one pass.
• Better text-to-scene parser for multi-scene scripts.
• Transitions, captions/overlays, TTS/VO integration, music bed.
• More MCP tools and richer schemas.

Follow along — more MCP projects are coming soon. Star and follow on GitHub: snippetWizard. Stay tuned!

File-by-File Quick Reference

• scenecraft_mcp/engine/script_parser.py:1 — parse_script(script) → minimal single-scene dict.
• scenecraft_mcp/engine/shot_planner.py:1 — plan_shots(...) calls LLM to output ShotPlan.
• scenecraft_mcp/llm/factory.py:1 — get_llm_client() from env-configured provider.
• scenecraft_mcp/llm/openai_client.py:1 — OpenAI JSON completions.
• scenecraft_mcp/llm/ollama_client.py:1 — Ollama JSON completions.
• scenecraft_mcp/llm/mock_client.py:1 — offline JSON synthesis for demos.
• scenecraft_mcp/models.py:1 — core Pydantic models and enums.
• scenecraft_mcp/storage/file_repository.py:1 — save/load Storyboard to ~/.scenecraft_mcp/projects/.
• scenecraft_mcp/video/framegen_sd.py:1 — Stable Diffusion image generator per shot.
• scenecraft_mcp/video/assembler.py:1 — MoviePy image sequence → MP4.
• scenecraft_mcp/mcp_server.py:1 — MCP tool create_storyboard.
• examples/test_scene_local.py:1 — storyboard demo.
• examples/script_to_video.py:1 — full pipeline demo.

License

This project is licensed under the MIT License. See LICENSE for details.

Copyright (c) 2025 snippetWizard