VOCALOID MCP

An agent-native production bridge for composing, tuning, rendering, mixing, and auditing native VOCALOID3/4 projects — built just for fun.

This repository lets a coding agent begin with an empty timeline or a creative brief and work through:

intent
  → theory / form / harmony / melody
  → Japanese note allocation
  → native IA delivery
  → arrangement
  → original VOCALOID3 Editor rendering
  → isolated stems / mix / master / QC
  → VSQX + hash-bound creative record

It is not a MIDI-to-VSQX converter and it is not a one-button hit-song generator. Notes, lyrics, phonemes, note expression, vibrato, and VOCALOID control curves compile into native VSQ3/VSQ4 structures. The licensed original Editor remains the synthesis authority.

The project has proven that an agent can be a useful composer assistant, production engineer, and experiment partner. It has not engineered taste or the “perfect song.” Our latest technically valid 190 BPM IA_ROCKS test still received the honest human verdict: the vocal was masked by the accompaniment, and the result did not really feel like rock.

That distinction is a feature.

What is actually proven

• Create native VSQ3/VSQ4 projects from zero, without copying a source project.

• Validate with Yamaha's schemas and a hash-pinned native Vsq3.dll loader probe.

• Render the selected locally licensed singer through the original VOCALOID3 Editor.

• Fail closed on singer substitution and suspicious cross-component probe collisions.

• Author Japanese lyrics with explicit kana readings, manual-backed phonemes, mora allocation, rests, melisma, and small-っ timing choices.

• Compile high-level articulation, dynamics, timbre, vibrato, and pitch gestures into native note styles and control curves.

• Maintain a revisioned canonical song manifest with stable IDs, dry-runs, bounded edits, history, and separate content/composition/mix hashes.

• Render multi-instrument accompaniment, isolated vocal/instrument stems, deterministic effect chains, section automation, mixes, previews, A/B pairs, and masters.

• Measure LUFS, LRA, sample/true peak, clipping, channel balance, score/render pitch, timing, release, and vibrato evidence.

• Run deterministic music-theory and arrangement diagnostics without letting rules or corpus data compose the next note.

• Preserve concise, hash-chained creative decisions across:

  intent → theory → note allocation → IA delivery → arrangement
  

• Transfer the workflow to genuinely fresh Codex sessions using one fixed prompt plus one strict per-song intent file.

Current automated result:

102 tests discovered
100 passed
2 intentionally skipped by environment
0 failed

The important non-claim

Passing every engineering gate does not mean a song is musically successful.

Engineering can establish	Engineering cannot decide
The VSQX is structurally valid	The melody is memorable
The requested singer rendered non-silent audio	The performance feels alive
Native controls were materialized	The tuning is tasteful
The delivery is current and unclipped	IA sits correctly in the mix
A phrase avoids concrete theory risks	The phrase says what the listener needs to hear
A 190 BPM arrangement is stable	It actually feels like rock

Read the story

Every long-form article is available in Traditional Chinese and English. The Traditional Chinese editions are the primary, more personal narrative; the English editions preserve the same technical claims for a wider audience.

Article	繁體中文	English
Native engineering deep dive	從空白 Timeline 到一首完成的 VOCALOID 歌曲	From a Blank Timeline to a Finished VOCALOID Song
Intent, theory, and fresh-agent composition	從「它能唱」到「我希望 IA 在這一句唱出什麼？」	From “Can It Sing?” to “What Should IA Sing in This Line?”
The fun whole-project story	我們叫一個 Coding Agent 去做 VOCALOID，然後事情失控了	We Told a Coding Agent to Make VOCALOID Songs

Architecture

                         CREATIVE / CONTROL PLANE

         strict intent file + fixed agent workflow + listening question
                                   │
                                   ▼
          22-tool production MCP profile + 14 core resources
                       + 1 optional local lyric resource
             typed schemas │ dry-run │ preflight │ bounded proposals
                                   │
                                   ▼
                         canonical *.song.json
       intent │ timeline │ harmony │ vocals │ arrangement │ mix │ review
                                   │
                    ┌──────────────┴──────────────┐
                    ▼                             ▼
            NATIVE VOCAL PLANE            INSTRUMENT PLANE
          VSQ3/VSQ4 compiler               pattern expansion
        XSD + native loader probe          FluidSynth / SF2
          VOCALOID3 Editor                 per-track stems
                    └──────────────┬──────────────┘
                                   ▼
                         AUDIO PRODUCTION PLANE
             alignment → effects → automation → sum → master
                                   │
                                   ▼
                          EVIDENCE / DELIVERY
           hashes │ dependencies │ history │ journal │ previews │ QC

VSQX is a native Editor artifact, not the sole database. The canonical manifest retains information that VSQX does not naturally own: renderer choices, mix chains, artifact dependencies, reviews, intent, and mutation history.

Creative boundaries

Intent comes before score

The canonical vocaloid-composition-intent/v1 contract can declare song, section, and anchor-phrase purpose before any note exists. A phrase direction carries one listening question and can later bind to stable score, tuning, harmony, and arrangement objects.

Theory is a guardrail

The deterministic core/v1 analyzer distinguishes structural errors, musical risks, and style observations. It can find timeline contradictions, breath pressure, unresolved tones, voice collisions, register crowding, and arrangement redundancy. It does not infer that a phrase is moving, “IA-like,” or good.

Phrase grammar is not a melody template

The optional DSL provides versioned role, entry, contour, rhythm, motion, cadence, development, allocation, articulation, and space vocabulary. Bundles are unordered option pools with melody_material=absent. They never compile a pitch sequence or provide the next note.

The local IA library is outside production runtime

Production agents cannot query local song paths, titles, per-track features, nearest neighbors, reference melodies, continuations, or post-composition percentile fits. Offline library work may only motivate human-reviewed, non-reconstructive phrase-grammar vocabulary.

Cold-start compositions

These tracked packages contain a canonical manifest snapshot, append-only creative journal, readable report, and a digest-bound index. Final WAV files remain local-only.

Composition	Voice	Scale	Record
朝がほどく前に	IA_ROCKS	28-second intent-first demo, 11 decisions	record
遠い灯、青い夜	IA_ROCKS	2:10 cold-start song, 13 decisions	record
風は名を呼ばない	original IA	3:10 fresh-agent song, 23 decisions	record
雨と蝉のあいだ	IA_ROCKS	64 seconds at 190 BPM, 24 decisions	record

Native projects that are safe to publish without the local audio workspace live in artifacts/vsqxs/.

Quick start

Requirements

• Node.js 20 or later.
• TypeScript toolchain installed through npm.
• xmllint for schema validation.
• FFmpeg for analysis/mixing/mastering.
• FluidSynth plus a compatible SoundFont for the deterministic draft accompaniment backend.
• For actual singing: your own legitimate Windows installation of VOCALOID3 Editor and a properly installed/licensed voicebank.

This repository does not contain VOCALOID binaries, voicebanks, activation data, or final song WAV files.

Build and test

npm install
npm run build
npm test

Start the MCP server

VOCALOID_MCP_PROFILE=production npm start

Generic local MCP client configuration:

{
  "mcpServers": {
    "vocaloid": {
      "command": "node",
      "args": ["/absolute/path/to/vocaloid/dist/src/index.js"],
      "env": {
        "VOCALOID_MCP_PROFILE": "production"
      }
    }
  }
}

The server uses MCP over stdio. It is designed as a local single-user desktop bridge, not as a remotely exposed multi-tenant service.

Render a reusable composition-agent prompt

Copy the template and write one bounded creative brief:

cp intents/template.intent.json intents/my-song.intent.json
npm run --silent compose:prompt -- --intent=intents/my-song.intent.json

The brief requires a BPM range and supports an optional preferred BPM inside that range:

{
  "tempo_bpm": {
    "min": 180,
    "max": 200,
    "preferred": 190
  }
}

It also declares exact voice component identity, duration, scene, dramatic motion, listener question, must-avoid constraints, and corpus/reference boundaries.

Evidence ladder

Never collapse these levels:

declared     a gesture or intent exists in the canonical state
materialized native VSQX notes / styles / controls contain it
rendered     the original Editor produced non-silent bound audio
heard        a named listener answered one explicit question

A valid VSQX does not prove a singer rendered. A non-silent WAV does not prove singer identity. Native PIT/DYN/vibrato data does not prove tasteful delivery. Technical QC never self-signs a human keep decision.

Repository artifact policy

The public repository intentionally tracks durable, Git-safe evidence and excludes the multi-gigabyte local production workspace.

Path	Policy	Contents
artifacts/songs/	ignored	final local listening WAV files
artifacts/evals/	ignored	stems, previews, A/B audio, debug projects, traces
build/	ignored	canonical working state, cache, temporary renders
artifacts/creative-records/	tracked	manifest snapshots, journal chains, reports, digest indexes
artifacts/vsqxs/	tracked	native VSQX delivery projects
notes/	tracked	contracts, evaluations, failure evidence
blogs/	tracked	long-form engineering and creative retrospectives

VSQX references backing audio by path; it does not embed that WAV. Creative-record indexes retain final artifact digests without copying the media into Git.

Repository map

src/
  index.ts                 MCP profiles, tools, resources, prompt
  song-manifest.ts         canonical state, revisions, stable IDs, hashes
  theory/                  deterministic symbolic analysis and proposals
  intent/                  composition-intent schema and realization evidence
  phrase-grammar.ts        non-generative phrase vocabulary and diagnostics
  creative-journal.ts      append-only decisions and publishable records
  vocal-phrase.ts          Japanese rhythm-cell / mora compilation
  render.ts                original-Editor export boundary
  accompaniment.ts         deterministic draft accompaniment
  stems.ts                 isolated vocal/instrument rendering
  stem-mix.ts              effects, automation, alignment, summing
  audio-quality.ts         loudness, peaks, clipping, transparency
  vocal-analysis.ts        score-versus-render observations

native/
  vsq_probe.c              hash-pinned Yamaha native-loader inspection
  vocaloid_export.c        bounded Win32 export helper

prompts/                   reusable agent workflow
intents/                   strict per-song briefs
scripts/                   evaluations, rendering, prompt and MCP helpers
tests/                     deterministic and integration regression coverage
notes/                     detailed technical contracts and issue logs
blogs/                     full project story
artifacts/creative-records durable creative evidence
artifacts/vsqxs/           publishable native projects

Selected technical notes

• Music theory policy
• Composition intent contract
• Phrase grammar DSL
• Creative decision journal
• Japanese lyrics and gestures
• Canonical song manifest design
• Stem mix workflow
• Score-versus-render analysis
• Synthesis reverse engineering
• Stable problem/solution log

Safety and legal boundary

• Filesystem writes are restricted to configured roots; overwrites are explicit and revision-bound.
• Editor operations use a host-wide lease and only terminate processes owned by the active render transaction.
• Singer availability requires an actual original-Editor probe; VVD discovery alone is metadata evidence.
• The project does not patch activation checks, copy licensing markers, redistribute voice data, or replace the licensed synthesis engine.
• Binary reverse-engineering findings are tied to exact local hashes and documented as observations, not stable vendor APIs.
• The bundled/general GM SoundFont path is a reproducible draft renderer, not proof of authentic acoustic or rock performance.

Project status

The end-to-end local research prototype is complete for this phase. No claim is made that it is production-ready for arbitrary hosts, all voicebanks, or autonomous music release.

The most useful conclusion is deliberately modest:

Agents can already be strong composition assistants, production engineers, and experimental partners. The hit song is still not an engineering primitive.

And that is fine. This repository exists because making a real Vocaloid production system with an agent was fun — and because the failures turned out to be as interesting as the songs.

License

The source code and repository documentation are available under the MIT License. VOCALOID, voicebanks, Yamaha software, supplied lyrics, local corpus material, and other third-party assets remain subject to their respective owners and licenses; they are not relicensed or distributed by this repository.