voice-mcp

Speech-to-text and text-to-speech as one small service, exposed over both an MCP (streamable-HTTP) interface and a plain REST interface.

• STT — faster-whisper on CPU (int8). Transcribe voice messages (ogg/opus, mp3, wav, m4a…) to text.
• TTS — Vietnamese speech via gTTS, with an optional Google Cloud TTS fallback. Server-side ffmpeg transcodes the mp3 to Ogg/Opus (libopus, 48 kbit/s) so the clip renders as a native Telegram voice bubble. Returns a URL to a cached .ogg, never a base64 blob. No bot needs ffmpeg — it is baked into the image.

This is the packaged evolution of the original single-file whisper-mcp (STT only); the STT behaviour and API are kept 100% backward-compatible.

About

voice-mcp gives AI agents (Claude Code, chat bots, automation) a single, self-hosted voice endpoint: send audio, get text back; send text, get a shareable voice clip back. It runs on a modest CPU VPS, keeps audio on your own infrastructure (nothing is sent to a third-party STT/TTS service beyond the optional cloud-TTS fallback you opt into), and exposes the same two capabilities over both MCP and REST so any tool can consume it. Built for Vietnamese first (Whisper + gTTS), usable for any language Whisper/gTTS support.

Interfaces

Route	Auth	Description
POST /mcp (MCP streamable-HTTP)	Bearer	tools: transcribe, speak
POST /transcribe {audio_base64, language}	Bearer	→ {text, language}
POST /speak {text, lang}	Bearer	→ {url, cached, engine, format}
GET /audio/<sha>.ogg	none	serves a cached .ogg/.mp3 (opaque hashed name)
GET /healthz	none	liveness / model info

MCP endpoint path is /mcp with no trailing slash.

speak (both the MCP tool and REST) returns a short URL string to an .ogg (Ogg/Opus) clip, never base64 audio — a base64 blob in a tool result would blow up the model's context window. The file is served by GET /audio/<sha>.ogg with Content-Type: audio/ogg (the same route also serves the intermediate .mp3 as audio/mpeg).

Run

Prebuilt image (GHCR)

docker run -d --name voice-mcp \
  -p 127.0.0.1:8767:8767 \
  -e MCP_TOKEN=your-secret \
  -e WHISPER_MODEL=small \
  -e PUBLIC_BASE_URL=https://whisper.veasy.vn \
  -v voice-models:/models \
  -v voice-audio:/audio \
  --memory 1200m --cpus 2 \
  ghcr.io/<owner>/voice-mcp:latest

Build locally

docker build -t voice-mcp .
# or, without Docker:
pip install -e ".[dev]"
MCP_TOKEN=dev PUBLIC_BASE_URL=http://localhost:8767 voice-mcp

Config (env)

See .env.example: MCP_TOKEN, WHISPER_MODEL, AUDIO_DIR, PUBLIC_BASE_URL, and the optional GOOGLE_APPLICATION_CREDENTIALS.

Model sizing

CPU int8, rough footprint / accuracy trade-off:

Model	RAM (approx)	Notes
tiny / base	~0.3–0.5 GB	fastest, lower accuracy
small	~0.7–1 GB	default — good VN accuracy / speed balance
medium	~2–3 GB	better accuracy, heavier
large-v3	~4–5 GB	best accuracy, needs plenty of RAM

Cap the container (e.g. --memory 1200m --cpus 2) on small hosts so it can't OOM the box; small fits comfortably.

TTS engine rationale

The provider chain lives in src/voice_mcp/tts.py:

1. gTTS (Google Translate TTS, lang="vi") — primary, free, no credentials, outputs mp3.
2. Google Cloud TTS (voice vi-VN-Wavenet-A, female) — fallback, only used when gTTS raises (e.g. HTTP 429). Requires GOOGLE_APPLICATION_CREDENTIALS; skipped entirely when unset.

Deliberately not used (do not re-add):

• Piper — voice quality rejected.
• edge-tts — Microsoft blocks datacenter IPs (403 from the VPS).

The mp3 is then transcoded to Ogg/Opus by ffmpeg (ffmpeg -y -loglevel error -i in.mp3 -c:a libopus -b:a 48k out.ogg), baked into the image. Both the .mp3 (kept for the chain) and the returned .ogg are cached content-addressed by sha256(text + lang + engine) in AUDIO_DIR, so repeated requests return the .ogg instantly without re-running gTTS or ffmpeg.

Serving audio

The app serves GET /audio/<file> itself (streams the .ogg/.mp3 with the matching Content-Type, no auth — the filename is an opaque hash), so no extra web-server config is required. For efficiency you can instead let nginx serve AUDIO_DIR directly:

location /audio/ {
    alias /path/to/audio/;
    add_header Cache-Control "public, max-age=31536000, immutable";
}

Speech-to-speech helper pattern

Chain the two tools for a voice-in → voice-out loop:

text = transcribe(audio_base64=clip_b64, language="vi")   # STT
reply = my_llm(text)                                       # your logic
url = speak(text=reply, lang="vi")                         # TTS -> Ogg/Opus URL
# download `url` (an .ogg) and send it back as a Telegram voice message

Development

pip install -e ".[dev]"
ruff check .
pytest        # pure-logic tests (cache keying, config parsing) — no network/models

Contributing

Contributions are welcome. See CONTRIBUTING.md for the dev setup, the ruff + pytest checks to run before a PR, and the design principles (two APIs / never return bulk audio inline / engines as a fallback chain).

License

MIT © Edward Nguyen