Voice Skill - Speech and Local Audio Processing

Scheduling

Goal

Generate and transcribe speech through the local Voicebox app, and process existing audio with local media tools. Voicebox owns synthesis engines and profiles; FFmpeg owns file conversion, duration adjustment, and loudness processing.

Intent signature

• User asks to generate speech, narrate text, produce a voiceover, create an mp3 or wav from text.
• User wants an audio file transcribed into text, meeting notes, or a transcript.
• User asks for a voice notification when a long task completes or a workflow step is blocked.
• Another skill needs local audio generation infrastructure.
• User needs existing speech fitted to a time window, resampled, converted, or loudness-normalized.

When to use

• Generating short notification audio for agent task completion or blockers.
• Producing voiceover, narration, or audio assets (mp3 or wav) for apps and content.
• Transcribing local audio files (mp3, wav, m4a, webm, flac) to Markdown.
• Comparing voice profiles by re-running the same text against different profile ids.
• Processing an existing audio asset, including speech used in an oma-video dubbing job.

When NOT to use

• Cloud TTS or high-fidelity multilingual cloud voices -> out of scope; future multi-vendor extension.
• Real-time microphone dictation loop in the terminal -> use Voicebox app's built-in hotkey dictation.
• Voice cloning sample upload and profile creation -> done in the Voicebox desktop app UI.
• Video editing and audio/video placement -> oma-video; music and sound design are out of scope.
• Stories Editor multi-voice timeline composition -> use the Voicebox app UI.

Expected inputs

• TTS: text (<= 5000 chars per call), optional profile id, optional engine, optional language, optional output path.
• STT: audio file path (absolute or relative to $CWD), optional language hint.
• Notification: short message (<= 240 chars), profile id resolved from config.
• Processing: existing audio path, requested operations, and any stated duration, sample rate, channel layout, loudness, or format constraints. Derive targets from the brief, not a universal mastering preset.

Expected outputs

• TTS: audio file (wav — Voicebox's native TTS output format; mp3 optional via a local ffmpeg transcode) at .agents/results/voice/{timestamp}-{shortid}/output.{wav|mp3} plus manifest.json.
• STT: transcript.md at .agents/results/voice/transcripts/{timestamp}-{shortid}/ plus manifest.json.
• Notification: ephemeral playback through Voicebox; no disk write by default.
• Processing: a separate processed audio file plus manifest.json containing source/output paths, operations, and measured audio properties. Keep the source intact. A generation id is present only when Voicebox actually generated the audio.

Dependencies

• Processing only: installed ffmpeg and ffprobe; no Voicebox app or voice profile is required.
• Generation/transcription: Voicebox desktop app installed and running locally.
• Generation/transcription only: Voicebox MCP registered (claude mcp add --transport http voicebox http://127.0.0.1:17493/mcp).
• TTS only: at least one voice profile created in the Voicebox app UI.
• TTS only: optionally pre-downloaded engine models for the selected profile.

Control-flow features

• Branches by mode (notify, asset, transcribe), language, and profile availability.
• Calls voicebox via MCP tools, with REST GET /health as the handshake probe.
• Reads input audio files and writes generated audio plus manifests.
• Caches discovered MCP tool names after the first successful tools/list.

Structural Flow

Entry

1. Detect the requested mode: notification, asset TTS, transcription, or existing-audio processing. For processing, follow resources/audio-processing.md directly, before any Voicebox health, profile, or model lookup.
2. Verify Voicebox is reachable via MCP handshake or GET /health.
3. On the first run only, call MCP tools/list and cache the resolved tool names.
4. For notification or asset TTS, resolve the target voice profile id. For transcription, validate the audio input and continue without a profile.

Scenes

1. PREPARE: Resolve the operation and its inputs. Voicebox modes use text/audio and language; TTS additionally needs a profile. File processing uses the source and requested media properties.
2. ACQUIRE: If a required signal is missing, run the clarification protocol once.
3. ACT: Invoke the appropriate MCP tool (TTS or STT), or process the existing asset with local FFmpeg commands.
4. VERIFY: Confirm the response carries audio output or transcript content. Validate manifest fields.
5. FINALIZE: Write manifest.json alongside the output. Report the path or transcript to the user.

Transitions

• For generation/transcription, if Voicebox is unreachable, surface the install or launch hint and exit. Do not attempt auto-relaunch. Existing-file processing continues without it.
• Loopback addresses refer to the process's own host or container. Invoke a host Voicebox service from the host, then transfer the resulting file to the task container; a container's 127.0.0.1 does not probe the host app.
• For timed narration, keep the raw generation. Select a language-capable available profile, measure the raw duration, and use resources/audio-processing.md for fitting. A neural engine or correct loudness does not establish natural pronunciation or prosody.
• If a TTS request has no usable profile, point the user at the Voicebox app UI to create a profile, then exit. A transcription request never needs a profile.
• If a TTS request exceeds 5000 chars, ask whether to truncate or split. Do not auto-chunk in v1.
• If an STT input exceeds 30 minutes, ask whether to proceed. Do not auto-split.
• If the selected engine model is not loaded, ask the user before triggering a download.

Failure and recovery

Failure	Recovery
Voicebox app not running	Print install/launch hint, exit code 5
No voice profile for TTS	Print "create a profile in Voicebox" hint, exit code 3
Engine model missing	Ask before triggering download
Output path outside $PWD	Use an explicitly requested path; ask only if the destination is ambiguous or overwrites unrelated data
TTS over 5000 chars	Ask the user to split or truncate
STT over 30 minutes	Confirm only if the requested duration or resource cost is unresolved
MCP tool name drift	Re-run tools/list and update the cache
SIGINT	Abort the MCP call, write no partial output
Local media executable missing	Report the missing tool for the requested operation; do not start Voicebox or replace audio with a placeholder
Speech cannot fit naturally	Retain the raw audio; adjust generation pacing when synthesis is available and authorized, or report the unresolved timing constraint instead of clipping words

Exit

• Success: audio file or transcript exists with a complete manifest, and the path is reported.
• Partial success: output exists but a guardrail warning is surfaced (length, disk, model fallback).
• Failure: no output, the blocker (auth, profile, engine, network) is explicit.

Logical Operations

Actions

Action	SSL primitive	Evidence
Validate mode and inputs	VALIDATE	Clarification protocol in execution-protocol.md
Resolve TTS voice profile	SELECT	voicebox_list_profiles + config defaults
Health check	READ	MCP handshake or GET /health
Generate speech	CALL_TOOL	MCP voicebox_speak
Transcribe audio	CALL_TOOL	MCP voicebox_transcribe
Write output and manifest	WRITE	Audio or transcript plus manifest.json
Inspect result	VALIDATE	Output presence, duration, manifest fields
Report result	NOTIFY	Final user-facing summary

Tools and instruments

• Voicebox MCP server at http://127.0.0.1:17493/mcp for generation/transcription; local FFmpeg and ffprobe for existing audio.
• REST surface for health and audio retrieval (GET /health, GET /audio/{generation_id}).
• Resource references: voice matrix, prompt tips, execution protocol, checklist.

Canonical command path

Dispatch existing-audio work to resources/audio-processing.md. The path below is for Voicebox generation/transcription only; timed TTS follows the processing reference after fetching the raw WAV.

# 1. MCP handshake or REST health
GET http://127.0.0.1:17493/health  ->  200 OK

# 2. Discover tool names on first run
MCP tools/list                      ->  cache real names

# 3. TTS only: resolve profile
MCP voicebox_list_profiles          ->  pick profile by name or config default

# 4. Generate or transcribe (STT skips profile lookup and model-status check)
MCP voicebox_speak     { text, profile, language?, engine?, personality? }
MCP voicebox_transcribe { audio_path | audio_base64, language?, model? }

# 5. Fetch the generated audio (MCP has no save-to-disk; TTS output is wav)
GET http://127.0.0.1:17493/audio/{generation_id}  ->  wav bytes

# 6. Persist output + manifest
.agents/results/voice/<timestamp>-<shortid>/output.wav + manifest.json
.agents/results/voice/transcripts/<timestamp>-<shortid>/transcript.md + manifest.json

MCP tool mapping (verified against Voicebox 0.5.0)

Use case	MCP tool	REST backing
TTS generation	voicebox_speak	POST /speak
STT transcription	voicebox_transcribe	POST /transcribe
Profile listing	voicebox_list_profiles	GET /profiles
Captures listing	voicebox_list_captures	GET /history (captures view)

Tools not exposed via MCP (REST only): model status (GET /models/status), audio file serving (GET /audio/{generation_id}), per-version audio (GET /audio/version/{version_id}). The skill calls those over loopback HTTP when needed.

Notes on voicebox_speak:

• Required: text. Optional: profile, engine, language, personality (bool).
• Audio plays on the user speakers and is saved to the Captures / History panel automatically. There is no save_to_disk toggle on the MCP tool itself; to persist a local copy, fetch GET /audio/{generation_id} (Voicebox TTS always stores wav).
• Without a default profile set in Voicebox Settings, profile= is required.

Notes on voicebox_transcribe:

• Accepts exactly one of audio_base64 or audio_path (loopback only). Optional language, model.

Resource scope

Scope	Resource target
LOCAL_FS	Input audio, generated audio, transcripts, manifests
PROCESS	Local Voicebox app (managed by the user), FFmpeg/ffprobe processing
NETWORK	Loopback HTTP to 127.0.0.1:17493 only
MEMORY	Cached MCP tool names, resolved profile metadata
CREDENTIALS	None. Voicebox is local and key-free.

Preconditions

• Processing: local ffmpeg and ffprobe can read the source asset.
• Generation/transcription: Voicebox app is running and the MCP handshake succeeds.
• TTS only: at least one voice profile exists.
• TTS only: the selected engine model is loaded or the user approves a download.
• Output directory is inside $PWD unless explicitly allowed.

Effects and side effects

• Creates audio files, transcripts, and manifests under .agents/results/voice/.
• Existing-audio processing runs local media subprocesses and preserves the input file.
• Triggers local Voicebox generation, which consumes CPU or GPU.
• May trigger an engine model download when the user approves.
• Does not call any cloud service. No external network traffic.

Guardrails

1. Voicebox modes: generation/transcription requires the MCP handshake or GET /health. On failure, exit with a one-shot install or launch hint; do not retry or auto-relaunch. Local processing does not require Voicebox.
2. Profile required for TTS only: resolve a profile and check the TTS engine only for notification or asset mode. Transcription proceeds with a valid audio input and optional STT model even when no profile exists.
3. Tool-name discovery: on the first Voicebox invocation, call MCP tools/list and cache the resolved names. Reuse the cache for subsequent calls in the same session.
4. Length limits: TTS calls cap at 5000 chars per call; warn at 2000. STT inputs cap at 30 minutes. v1 does not auto-chunk or auto-split.
5. Auto-invocation transparency: notifications fire automatically only when the active task exceeds auto_notify_after_sec (default 60s). This threshold is agent-enforced guidance — no hook measures task duration — so apply it by judgment when a long task completes or blocks. Always announce intent in one short line before generating audio.
6. Path safety: an explicitly requested output path authorizes writing there. Resolve ambiguity or unrelated-data replacement before the dependent write; preserve required CLI path flags.
7. Cancellation: SIGINT aborts the MCP call and writes no partial output.
8. Manifest required for persisted output: asset TTS and transcription modes write manifest.json with at minimum: skill, mode, voicebox_generation_id, text (or transcript_preview), profile, engine, language, format (TTS only), created_at. Processing records the source, operations, and measured output properties as specified in resources/audio-processing.md; it never invents Voicebox provenance. Notification mode is exempt because Voicebox Captures is its system of record and no disk output is written by default.
9. Out of scope: voice cloning UI, captures archive, stories editor, microphone dictation loop, and cloud vendors are intentionally not exposed.
10. No cost guard: Voicebox is free. The cost guardrail from oma-image does not apply.

Clarification protocol

Before invoking a TTS or STT call, the agent checks the following. If any required signal is missing, clarify with the user first.

TTS (asset mode) required:

• Text content provided?
• Voice profile id or tone description provided?

TTS strongly recommended:

• Language explicit or detectable from the text?
• Output format (wav default; mp3 requires a local ffmpeg transcode)?

STT required:

• Audio path provided and the file exists?
• Duration within 30 minutes, or user approves splitting?

Notification mode skips clarification. It uses notification_profile from config and language is auto-detected from the message.

Invocation

Standalone

/oma-voice "build succeeded, 4 minor warnings"
/oma-voice transcribe ~/Downloads/standup.m4a
/oma-voice --profile prof_warm_korean "다음 단계 진행 준비됐어요"

Shared infrastructure (other skills)

Other skills can request audio output by calling the same MCP tools directly, or by invoking /oma-voice with their text. There is no separate CLI; the skill is MCP-native.

References

• Voice engine matrix: resources/voice-matrix.md (TTS profile/engine selection)
• Prompt writing rules: resources/prompt-tips.md (TTS text)
• Execution protocol: resources/execution-protocol.md (Voicebox generation/transcription)
• Existing audio and timed narration: resources/audio-processing.md (processing, duration fitting, or requested loudness conversion)
• Pre-flight checklist: resources/checklist.md (Voicebox calls)
• Configuration: read the voice: section of .agents/oma-config.yaml first, then fall back to config/voice-config.yaml for any key it does not set. Both profiles ship as null and must be set per machine — write them to .agents/oma-config.yaml, since oma update overwrites the skill config.
• Context loading: ../_shared/core/context-loading.md
• Quality principles: ../_shared/core/quality-principles.md
• Design reference: ../../../docs/plans/designs/012-oma-voice.md (source repo only; absent in global-mode installs)