October 23 provider integrations

Read the tool's current input schema before use. Announce tool, host, exact model, operation and quoted cost before production. No automatic host/model fallback. Use explicit output paths under the production project. Account access is not established merely by having a credential configured. Most new prices require a fresh account quote; do not use expired promotions as permanent prices.

Google and ElevenLabs migrations

• gemini_omni_video: direct Gemini Interactions API model is gemini-omni-1.1-flash. Old preview retired October 22. Continue editing with previous_interaction_id; do not substitute it inside a Veo request.
• veo_video: Google backend requires Vertex; fal retains its own contracts. Gemini API Veo has retired. Select Omni explicitly for the replacement workflow.
• google_imagen: new default gemini-nano-banana-2.1; Imagen models require Vertex. Native Gemini image models use generate_content, not Imagen predict.
• elevenlabs_tts: eleven_v4 uses HTTP Text to Dialogue with inputs and settings. Keep each narration request <=2,000 characters. next_text maps to future_text; context strings <=100 characters. Style/speed/speaker boost are not dialogue settings. eleven_v4_turbo uses the dialogue WebSocket; this file adapter does not support timestamps or continuity controls on Turbo. It waits for global is_final, not the per-turn marker. The current official realtime tutorial documents Turbo although the older WebSocket reference says v3-only.

Narration and conversion

• heygen_tts: model_id heygen-voice-1, heygen-voice-1-turbo, or heygen-voice-1-turbo-free. Pass an ACTIVE native public/workspace voice ID; operation=list_voices discovers native voices. Professional clones only work with the non-Turbo model. voice_type can prevalidate the category; the service validates actual ownership/status. Completed speech and SSE both write WAV. SSE parts are individual WAV containers, decoded before assembly. Timestamps become an alignment sidecar and word_timestamps. Free access ends November 30, 2026; failures never switch to paid. Rate rejection retries are bounded and honor short Retry-After delays; larger delays surface to the caller.
• heygen_catalog_tts: separate catalog IDs, /v3/voices/speech; explicitly choose engine (starfish, heygen_voice, elevenlabs). Native professional clone IDs do not belong here. Catalog output can be MP3 or WAV. Supply either suffix; the returned artifact path uses the actual response codec suffix and preserves the audio.
• heygen_voice_convert: ACTIVE instant workspace voice only, at most300 seconds. Exactly one audio_path (<=16 MB), audio_url or asset_id. Output WAV. Supply a stable idempotency_key if resuming after an ambiguous timeout; no automatic retry.
• grok_tts: xAI narration /v1/tts, not Grok Voice realtime. language=auto by default, voice_id defaults to eve; list_voices discovers voices. output_format is an object, e.g. {"codec":"wav","sample_rate":24000}; this file adapter exposes MP3/WAV, not headerless telephony codecs. Optional character alignment is saved.
• mistral_tts: model_id voxtral-mini-tts-3-0; exactly one voice_id or local reference_audio_path. mp3/wav/flac/opus. Inline emotion/pace tags and expressiveness; free-form style instructions are not available at launch. No realtime TTS claim.
• mai_tts: Azure Speech key/region, model_id MAI-Voice-2.1 or MAI-Voice-2.1-Flash. voice_id is a managed voice base such as en-US-Harper (the model suffix is added). The TTS selector translates speed and pitch through the Azure SSML adapter. This adapter does not provision personal voice clones or grant gated access.
• mistral_stt: Voxtral Mini/Small Transcribe3, local audio files; timestamps are segments, not words. Diarization implies segment timestamps. Raw transcript JSON is persisted, including speaker and usage metadata.
• elevenlabs_soundtrack: video_paths (<=10, combined <=200 MB / 600seconds), optional description/tags, add_sfx/use_audio_as_reference/separate_tracks. Separate tracks return playable files in a sibling *_tracks directory; never treat a ZIP as an audio file. The adapter rejects unsafe archive paths and oversized archives.

Images and video

Model	Explicit tools / hosts
Nano Banana2.1	google_imagen, gemini_fal_image, gemini_replicate_image
FLUX3 Image	bfl_image, flux3_fal_image, flux3_replicate_image
FLUX3 Max video	flux3_video (fal)
Vidu Q4	vidu_video (fal), image_to_video or reference_to_video; no text-only route
Fibo2	bria_image direct or bria_fal_image
MAI Image2.6 / Flash	mai_image, deployed Foundry models
Grok Imagine1.5 / Lite	grok_video direct xAI

fal and Replicate have checked-in endpoint schemas under tools/provider_contracts. Pass provider_params only when documented by that host. Do not copy fal fields to Replicate or direct APIs. Existing Pro image model defaults remain unchanged on aggregators; select the new model explicitly.

BFL bounding-box layouts are encoded in the prompt per BFL docs, not a fabricated boxes endpoint field. BFL supports up to10 reference images; width/height must be provided together. Its API reference is authoritative over older narrative examples. Bria persists structured metadata for later editing. Its shared endpoint can route some legacy combinations to older Fibo models; the direct tool rejects those combinations when advertising Fibo2. MAI requires model identity to match the configured deployment; PNG output and <=2,359,296 pixels with each side >=768.

Grok1.5 references are limited to720p. Lite has no reference audio, reference images, last_frame or keyframes. Full1.5 supports preset voice references; custom audio references remain partner gated. Mid-video pins must be inside duration and occupy different1/3-second slots. No model substitution on failure. fal/Replicate/BFL/Grok jobs expose resume metadata; save job_path before polling and resume pending jobs instead of submitting duplicate paid generations.

Realtime is separate from narration

realtime_audio is a bounded one-response file runner for Grok Voice3, Eleven Realtime Mini/Max and MAI-Transcribe-2-Streaming. Input is text or mono PCM16 WAV (16/24kHz; Eleven requires24kHz). Voice output is WAV; MAI output is event JSON. Input sending and event receiving run concurrently. MAI commits explicitly and waits for final transcription, not a commit acknowledgment. Eleven uses ephemeral session credentials and bursty input pacing for recorded files.

For live audio, VAD, interruptions, multiple turns, or function calls, use the context-managed tools.audio._realtime_session.RealtimeAudioSession transport: append_audio, commit, receive, request_response, cancel, and submit_tool_result. The caller owns the session lifecycle and tool approvals. Received tool calls are never executed automatically. The finite file runner rejects tools because it cannot continue them after closing the socket. No microphone UI, browser app, or voice-agent orchestration is implied by this transport layer. Secrets remain in memory. No realtime translation or Kling4 support was added.

Primary references

• https://ai.google.dev/gemini-api/docs/deprecations
• https://elevenlabs.io/docs/api-reference/text-to-dialogue/convert
• https://elevenlabs.io/docs/eleven-api/guides/how-to/websockets/realtime-tdd
• https://elevenlabs.io/docs/api-reference/music/video-to-music
• https://developers.heygen.com/docs/voices/heygen-voice-speech
• https://developers.heygen.com/docs/voices/speech-to-speech
• https://developers.heygen.com/reference/generate-speech
• https://docs.x.ai/developers/model-capabilities/audio/text-to-speech
• https://docs.x.ai/developers/model-capabilities/video/overview
• https://docs.mistral.ai/studio/audio/overview
• https://docs.bfl.ml/api-reference/utility/generate-an-image-with-flux-3
• https://docs.bria.ai/whats-new/fibo-2
• https://learn.microsoft.com/azure/ai-services/speech-service/mai-voices
• https://learn.microsoft.com/azure/foundry/foundry-models/how-to/use-foundry-models-mai-image
• https://learn.microsoft.com/azure/ai-services/speech-service/mai-transcribe-2-streaming-realtime
• https://elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility
• https://docs.x.ai/developers/rest-api-reference/inference/voice