AssemblyAI

Overview

AssemblyAI turns audio and video into text over a REST API and a streaming WebSocket, with official Python and Node.js SDKs. The pieces you combine:

• Pre-recorded transcription: submit a URL or file, the SDK polls until the job is completed or error. Models are chosen with speech_models (default ["universal-3-5-pro", "universal-2"], an ordered fallback list).
• Speech Understanding and Guardrails: options on the same request — speaker labels, sentiment, entities, key phrases, topics, content moderation, PII redaction.
• LLM Gateway: an OpenAI-style chat completions endpoint for summaries, chapters and questions about a transcript. It replaces LeMUR, which was shut down on 2026-03-31.
• Real-time STT: a WebSocket session (/v3/ws) that returns Turn events while audio is being spoken.

This skill targets Python SDK 1.x (1.6.1 at the time of writing). Version 1.0 removed every aai.Lemur* class and the microphone helper in assemblyai.extras; the old aai.RealtimeTranscriber (streaming v2) is gone as well.

Instructions

Step 1: Install and authenticate

pip install -U assemblyai
read -rs ASSEMBLYAI_API_KEY && export ASSEMBLYAI_API_KEY   # paste the key from the dashboard

import os
import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]
# EU data residency: aai.settings.base_url = "https://api.eu.assemblyai.com"

Step 2: Transcribe a file or URL

config = aai.TranscriptionConfig(speech_models=["universal-3-5-pro", "universal-2"])
transcriber = aai.Transcriber(config=config)

# A public URL, a local path, a pathlib.Path, bytes or an open binary file all work.
transcript = transcriber.transcribe("https://assembly.ai/wildfires.mp3", poll_timeout=600)

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.id)
print(transcript.text[:300])

A failed job is returned, not raised — always check status before reading text. poll_timeout (seconds) raises aai.TranscriptError if the job is still running; fetch it later with aai.Transcript.get_by_id(transcript_id). Use transcriber.submit(...) plus webhook_url= in the config to skip polling.

Step 3: Speaker labels and Speech Understanding

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,        # who said what -> transcript.utterances
    sentiment_analysis=True,    # POSITIVE / NEUTRAL / NEGATIVE per sentence
    entity_detection=True,      # people, places, organizations
    auto_highlights=True,       # key phrases
    iab_categories=True,        # topics -> transcript.iab_categories.summary
    content_safety=True,        # content moderation
    language_detection=True,
)
t = aai.Transcriber().transcribe("https://assembly.ai/wildfires.mp3", config)
if t.status == aai.TranscriptStatus.error:
    raise RuntimeError(t.error)

for utt in t.utterances:
    print(f"[{utt.start // 1000:>5}s] Speaker {utt.speaker}: {utt.text}")

for s in t.sentiment_analysis[:5]:
    print(s.sentiment.value, s.speaker, s.text[:80])

for entity in t.entities:
    print(entity.entity_type.value, entity.text)

for phrase in t.auto_highlights.results[:10]:
    print(phrase.rank, phrase.count, phrase.text)

for result in t.content_safety.results:
    for label in result.labels:
        print(label.label.value, f"{label.confidence:.2f}", result.text[:60])

If you know the number of speakers, pass speakers_expected=2; otherwise leave it out. Timestamps (start, end) are milliseconds.

Step 4: Redact personal data

config = aai.TranscriptionConfig(speaker_labels=True).set_redact_pii(
    policies=[
        aai.PIIRedactionPolicy.person_name,
        aai.PIIRedactionPolicy.phone_number,
        aai.PIIRedactionPolicy.email_address,
        aai.PIIRedactionPolicy.credit_card_number,
    ],
    substitution=aai.PIISubstitutionPolicy.entity_name,   # "[PERSON_NAME]" instead of "####"
    redact_audio=True,                                    # also produce a beeped audio file
)
t = aai.Transcriber().transcribe("./calls/support-2026-09-14.mp3", config)
print(t.text)
print(t.get_redacted_audio_url())    # link is valid for 24 hours

Step 5: Summaries, chapters and questions with LLM Gateway

The transcript parameters auto_chapters, summarization, summary_model and summary_type are deprecated. Send the transcript to LLM Gateway instead. The literal tag {{ transcript }} is replaced server-side with the text of transcript_id:

gateway = aai.LLMGateway()

completion = gateway.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{
        "role": "user",
        "content": "List the decisions and action items from this meeting, "
                   "one bullet each, with the owner's name.\n\n{{ transcript }}",
    }],
    transcript_id=transcript.id,
    max_tokens=1000,
)
print(completion.choices[0].message.content)

for model in gateway.models.list().data:    # exact, current model ids
    print(model.id)

Model ids are exact strings (claude-sonnet-4-6, gpt-5-mini, gemini-2.5-flash); list them rather than guessing. Without the SDK, POST the same JSON to https://llm-gateway.assemblyai.com/v1/chat/completions with the header Authorization: $ASSEMBLYAI_API_KEY (no Bearer). Errors raise aai.LLMGatewayError with .status_code and .request_id.

Step 6: Real-time transcription

The SDK no longer captures the microphone; read 16-bit mono PCM yourself and pass the chunks to stream(). This example uses pip install pyaudio, which compiles against PortAudio on Linux and macOS: install portaudio19-dev (Debian/Ubuntu) or brew install portaudio first.

import os
import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeError, RealTimeEvents, RealTimeParameters, RealTimeTranscriber,
    TerminationEvent, TurnEvent,
)

RATE, FRAMES = 16_000, 1_600      # 100 ms chunks; the API accepts 50-1000 ms

def microphone():
    audio = pyaudio.PyAudio()
    mic = audio.open(format=pyaudio.paInt16, channels=1, rate=RATE,
                     input=True, frames_per_buffer=FRAMES)
    try:
        while True:
            yield mic.read(FRAMES, exception_on_overflow=False)
    finally:
        mic.stop_stream(); mic.close(); audio.terminate()

def on_turn(client, event: TurnEvent):
    if event.end_of_turn:
        print(f"[final] {event.transcript}")
    else:
        print(f"\r{event.transcript}", end="")

def on_terminated(client, event: TerminationEvent):
    print(f"\nSession closed after {event.audio_duration_seconds}s of audio")

def on_error(client, error: RealTimeError):
    print(f"Streaming error {error.code}: {error}")

client = RealTimeTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
client.on(RealTimeEvents.Turn, on_turn)
client.on(RealTimeEvents.Termination, on_terminated)
client.on(RealTimeEvents.Error, on_error)
client.connect(RealTimeParameters(sample_rate=RATE, speech_model="universal-3-6-pro"))
try:
    client.stream(microphone())   # Ctrl+C to stop
except KeyboardInterrupt:
    pass
finally:
    client.disconnect(terminate=True)

Streaming models: universal-3-6-pro (default), universal-3-5-pro, universal-streaming-english, universal-streaming-multilingual. The former StreamingClient/StreamingParameters/StreamingEvents names still import as aliases of the RealTime* classes.

Feature reference

Feature	Config	Read the result from
Speaker labels	speaker_labels=True	transcript.utterances
Sentiment analysis	sentiment_analysis=True	transcript.sentiment_analysis
Entity detection	entity_detection=True	transcript.entities
Key phrases	auto_highlights=True	transcript.auto_highlights.results
Topic detection	iab_categories=True	transcript.iab_categories
Content moderation	content_safety=True	transcript.content_safety
Language detection	language_detection=True	transcript.language_code
PII redaction	.set_redact_pii(policies=[...])	transcript.text, get_redacted_audio_url()
Domain vocabulary	keyterms_prompt=["Kubernetes", "Grafana"]	better spelling in text
Subtitles	—	transcript.export_subtitles_srt(), export_subtitles_vtt()
Chapters, summaries, Q&A	LLM Gateway (Step 5)	completion.choices[0].message.content

Examples

Example 1: Podcast episode to speaker-labeled transcript and show notes

User prompt: "Transcribe episode 42 of our podcast with speaker labels and write show notes with the key takeaways."

import os
import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(speaker_labels=True, speakers_expected=2)
episode = aai.Transcriber().transcribe("./episodes/ep42-observability.mp3", config, poll_timeout=1800)
if episode.status == aai.TranscriptStatus.error:
    raise RuntimeError(episode.error)

with open("ep42-transcript.txt", "w") as out:
    for utt in episode.utterances:
        out.write(f"Speaker {utt.speaker}: {utt.text}\n\n")
with open("ep42.srt", "w") as out:
    out.write(episode.export_subtitles_srt())

notes = aai.LLMGateway().chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content":
        "Write podcast show notes in Markdown: a one-paragraph summary, five key "
        "takeaways as bullets, and the tools mentioned.\n\n{{ transcript }}"}],
    transcript_id=episode.id,
    max_tokens=1500,
)
print(notes.choices[0].message.content)

Result: ep42-transcript.txt with turns labeled Speaker A: and Speaker B:, an ep42.srt subtitle file, and Markdown show notes printed to the terminal.

Example 2: Sentiment report for a support call with personal data removed

User prompt: "Analyze yesterday's support call: who was negative and when? Customer names and card numbers must not appear in the output."

import os
from collections import Counter
import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speaker_labels=True, speakers_expected=2, sentiment_analysis=True,
).set_redact_pii(
    policies=[aai.PIIRedactionPolicy.person_name, aai.PIIRedactionPolicy.credit_card_number],
    substitution=aai.PIISubstitutionPolicy.entity_name,
)
call = aai.Transcriber().transcribe("./calls/support-2026-09-30-1412.wav", config)
if call.status == aai.TranscriptStatus.error:
    raise RuntimeError(call.error)

totals = Counter((s.speaker, s.sentiment.value) for s in call.sentiment_analysis)
for (speaker, sentiment), count in sorted(totals.items()):
    print(f"Speaker {speaker}: {sentiment} x{count}")

for s in call.sentiment_analysis:
    if s.sentiment == aai.SentimentType.negative:
        print(f"{s.start // 60000:02d}:{s.start // 1000 % 60:02d} Speaker {s.speaker}: {s.text}")

Result: a per-speaker count such as Speaker B: NEGATIVE x7, followed by each negative sentence with its timestamp, where every name reads [PERSON_NAME] and card numbers are replaced the same way.

Guidelines

• Do not write LeMUR code. The LeMUR API was shut down on 2026-03-31 and SDK 1.0 removed transcript.lemur.*, aai.Lemur and aai.LemurModel. Use LLM Gateway.
• Do not use aai.RealtimeTranscriber or wss://api.assemblyai.com/v2/realtime/ws; that is streaming v2. Use assemblyai.streaming.v3.
• Streaming is billed for the time the session is open, not for audio sent. Always call disconnect(terminate=True); an abandoned session runs until the 3-hour cap.
• Keep the API key on the server. For browsers and mobile apps, mint a short-lived token with RealTimeTranscriber(api_key=...).create_temporary_token(expires_in_seconds=60) and send only the token.
• REST authentication is the raw key in the Authorization header, without a Bearer prefix.
• Limits: up to 5 GB and 10 hours per file (2.2 GB for a local upload); redacted audio needs a source file under 1 GB. Free accounts run 5 transcription jobs in parallel, paid accounts 200 or more; extra jobs are queued, not rejected.
• Cost: new accounts get $50 of free credit for transcription features; LLM Gateway usage is not covered by it and is billed per token.
• Audio leaves your machine. For recordings that may not be sent to a third party, use a local model instead, and use set_redact_pii when transcripts are stored or shared.
• Parameters change between models. Check a flag against https://www.assemblyai.com/docs/llms.txt before relying on it — for example prompt and keyterms_prompt behave differently on Universal-3.5 Pro and Universal-2.