Cerebras — Wafer-Scale LLM Inference

Overview

Cerebras Inference serves open-weight models from its own hardware and exposes them at https://api.cerebras.ai/v1, compatible with the OpenAI chat-completions API. It is chosen for speed: time to first token and output rate are far above typical GPU hosting, which matters for interactive chat, autocomplete and multi-step agents.

The model catalog changes quickly and old IDs are removed. At the time of checking (October 2026) the Shared Inference models were:

Model ID	Notes
gpt-oss-120b	about 3,000 tokens/s; context 65k on the free tier, 131k paid; reasoning efforts low/medium/high (default medium)
qwen-3.8-27b	about 1,850 tokens/s; context 64k free, 128k paid; reasoning effort none/low/medium/high (default high)

Other models (for example gemma-4-31b) are Dedicated Inference only. Retired IDs include llama3.1-8b, llama-3.3-70b (the old skill text used llama3.3-70b), qwen-3-32b, qwen-3-235b-a22b-instruct-2507, zai-glm-4.7 and the Llama 4 models; the docs recommend gpt-oss-120b as the replacement for most of them. Always check https://inference-docs.cerebras.ai/models/overview and the deprecations page before hard-coding an ID, or list models at runtime with client.models.list().

Instructions

Install and authenticate

pip install --upgrade cerebras_cloud_sdk      # Python
npm install @cerebras/cerebras_cloud_sdk      # Node.js
export CEREBRAS_API_KEY="csk-..."             # key from cloud.cerebras.ai; keep it out of source control

The official SDKs read CEREBRAS_API_KEY automatically. Any OpenAI SDK also works by setting base_url="https://api.cerebras.ai/v1".

Chat and streaming (Python SDK)

from cerebras.cloud.sdk import Cerebras

client = Cerebras()  # uses CEREBRAS_API_KEY

response = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[
        {"role": "system", "content": "You are a concise coding assistant."},
        {"role": "user", "content": "Write a Python function that merges overlapping intervals."},
    ],
    max_completion_tokens=800,
    temperature=0.3,
)
print(response.choices[0].message.content)
print(response.time_info.completion_time, response.usage.completion_tokens)  # seconds, tokens

stream = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Explain B-trees in five sentences."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Usage and time_info (queue_time, prompt_time, completion_time, total_time) arrive in the final streamed chunk. Output speed is completion_tokens / completion_time. AsyncCerebras is the async client; the SDK retries connection errors, timeouts and 429s twice by default (max_retries changes it).

Chat with the OpenAI SDK (TypeScript)

import OpenAI from "openai";

const cerebras = new OpenAI({
  apiKey: process.env.CEREBRAS_API_KEY!,
  baseURL: "https://api.cerebras.ai/v1",
});

const stream = await cerebras.chat.completions.create({
  model: "qwen-3.8-27b",
  messages: [{ role: "user", content: "Suggest three names for a log-search CLI." }],
  stream: true,
});
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

Structured output

Prefer strict JSON schema (constrained decoding) over plain JSON mode:

ticket_schema = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": ["billing", "bug", "feature"]},
        "urgency": {"type": "integer"},
    },
    "required": ["category", "urgency"],
    "additionalProperties": False,
}
response = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Customer was charged twice for the March invoice, please fix today."}],
    response_format={"type": "json_schema", "json_schema": {"name": "ticket", "strict": True, "schema": ticket_schema}},
)

Schema limits: root must be an object, additionalProperties: false on every object, at most 5,000 characters and 10 nesting levels, arrays need items. {"type": "json_object"} only guarantees valid JSON and cannot be combined with streaming.

Tool calling

Supported by both shared models, with tool_choice of none, auto, required or a named function, optional "strict": true on a function, and parallel_tool_calls=True.

tools = [{
    "type": "function",
    "function": {
        "name": "get_order_status",
        "description": "Look up the shipping status of an order",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
            "additionalProperties": False,
        },
    },
}]
messages = [{"role": "user", "content": "Where is order ORD-48213?"}]
msg = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools).choices[0].message
while msg.tool_calls:
    messages.append(msg)
    for call in msg.tool_calls:
        result = lookup_order(**json.loads(call.function.arguments))  # your function
        messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
    msg = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools).choices[0].message
print(msg.content)

Loop until a response has no tool_calls. Do not combine tools with response_format unless the docs say it works for your model.

Reasoning control

Pass reasoning_effort ("low", "medium", "high"; Qwen also "none" to skip thinking for simple requests). reasoning_format can be parsed (reasoning in choices[0].message.reasoning), raw, hidden or none. Via the OpenAI SDK, put Cerebras-only fields such as reasoning_format and clear_thinking in extra_body. Reasoning tokens count toward max_completion_tokens, so raise the limit for hard problems.

Prompt caching

Automatic, no flags: prefixes are cached in 128-token blocks (guaranteed 5 minutes, sometimes up to an hour). Put system prompts, tool definitions and documents first and the user's turn last, and read usage.prompt_tokens_details.cached_tokens. Cached tokens do not count against the uncached TPM limit.

Examples

Example 1: Replace a retired model ID

User request: "Our service started returning 404 model_not_found from Cerebras with llama3.3-70b."

Check https://inference-docs.cerebras.ai/support/deprecation, then switch the model string to gpt-oss-120b, set reasoning_effort="low" where the old model was used for quick answers, and run one request to compare latency. Result: calls succeed again; replies may include reasoning, so use reasoning_format="hidden" or "parsed" if the old output shape must be preserved.

Example 2: Fast ticket triage endpoint

User request: "Classify incoming support emails into billing, bug or feature and return JSON in under a second."

Use the strict json_schema call above with qwen-3.8-27b, reasoning_effort="none", temperature=0 and a fixed system prompt first so the prefix is cached. Result: a schema-valid object such as {"category": "billing", "urgency": 2} on every call, with typical latency dominated by network time.

Guidelines

• Rate limits: the free trial allows 5 requests/minute, 30K uncached and 90K total tokens/minute and $5 of credit expiring after 30 days; pay-as-you-go raises this (for example 1,000 RPM and 1M uncached TPM on gpt-oss-120b, 300 RPM and 150K on qwen-3.8-27b). A 429 says which bucket was exceeded; back off and retry, or raise cache hit rates.
• Differences from OpenAI: n must be 1; images must be base64 data URIs, not external URLs; use stream: true rather than tool_stream.
• max_tokens is an alias of max_completion_tokens; send only one.
• Do not hard-code speed claims or model lists; both change. Measure time_info on your own prompts.
• Use it when latency or tokens per second dominate. For proprietary frontier models, very long context beyond the listed limits, or image generation, use another provider.
• Keep a fallback provider for outages and rate-limit bursts, and never log the API key.