Processor model parity

rust/sglang-processor renders chat requests for SGLang's Rust hosts (the Rust server, sgl-router, the renderer). A model is supported only when, for the same request body, the processor produces the same prompt text and the same prompt token ids as Python's OpenAIServingChat. Python is the reference, Dynamo is the implementation, and SGLang code exists only where the two differ.

Scope: the request -> prompt -> token ids path, plus the checks that let hosts serve the model's OpenAI requests through /generate (see "OpenAI layer"). Multimodal preprocessing is not covered here: render_request fails on media content parts, so hosts send those chats to the engine. The fixture generator serves every checkpoint text-only, so a media case records that rejection.

Done means

• All text should work: every text chat the engine accepts renders and tokenizes in the processor (sgl-router's AllText). Only what SGLang rejects or the processor cannot reproduce goes to the engine; list it in the PR.
• Chat and completions go through the engine's /generate, with the router rendering, tokenizing and parsing (lower_chat, lower_completion); the engine's OpenAI route is only the Unsupported fallback. That needs the cookbook's --reasoning-parser and --tool-call-parser ported. Verify the route against a live engine.

The worked example is DeepSeek-V4-Flash-0731: src/render/models/deepseek_v4.rs plus tests/fixtures/parity/deepseek-v4-flash-0731.json. Paths below are relative to rust/sglang-processor/ unless they start with python/.

Where a model's code goes

Piece	File	Needed when
Identity -> formatter	src/render/selection.rs (select_chat_formatter)	always; mirror the order of chat_encoding.resolve_chat_encoding_spec
Formatter variant	src/render/mod.rs (ChatFormatter, its render_request arm, plus the render_prompt, stop_strs and resolve_thinking arms)	always
SGLang-specific steps	src/render/models/<model>.rs, re-exported in models/mod.rs	only where Dynamo diverges from Python
Parity cases	tests/fixtures/parity/<model-id>.json	always
README row	the render table in README.md	always

No new test code: tests/parity.rs checks every fixture in the directory.

render_request(Value, ..) takes the SGLang request body, not Dynamo's OAIChatLikeRequest, because Python reads fields the trait lacks (task, continue_final_message, the reasoning object). Pick the shape by how Python renders the model:

• Python calls its own encoder (a non-None spec such as dsv4 or dsv32): give the model a dedicated ChatFormatter variant and render_request arm, like DeepSeek-V4. If selection.rs already wires the model to a Dynamo PromptFormatter, replace that branch.
• Python calls apply_chat_template (Jinja, spec None): give the model a variant holding Dynamo's Jinja PromptFormatter, picked in ChatFormatter::from_tokenizer_config from the template text (as parser/template_detection.py picks it, where it has a detector), and build the template context that _process_messages and _apply_jinja_template build, like GLM-5.3 (models/glm53.rs) and Qwen3.8 (models/qwen38.rs). Dynamo's formatter rewrites shapes its load-time probes find the template rejects (a later system turn, for one); SGLang raises on them, so check those before rendering.

Method

0. Pick the checkpoint. Use the one the model's cookbook page recommends (docs/cookbook/), pin its Hugging Face commit as revision, and cache it. Note whether its tokenizer ships a chat_template; that can change the spec.

1. Trace the Python path for the model. Write down every transformation between the HTTP body and prompt_ids, in order:

   • protocol.py::ChatCompletionRequest.normalize_reasoning_inputs: reasoning and reasoning_effort set the default thinking / enable_thinking. Reuse render/reasoning.rs for this rather than porting it again.
   • serving_chat._convert_to_internal_request: chat_template_kwargs.reasoning_effort replaces the request effort; server --default-chat-template-kwargs merge in.
   • chat_encoding.resolve_chat_encoding_spec: which branch renders the model (a spec such as dsv4, dsv32, kimi_k3, inkling, or None for Jinja). It also reads the server's --tool-call-parser and whether the tokenizer has a chat template. If the deployment depends on the parser, set tool_call_parser in the fixture, and add it to ChatFormatterOptions in the same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's effort profile reads encoding/encoding_dsv4.py).
   • That spec's branch in serving_chat._apply_jinja_template / _encode_messages: message dumping, content flattening, _handle_last_assistant_message, system insertion, tools, model-specific fields, the encoder or apply_chat_template call, and how the prompt is tokenized.
   • The encoder itself (for example encoding_dsv4.py), line by line. History rules live there: dropped turns, </think> on empty reasoning, argument formatting, and the errors it raises.

   This list is the divergence inventory.

2. Find Dynamo's counterpart. Check dynamo_renderer::native_formatter_for, the model's low-level encoder (for example deepseek::v4::encode_messages_with_options), or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own request semantics. Dynamo's OpenAI-level formatters apply their own defaults (they filter tools by tool_choice, inject response_format, and map effort), which often differ from SGLang's. Diff the low-level encoder against Python's for the rules in step 1.

3. Write one request case per inventory item in the fixture (name + request only), then generate the expected outputs from Python (below). Cover the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.

4. Implement. Start from a plain Dynamo call and run tests/parity.rs. For each failing case, add the smallest SGLang step that fixes it, with a one-line doc comment naming the Python source it mirrors. Never patch the fixture to match Rust.

5. Verify text offline, then token ids with the checkpoint cached (commands below). If sgl-router or the renderer uses the formatter, run their suites too.

Checklist (DeepSeek-V4 case that covers each)

• Message dump: roles lowercased; unknown and null fields dropped; user reduced to role and content; null content becomes ""; history tool calls keep only ToolCall's fields. (ignored_fields, tool_results; GLM-5.3's tool_call_extra_fields)
• Content parts: which part types survive and the join separator. (parts)
• Tools: all request tools or only those tool_choice allows; field order and defaults of Function.model_dump; message-level tools; empty lists; schema types _validate_request rewrites (varchar becomes string). (tools, tools_none, tools_named, message_tools, empty_message_tools; GLM-5.3's tool_schema_type_aliases)
• Template kwargs: check reserved names (apply_chat_template's own arguments) after the server's defaults merge in. (GLM-5.3's kwargs_tokenize, default_tokenize)
• Pydantic coercion: typed request fields are coerced before rendering. For example, strict: 1 and defer_loading: "false" become booleans, and strict: null is rejected. Probe the pydantic model in Python to get its exact rules; don't guess them. (tool_bool_coercion, tool_strict_null, continuation_coerced)
• Tool-call arguments: how the encoder wants them. DeepSeek-V4 takes a compact JSON string; other encoders format them their own way. Keep key order (serde_json preserve_order is declared in Cargo.toml). Floats must print as Python's json.dumps prints them (1e-06, 1e+16, 100.0). (tool_results, agentic_thinking, tool_float_values, history_float_arguments)
• Thinking: kwargs thinking > request effort (!= "none") > reasoning.enabled

  server default kwargs > SGLANG_DEFAULT_THINKING. enabled follows Python truthiness (1 is on); strings use Python's yes-word set. (thinking_false, thinking_none, drop_thinking_ignored, reasoning_enabled_numeric, reasoning_enable_string)

• Effort: precedence between kwargs, reasoning, reasoning_effort and env; the checkpoint's profile mapping. (effort_high, effort_max, kwargs_effort, effort_conflict, reasoning_object)
• Final assistant turn: becomes a user turn, or with continue_final_message a separately tokenized prefix. Check whether content is flattened before the split (it is for DeepSeek-V4). Run each check where Python runs it: a final assistant turn's tool calls are discarded, so object checks on their arguments belong after the split. Hosts reach the model through render_prompt too, so check their continuation handling as well. (final_assistant, continuation_parts, continuation_bos, continuation_only, final_tool_call_no_arguments, continuation_tool_call_null_arguments)
• Model-specific fields and turns: task placement, a system turn mid-conversation, inserted empty system turns. (task_action, task_after_developer, consecutive_task, mid_system)
• History: the encoder's rules for earlier turns: reasoning kept or dropped, whole turns dropped, closing tags on empty reasoning. (multi_turn, thinking_multi_turn)
• Tokens: Python's tokenizer.encode adds special tokens by default; the continuation prefix is encoded alone and loses a leading BOS (_append_assistant_prefix_to_prompt_ids). tests/parity.rs does both, using the fixture's bos_token_id; render_prompt returns the prefix as its own segment so hosts keep that boundary. (continuation_bos, continuation_after_text)
• Errors: reject what Python rejects. A case Python raises on is recorded with error, and tests/parity.rs then requires render_request to fail. That includes pydantic rejections and Python crashes (a 500 is still a rejection). (invalid_tool_arguments, continuation_only) If Dynamo rejects a shape Python accepts, say in the PR that hosts fall back to the engine for it.
• Known gaps: when a mismatch is inside Dynamo and out of the processor's reach, fix it upstream and mark the case known_gap with the reason. The test reports the case without failing, and fails once it matches so the marker gets removed. Check a newer Dynamo release first. (tool_float_values: Dynamo's deepseek::common::to_json)

Generating fixtures

tests/scripts/generate_parity.py loads the pinned snapshot and resolves the spec and per-checkpoint state with SGLang's own resolvers. It then validates every case with OpenAIServingChat._validate_request, which rewrites tool schemas, and renders it through the real _process_messages, under the case's default_chat_template_kwargs (the server's --default-chat-template-kwargs), and writes:

• prompt, token_count and token_sha256 for each case, or error when Python raises (name, request, default_chat_template_kwargs and known_gap are kept as written). An AttributeError aborts instead: the stub server lacks something, so extend it;
• the fixture-level config (the config.json fields the processor reads, plus resolved overrides such as the effort profile), bos_token_id, and for a Jinja checkpoint its chat_template, so the test renders offline.

cd rust/sglang-processor
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
    python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json

When a new spec needs more, extend serving_chat() in the script (one place):

• More server state: set the extra attribute that __init__ resolves, for example _dsv41_default_reasoning_effort.

A spec that tokenizes inside apply_chat_template(tokenize=True) (Kimi K3) is recorded by the template's text output. Its render_request returns the segments the checkpoint encodes, and tests/parity.rs encodes them one by one.

Pin revision to a commit hash and regenerate only on purpose.

Verify

cd rust
cargo test -p sglang-processor --locked                                       # text, offline
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked  # + token ids
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked

The token check loads the pinned commit's snapshot from the cache. When the snapshot is missing, the test prints token ids not checked (visible with -- --nocapture), so confirm that line is absent before claiming token parity. With the snapshot cached, the test also checks that the checkpoint resolves the fixture's recorded DeepSeek-V4 profile.

OpenAI layer

A host serves a model's OpenAI requests through /generate (src/openai/) only once its render parity holds. Two more fixtures then cover the rest of Python's OpenAI layer:

• tests/fixtures/reasoning_parity/<parser>.json, from tests/scripts/generate_reasoning_parity.py: SGLang's ReasoningParser on chunked outputs. A parser is served once src/think/models/ ports it.
• tests/fixtures/tool_parity/<parser>.json, from tests/scripts/generate_tool_parity.py: SGLang's FunctionCallParser on chunked outputs, replayed by the test in src/tool_call/models/mod.rs. A tool parser is served once src/tool_call/models/ ports it.
• tests/fixtures/openai_parity/<model-id>.json, from tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>: the /generate body OpenAIServingChat builds for each case, a live engine's output for it, and the response Python builds from that output. Launch the engine with the model's cookbook flags, including its parsers. Without the weights, mark cases synthetic and write the engine output by hand, as GLM-5.3's fixture does.

Pitfalls

• Dynamo version bumps change rendering silently: rerun every fixture after bumping dynamo-renderer or dynamo-tokenizers.
• /generate accepts only the API's sampling fields (proto/sglang/api/v1/sampling.proto). A field Python sets in-process, such as glm47's ebnf_full_assistant, cannot be sent, so such requests stay on the engine's route. Fixtures don't show this; a live engine does.
• Python mutates the request before rendering (_validate_request rewrites tool schemas), so fixture generators drive the server's own entry points.
• Env vars (SGLANG_DEFAULT_THINKING, SGLANG_DSV4_REASONING_EFFORT, SGLANG_DSV41_REASONING_EFFORT) are read per request, as in Python. The generator pins them and tests/parity.rs clears them, so fixtures do not depend on the machine.
• Typed hosts (the renderer's OAIChatLikeRequest path) cannot carry every field, such as task and message-level tools. Parity is defined on render_request; report host-adapter gaps separately rather than bending the model code.
• Keep model code in its own file: no model-specific branches in render/mod.rs beyond the dispatch arm.