<!-- SYNCED COPY — do not edit here. Canonical: common-docs/skills/test-model-selection/SKILL.md This file is distributed to every consuming repo by common-docs/meta/scripts/sync_skills.py. Edit the canonical, run the sync, and commit each repo. Edits made here are overwritten and lost. -->

test-model-selection — which model a coding agent tests with

Arman, 2026-10-10: "The point is to know what is being tested and adjust accordingly." Never pick a test model from memory. Ask what the test is for, then ask the catalog.

1. What are you testing?

The test is about	Purpose	Gets you
Plumbing: UI, persistence, wiring, tools, streaming. The answer's content does not matter.	plumbing	The cheapest unrestricted priced chat model that supports tool calls. Cheaper than Haiku.
Something that depends on the provider: its API, its streaming shape, its errors.	provider + name	The cheapest unrestricted model of THAT provider (Haiku for Anthropic).
The actual intelligence: answer quality, reasoning, an eval.	intelligence	The recommended default (Sonnet 5.5 today) plus the list of restricted models.

2. Ask (one call)

agent_catalog action=test_model purpose=plumbing
agent_catalog action=test_model purpose=provider provider=Anthropic
agent_catalog action=test_model purpose=intelligence

No MCP? From aidream/: uv run python scripts/test_model.py plumbing, or provider Anthropic, or intelligence. Both return the model id, name, price per 1M tokens in and out, and why. Use that model name as given.

Plumbing that does not need a fresh answer at all: don't call the model. Run the test in AI test mode (?ai_test_mode=stand_in on the app URL, or header X-Matrx-AI-Test-Mode: stand_in): automated AI steps answer from saved baselines and cost nothing. Never for an intelligence test, and you still owe one real run before calling an AI-backed flow done. ai-test-mode.

2b. Never test through General Chat

Arman, 2026-10-17: "General chat is for users who chat ... if you want a test agent, create a super cheap, fast simple agent." When a test needs any AI reply, run Test Reply (agent 8a4b7c8e-144f-4391-b5f0-3d2cd2ecee0f, plumbing model, reasoning low, 300 output tokens, no tools, about $0.00002 a reply): agent_run agent_id=8a4b7c8e-144f-4391-b5f0-3d2cd2ecee0f user_message="ok". The test seats' new-chat default is already Test Reply (user binding of chat.default_new_chat). General Chat stays held for agent seats. A test that needs an agent to WORK WITH DATA (tables, records, a connected app such as Notion) runs Test Data Agent (5f533f4f-325a-4983-8702-d3a824394e68, Claude Haiku 5.5, reasoning low, tools: records, table, ask_person). The plumbing 20B model could not drive the records tool (it asked for an export), so this one is the cheapest that does.

3. Never a restricted model

Max-tier, Fable, Astra, Mythos, and anything at or above the cost rating the answer reports (restricted_min_cost_rating, 6 today) are off limits for testing. An intelligence answer lists them in restricted_models. The top-tier gate refuses them for anyone not approved, and a test account that asks for one raises a spend alarm to the super admins (one per account per day). The guard warns; it does not block, so the refusal is not your signal. This skill is.

4. Keep the test small

Fresh short context, as few calls as prove the point, and no premium pipeline re-run to check a code fix that does not depend on the model's reasoning. Rules: ai-model-and-spend-rules §5.

5. Where the answer comes from

One function, ai.test_model_choice(purpose, provider), computed from the live catalog (active, priced offerings only). Knobs under ai.test_models override it: plumbing_model, provider_models, intelligence_model, restricted_min_cost_rating, restricted_name_pattern. Reviewed monthly; the catalog changes, so never hard-code the answer in a test.