Run a manually approved live compatibility matrix for supported CLI agents and the models advertised by the server. It checks whether requests complete through LobeHub Desktop and the official provider path, then produces per-cell evidence and a structured report.
Manually runs the live LobeHub official-provider compatibility matrix across Claude Code, Codex, Grok Build, Kimi Code, Pi, and TRAE when explicitly invoked by the user.
Run score Not tested
See how to installRun live compatibility checks through LobeHub Desktop for Claude Code, Codex, Grok Build, Kimi Code, Pi, TRAE, and models offered by the server. Requests go from each agent’s CLI through the official provider path. The checks record whether a request and response complete, along with relay, provider, and model information. First list available combinations, then run the selected cells sequentially. Results include an evidence file for each cell and a summary report, making it possible to investigate failures at boundaries such as CLI startup, authentication, and request routing.
Use it to check compatibility between external CLI agents and official providers before or after a change, while limiting the run to selected agents or models.
Live runs are manual and require approval for scope and cost. A signed-in Electron environment and the selected CLIs are required. The skill does not install or update CLIs or perform OAuth sign-in. Runs require --confirm-live and process cells sequentially.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.
AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).
Not counted in the score. Rated by AI (OpenAI Codex) as of Oct 26, 2026. Treat it as a guide and read the original before installing.
Review the instructions the agent will receive before you install.
This project skill extends acceptance with one scenario: proving that every
server-advertised official model completes through each supported external CLI
agent and LobeHub's server-default provider binding.
It does not replace Acceptance. Use Acceptance for the plan, approval gate,
evidence contract, immutable rounds, publishing, and teardown. Use
.agents/acceptance/PROJECT.md for LobeHub's Electron launch, auth, CDP, and
multi-instance commands. This skill owns only the compatibility-matrix semantics
and its executable harness.
The user invokes this skill with /testing-heterogeneous-agents in Claude Code
or $testing-heterogeneous-agents in Codex. Do not automatically load or execute
it as part of development, debugging, acceptance, or release tasks, and do not
schedule it. A reference from another skill is not permission to invoke it.
Manual invocation still requires scope and cost approval before live requests.
The matrix covers:
getServerDefaultHeterogeneousCapability response;anthropic-messages or
openai-responses);It does not cover user-defined providers, user-managed API keys, general model quality, tool-use quality, or ordinary chat-agent execution. Add a separate scenario instead of silently widening this matrix.
HARNESS=.agents/skills/testing-heterogeneous-agents/scripts/official-smoke.mjs
# Discover the live matrix and installed CLI versions. Makes no model calls.
node "$HARNESS" list --json
# Narrow discovery when validating one change.
node "$HARNESS" list --agent claude-code,codex --model MODEL_ID --json
# Execute selected or all cells through the real official provider.
node "$HARNESS" run --confirm-live --topic-id TOPIC_ID --cdp 9222
Filters, topic id, and non-default CDP port are optional when the attached Electron window already supplies the intended scope.
Exit codes:
| Code | Meaning |
|---|---|
0 | Every selected cell passed |
1 | At least one selected cell failed |
2 | No failures, but at least one cell was blocked |
3 | Electron, authentication, topic, or capability preflight failed |
Read the acceptance skill, .agents/acceptance/PROCESS.md, and
.agents/acceptance/PROJECT.md. Follow their approval and teardown rules. This
scenario makes billable requests and creates server operation records, so never
infer permission to run it from a request to inspect or list the matrix.
Run list --json first. It reads the live capability response and probes only
already-installed CLI binaries with version/help flags. Use its exact cells as
the plan; do not maintain a second hard-coded model list.
Report missing CLIs as blocked. Do not install, update, or sign in to an external CLI during this workflow. To suppress Claude Code's background update paths even during detection, start or restart the Electron instance with:
DISABLE_AUTOUPDATER=1 DISABLE_UPDATES=1 \
.agents/acceptance/scripts/electron-dev.sh start
Use the Electron section of .agents/acceptance/PROJECT.md to attach to an
already-running signed-in instance or start one for this run. Confirm an active
personal topic or pass --topic-id. Never trigger an interactive OAuth flow.
For each live run, present the discovered agents/models and obtain the
Acceptance plan approval before passing --confirm-live. Run sequentially; do
not parallelize cells because the CLIs share profiles and provider quota.
For repeated manual runs of the same compatibility scope, reuse one stable Acceptance subject so every execution becomes another immutable round in the same history. Previous runs do not authorize another live run.
Run the selected matrix with --confirm-live. By default each CLI receives an
isolated workspace inside the report directory. Pass --cwd only when a real
project workspace is part of the compatibility claim.
The harness starts each cell idempotently in the renderer and polls it with short
host-side evaluations. Do not replace this with one long agent-browser eval:
long evaluations can be retried by the driver and duplicate live requests.
The harness writes an Acceptance-compatible result.json, one text evidence
artifact per cell, and a structured matrix table under:
.records/reports/<timestamp>-heterogeneous-official-provider-smoke/
Unless --report-dir selects another location. Inspect the report and evidence;
do not infer success from the process exit code alone. Publish it with the clean
production lh acceptance run ingest environment required by
.agents/acceptance/PROCESS.md. Confirm that every matrix cell has its required
text evidence and return only the stable Acceptance URL.
Use each failed cell's evidence to identify the first broken boundary:
Load the heterogeneous-agent skill for raw trace, adapter, IPC, persistence, or
terminal-event diagnosis. A model refusing one ingress but passing another is a
request-compatibility signal, not proof of an adapter bug.
run requires --confirm-live; never bypass this gate.The self-test uses a fully stubbed agent-browser; it launches no browser,
external CLI, or model request:
bash .agents/skills/testing-heterogeneous-agents/scripts/official-smoke.test.sh
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Use this skill to verify a delivery by exercising the real product through a suitable surface such as a CLI, web app, desktop app, or iOS Simulator. It helps create or find a verification plan, capture evidence, and publish an acceptance report.
Use this skill to add an AI model provider to LobeHub across runtime integration, model cards, settings, branding, documentation, and acceptance checks. It also covers provider documentation updates and distinguishes new providers from models added to an existing provider.
Use this skill to add a server-side environment variable that sets a default for a user setting. It walks through validation, server configuration and store merging, plus updating the example environment file and self-hosting documentation.
Provides guidance for registering hooks around an agent’s execution steps, tool calls, human approvals, context compression, and sub-agent calls. Hooks can observe events, supply local mock tool results, or use webhooks to allow or deny tool results under configured policies.
Build event-driven background workflows for agents by defining sources, interpreting them as signals, and running actions through policies. It also guides asynchronous handoffs, deduplication, scoped execution, completion receipts, and observability.
Use this skill to run end-to-end acceptance checks for bot channels in real desktop chat apps. It drives apps such as Discord, Slack, Telegram, WeChat, Lark, and QQ to send a message, wait for a response, and capture the result.