Run existing Harbor or FrontierHarness evaluations against a local or cloud LobeHub target. The workflow covers target checks, CLI setup, smoke runs, job execution or resume, and failure diagnosis across supported Docker, SSH, and Runta environments.
Run and diagnose existing Harbor or FrontierHarness evaluations against local production LobeHub or LobeHub Cloud, on a local Docker host, SSH worker, or Runta. Use for eval infrastructure, target preflight, lh CLI injection, Harbor smoke or job runs, resume, and failure triage. Excludes authoring Harbor tasks and product acceptance.
Run score Not tested
See how to installThis skill helps run existing Harbor or FrontierHarness evaluations against a local production LobeHub server or a cloud target, and diagnose problems along the way. It guides you through confirming the server, execution environment, CLI source, agent, evaluation repository, and credentials. It then uses target-specific read-only checks and a smoke run. For a local Docker host or SSH worker, it supports Harbor runs and resumes; for Runta, it follows the specified FrontierHarness workflow and collects results. It also helps narrow failures to infrastructure, LobeHub, the gateway, CLI, or evaluation task.
Use it when you already have an evaluation task and need to prepare the environment, resume a job, or find whether a failure comes from the server, gateway, CLI, or task.
A clear request is required before running or resuming a real job. For a local environment, confirm that port 3210 and evaluation ports cannot be reached from untrusted networks. Put the Cloud CLI API key in the evaluation repository’s ignored .env file. This skill does not create Harbor tasks or perform product acceptance evaluations.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.
AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).
Not counted in the score. Rated by AI (OpenAI Codex) as of Oct 26, 2026. Treat it as a guide and read the original before installing.
Review the instructions the agent will receive before you install.
Run existing Harbor evaluations against either this checkout's isolated local
production harness or a remote LobeHub target. Use create-task to author or
grade tasks and acceptance for product acceptance.
Before preparing or running anything, obtain these independent choices. Do not
use runta as a LobeHub server target or checkout as an execution environment:
local production server from this checkout, or
cloud/remote.checkout build from apps/cli, or published npm release.LH_AGENT_ID; local uses the seeded
user's builtin inbox agent unless the user explicitly selects a different
agent..env. Local bootstrap seeds its CLI key. Reuse a provider
credential already configured in LobeHub or the ignored local env; if the
smoke reports InvalidProviderAPIKey, report the resolved agent's
provider/model and ask only for that provider's real credential.3210 and
every configured eval infrastructure port are unreachable from the public
internet and other untrusted networks. Do not bootstrap the local stack
without confirmation.For the Runta execution environment, also collect the harness repository and
pinned commit, provider/key choice required by FrontierHarness, and the
Terminal-Bench versus DeepSWE task subset. The Lh Cloud agent's model remains
selected by LH_AGENT_ID; do not replace it with FrontierHarness's --model
unless the run is explicitly non-comparable.
Do not infer these choices. DeepSeek is only one provider example, not a required credential or model.
3210 with bun run start; never target a
dev server or port 3010. Compose owns infrastructure; LobeHub stays on the
host.docker-compose/eval/.env from .env.example only when absent; never
overwrite an existing file.inbox for a cloud server, choose a separate model, or override
the agent with DEFAULT_AGENT_CONFIG. A local server may resolve the isolated
seeded user's builtin inbox through the authenticated CLI. Never invent,
print, or commit secrets.Read exactly one reference for the selected LobeHub server:
When the execution environment is Runta, additionally read the Runta/FrontierHarness playbook. It overlays the cloud server route; it is not a third server target.
For FrontierHarness on a local Docker host or SSH worker, use
scripts/frontierharness-local.py start to launch Harbor/Pier queues and
collect to prepare results and usage for reporting. See each subcommand's --help.
CLI source is orthogonal to the server target and execution environment:
checkout injects the built apps/cli; npm installs the release package. For
local execution, run the selected server preflight and then
scripts/run-smoke.sh <local|cloud> <checkout|npm> ... before the external eval
repository's own job command. The maintained Runta workflow uses a checkout CLI
artifact, two restored suite runtimes, and a detached start/status/collect
workflow; do not substitute a normal Harbor job for it.
Harbor and Pier default to LH_RUN_MODE=agent. Set LH_RUN_MODE=task to create
one persistent LobeHub Task per trial and run it on the connected eval device;
operation polling, output collection, and device cleanup remain the same. Task
mode records usage with lh task usage; agent mode keeps the existing usage
evidence flow scoped to the operation.
The runner exports persisted topic interactions to agent/trajectory.json with
lh topic export during cleanup, including interrupted runs. This requires a
CLI/server with ATIF export support. Export currently supports single-agent
topics without threads or branches; failures are logged without changing the
trial result. The trajectory includes recorded reasoning, tool calls/results,
and message usage; it does not reconstruct runtime-only model context.
3210, migrations, API-key auth, or /api/version: LobeHub.8787/8788, gateway health, or service tokens: gateway.LH_CLI_SOURCE or apps/cli/dist.create-task before changing it.For Harbor commands beyond these scripts, consult the official
Harbor Skills, especially its
harbor-cli skill. This harness pins harbor==0.23.0; when guidance differs,
the pinned CLI's --help is authoritative.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Use this skill to verify a delivery by exercising the real product through a suitable surface such as a CLI, web app, desktop app, or iOS Simulator. It helps create or find a verification plan, capture evidence, and publish an acceptance report.
Use this skill to add an AI model provider to LobeHub across runtime integration, model cards, settings, branding, documentation, and acceptance checks. It also covers provider documentation updates and distinguishes new providers from models added to an existing provider.
Use this skill to add a server-side environment variable that sets a default for a user setting. It walks through validation, server configuration and store merging, plus updating the example environment file and self-hosting documentation.
Provides guidance for registering hooks around an agent’s execution steps, tool calls, human approvals, context compression, and sub-agent calls. Hooks can observe events, supply local mock tool results, or use webhooks to allow or deny tool results under configured policies.
Build event-driven background workflows for agents by defining sources, interpreting them as signals, and running actions through policies. It also guides asynchronous handoffs, deduplication, scoped execution, completion receipts, and observability.
Use this skill to run end-to-end acceptance checks for bot channels in real desktop chat apps. It drives apps such as Discord, Slack, Telegram, WeChat, Lark, and QQ to send a message, wait for a response, and capture the result.