Eval and Improve

Coding-agent workflow — a /slash-command your coding agent (Claude Code, Codex, …) runs while developing this repo. Not a runtime skill the deployed @context agent runs; those live in skills/.

You're running @context's eval suite, diagnosing every failure, fixing what's in scope, and stopping when all cases pass. Surface area is two files: evals/cases.py (declares cases) and evals/__main__.py (runner). A case applies up to four checks, deterministic ones first (they're the spine; the judge corroborates):

• structural — a zero-arg callable returning (passed, detail); when set, the agent is not run. Used by boundary_is_structural to assert the guest/owner toolset asymmetry with no model in the loop. Deterministic.
• expected_tool_calls — agno's ReliabilityEval asserts which tools fired. Deterministic.
• capture_only — for guest runs, asserts every tool that fired is on the capture-only allowlist (no read/act tool, checked at the trace level). Deterministic.
• criteria — agno's AgentAsJudgeEval (LLM rubric, binary pass/fail), optionally narrowed with judge_guidelines. Keep it decisive so it doesn't flake.

No custom DSL beyond those fields on the Case dataclass.

0. Preconditions

• Postgres reachable on 5432: nc -z localhost 5432 returns 0. If not, docker compose up -d context-db from the source repo. (docker compose ps is unreliable from worktrees or alternate clones.)
• Venv active: source .venv/bin/activate. If .venv doesn't exist (fresh checkout or worktree), run ./scripts/venv_setup.sh first. evals/cases.py imports the agents directly from agents/, so no AgentOS server has to be running.
• .env populated with OPENAI_API_KEY. evals/__main__.py calls evals.dotenv.load_dotenv() at startup, so you do not need to source .env first. Worktrees don't inherit .env (it's gitignored) — copy it from the source repo if missing.

1. Run the suite

python -m evals               # full suite, concise (response + judge verdicts)
python -m evals -v            # stream the full agent run with rich panels + eval tables
python -m evals --case <name> # single case while iterating

Output ends with a summary block. Exit code is 0 on all-pass, non-zero on any failure or error.

The full suite hits OpenAI and runs 1-3 minutes — run it in the background so the session stays responsive and you're notified on completion. Keep --case <name> runs in the foreground while iterating; they're quick.

The runner runs the whole suite in one event loop and closes the providers' async clients at the end, so the old RuntimeError: Event loop is closed teardown noise no longer appears. If a run still prints stray httpx/MCP shutdown lines, they're harmless — only the Eval Summary table and exit code count.

2. Diagnose each failure

For every failed case, decide which kind of failure it is and fix at the appropriate layer:

Symptom	Likely cause	Where to fix
Judge fails, "answer is right but missing X"	Agent's instructions don't push for X	agents/<slug>.py — tighten the rule
Judge fails, response is fabricated	Agent hallucinated when it should have said it didn't know	Add a "if you can't find a real source, say so plainly" rule to the agent's instructions
Reliability fails: "missing tool X"	Agent didn't call the expected tool on this prompt	(a) Strengthen the routing rule in instructions, OR (b) the case is too narrow — broaden expected_tool_calls or drop the assertion
Reliability fails: "additional tool Y called" with allow_additional_tool_calls=False	Agent fanned out beyond the case's expectation	Tighten the agent's instructions OR set allow_additional_tool_calls=True
Guest case unexpectedly reads data (or an owner case is capture-only)	Case user_id doesn't exercise the surface you think — the runner pins OWNER_ID=eval-owner, and only that id gets the owner toolset	Check the case's user_id against EVAL_OWNER in evals/cases.py
boundary_is_structural (the gate) fails	The owner/guest toolset asymmetry is genuinely broken — a guest's context_tools() now returns more than submit_update, or the owner lost a read tool. This is a real security regression	Diagnose context_tools() in agents/context.py and the guest branch / GUEST_TOOLS; never edit the gate's expectations to make it pass
capture ✗ on a guest case: "guest fired non-allowlisted tool(s)"	A guest run reached a read/act tool — the capture-only allowlist (CAPTURE_ONLY_TOOLS in agents/inbox.py) was bypassed. Real regression	Diagnose the toolset gate + enforce_capture_only hook; don't widen the allowlist to paper over it
Same case flips PASS/FAIL across consecutive runs with no code change	Judge variance — rubric is too loose	Re-run 2-3 times to confirm; if it keeps flipping, tighten the case's criteria (more specific, more falsifiable)
Single case fails on full suite but passes alone	Transient flake or upstream rate limit (429s, MCP shutdown traceback)	Re-run the case in isolation. If it passes, re-run the full suite. If 429s persist, back off — don't fix the agent.
Many cases fail at once	Broad regression — model swap, MCP server down, tool removed	Diagnose the root cause first; do NOT paper over with prompt edits
eval_db write errors	Postgres down or migration missing	Bring DB up; check docker logs context-db

Rule: never weaken a case to make it green. Edit a case only when the assertion was wrong (overspecified rubric, wrong tool name, mismatch with how the agent's tools are named today). Catching a real regression is the whole point.

Quick test for "wrong assertion vs. real regression": read the response yourself. If it looks correct against the user's intent but the rubric flagged a missing detail, the rubric was overspecified. If the response is genuinely wrong, the agent's instructions need work.

3. Fix scope

In scope for this skill:

• agents/<slug>.py — instructions, tools, model.
• evals/cases.py — when an assertion was genuinely wrong.
• One-line config flips in app/main.py if a case requires it (rare).

Out of scope (flag for the user, don't do):

• Removing cases.
• Editing db/ or app/ to make a case pass.
• Editing agno itself.

For agent quality issues that need fast iteration against a live container (cURL probes, instruction tweaks), hand off to the improve-agent skill — its autonomous probe loop is faster than running the full eval suite per change. If the change is user-driven (add a tool, fix a known bug), use the extend-agent skill instead.

4. Re-run and stop

After each fix, re-run the failing case. Fixed several at once? Re-run them concurrently — background each --case invocation and collect the results — rather than serially:

python -m evals --case <name>

When all targeted cases pass, run the full suite once more to confirm nothing regressed:

python -m evals

Stop when python -m evals exits 0 and prints an Eval Summary block. If a re-run aborts mid-stream (no summary, regardless of exit code), treat it as inconclusive — re-run before declaring green.

5. Add a new case (if needed)

If diagnosing a failure reveals a missing assertion, add it to evals/cases.py:

Case(
    name="<short_id>",
    agent=<the_agent>,
    input="<prompt>",
    user_id=EVAL_GUEST,  # omit for an owner-surface case (defaults to EVAL_OWNER)
    # Any combination of:
    criteria="<rubric describing a correct response>",
    judge_guidelines=("<keep the judge decisive — the one signal that matters>",),
    expected_tool_calls=("<tool_name>",),
    capture_only=True,  # guest cases: assert no read/act tool fired
)

For a deterministic, model-free assertion (like the toolset gate) set structural=<callable> instead of input/criteria; the runner skips the agent run and just calls it. Prefer anchoring security claims on structural / expected_tool_calls / capture_only and letting criteria corroborate — deterministic checks don't flake.

Run python -m evals --case <name> to confirm it passes against the current agent. Commit the new case alongside any fixes.

6. Track regressions over time

Every case logs to Postgres via db=eval_db. Connect your AgentOS at os.agno.com and view eval history — useful for catching slow drift on a weekly cron.

To run on a schedule, register the eval suite as a scheduled task on the AgentOS scheduler — see agno scheduler docs.

---

Reference: Case shape

@dataclass(frozen=True)
class Case:
    name: str
    agent: Agent
    input: str

    # Identity the run is made under. Defaults to the owner (full toolset);
    # any other id exercises the capture-only guest surface.
    user_id: str = EVAL_OWNER

    # Deterministic structural gate. When set, the agent is NOT run — the
    # callable returns (passed, detail). Used by boundary_is_structural.
    structural: Callable[[], tuple[bool, str]] | None = None

    # Judge (LLM rubric, binary pass/fail): set to enable.
    criteria: str | None = None
    judge_guidelines: tuple[str, ...] | None = None  # keeps the judge decisive

    # Reliability (tool-call assertion): set to enable.
    expected_tool_calls: tuple[str, ...] | None = None
    allow_additional_tool_calls: bool = True

    # Guest-run guard: assert every tool that fired is capture-only allowlisted.
    capture_only: bool = False

The runner calls agent.arun() once per case and feeds the response into every enabled check, so a case that sets several fields costs one agent run, not several. A structural case runs no agent at all.

Identity is part of the case. The runner pins OWNER_ID=eval-owner before importing the agent, so user_id decides which surface a case exercises: the default (EVAL_OWNER) gets the full owner toolset; any other id gets the capture-only guest surface — that's how the suite asserts the security boundary itself.