Evals — Assertion-First AI Evaluation

What it is

An eval gives an AI an input, then applies assertions to its output to measure success (Anthropic's definition). A case is {id, prompt, assert:[...]}. Each assertion is either deterministic (code, fast/free) or model-graded (an LLM judge). Cases run multiple trials; we report pass^k (all trials pass — the honest metric for a reliability-critical agent) and pass@k (any trial passes). Everything routes through Inference.ts — subscription-billed, no API-key path, no external deps.

Grounded in Anthropic's current doctrine — Demystifying evals for AI agents, Define success criteria / develop tests, and the skill-creator {text, passed, evidence} assertion convention. The typed-assert layer is promptfoo-shaped but our own TS.

Freshness contract: "aligned to Anthropic's doctrine" is a live claim, not a snapshot. When designing a new suite class or touching the ## Doctrine section below, re-fetch the Demystifying-evals doc and flag where it has moved past what's encoded here. Advisory only — report divergence, never auto-adopt, and an unreachable URL never blocks a run.

The canonical path (v2)

Tool	Role
Tools/Assertions.ts	Deterministic assert engine: equals, contains, icontains, contains-all/any, regex, starts-with, ends-with, is-json, contains-json, max-length, min-length, each with not- negation. Sync, no model call.
Tools/Judge.ts	Model-graded asserts llm-rubric (1–5 → 0–1, threshold) and llm-assert (NL assertions → TRUE/FALSE/UNKNOWN). Forced-structured JSON verdict, reason-then-score, distinct judge level, Unknown→miss escape hatch.
Tools/EvalRunner.ts	Loads a suite, runs the agent-under-test per case (single-shot inference against the target system prompt), applies asserts, computes pass^k/pass@k, persists transcripts + latest.json.
Tools/SuiteManager.ts	Suite listing + saturation tracking.
Tools/FailureToTask.ts	Convert real failures into cases (seed from 20–50 real failures).

# Run a suite (USER-customization suites resolve before the skill's own)
bun run ${LIFEOS_SKILL_DIR}/Tools/EvalRunner.ts -s <suite> [-t trials] [--json]
# Sanity-check the assert engine / judge
bun run ${LIFEOS_SKILL_DIR}/Tools/Assertions.ts     # 16-case self-test
bun run ${LIFEOS_SKILL_DIR}/Tools/Judge.ts          # good-vs-bad discrimination

Workflow Routing

Workflow	Trigger	File
RunEval	"run the eval", "run suite", "evaluate this", "grade output"	Workflows/RunEval.md
CreateUseCase	"new eval", "create a suite", "eval for X", "what should I test"	Workflows/CreateUseCase.md
CreateJudge	"write a judge", "llm-rubric", "grading criteria", "judge prompt"	Workflows/CreateJudge.md
ComparePrompts	"compare prompts", "which prompt is better", "A/B this prompt"	Workflows/ComparePrompts.md
CompareModels	"compare models", "which model is better", "is the cheaper rung enough"	Workflows/CompareModels.md
ViewResults	"eval results", "how did it score", "show the last run", "saturation"	Workflows/ViewResults.md
CreateScenario	"create a scenario", "multi-turn eval", "scenario test"	Workflows/CreateScenario.md
RunScenario	"run the scenario", "run multi-turn"	Workflows/RunScenario.md

Suite / case schema (assertion-first)

name: my-suite
type: regression            # or capability
pass_threshold: 0.75
agent_level: medium         # agent-under-test inference level
judge_level: high           # judge != generator (Anthropic best practice)
trials: 3
# system_prompt: optional override; default = live system prompt + DA identity
cases:
  - id: descriptive_name
    prompt: "the user turn sent to the agent-under-test"
    assert:
      - type: not-contains       # deterministic
        value: "should work"
        weight: 1
      - type: llm-rubric         # model-graded, weighted for partial credit
        weight: 2
        value: "Does the output tie any done-claim to verification evidence?"
      - type: llm-assert
        weight: 1
        value: ["The output does not claim success without evidence"]
  - id: should_not_case          # balance: test should-do AND should-not
    negative: true
    prompt: "..."
    assert: [...]

Identity-bound suites (e.g. {{DA_NAME}}'s dispositions) live in LIFEOS/USER/CUSTOMIZATIONS/SKILLS/Evals/Suites/ — the public skill ships only generic suites/examples.

Doctrine (from Anthropic — encode, don't restate)

• Grade the output/outcome, not the path. Tool-call-sequence asserts are brittle and demoted to opt-in; the everyday suite grades what the agent produced. The legacy core-behaviors suite (tool-sequence graded) is retained only as an example of this anti-pattern — it is a v1 tasks: file and is not runnable by EvalRunner, which reports it as a named error rather than attempting it.
• Capability starts low (a hill to climb); regression targets ~100%; passing capability cases graduate into regression.
• pass^k for reliability, pass@k where one success suffices.
• Partial credit via assert weights. Balance should-do and should-not cases — one-sided evals create one-sided optimization.
• Judge discipline: distinct judge model, reason-then-score, forced structured verdict, an Unknown escape hatch.
• Never trust a score until you read transcripts — every run persists full case transcripts to MEMORY/STATE/Evals-Results/<suite>/<run>/run.json.

Harness integration

• Config-change regression: hooks/ConfigEvalFire.hook.ts → LIFEOS/TOOLS/ConfigEvalOnChange.ts fires the configured dispositions suite when a behaviour-defining file changes (default core-dispositions, the runnable v2 suite; override via LIFEOS/USER/CUSTOMIZATIONS/SKILLS/Evals/config.json config_change_suite — identity-bound suites live in that USER layer, never the public tree); regressions notify Pulse. Non-blocking, subscription-billed, debounced.
• ISA / Algorithm: an eval suite is the operational form of an ISA claim's falsifier (integration map kept on the maintainer machine — session notes, does not ship).

Legacy (v1, superseded)

The v1 grader-stack (Graders/, TrialRunner.ts) and the @langwatch/scenario path (ScenarioRunner.ts, LifeosAgentAdapter.ts, API-billed) predate the assertion-first rewrite. Prefer the v2 path above. The scenario path bills ANTHROPIC_API_KEY — do not use it for principal work.

Gotchas

• Single-shot agent-under-test narrates tool calls. Running the full agentic system prompt through tool-less inference makes the agent defer and simulate tool use instead of answering — which tanks "lead with the answer" style cases. EvalRunner injects an [EVALUATION CONTEXT] no tools, answer directly suffix to fix this; keep it when authoring output-graded disposition cases.
• judge_level must differ from agent_level (Anthropic: judge ≠ generator). Default agent=medium, judge=high.
• Unknown counts as a miss. A judge that can't verify an assertion returns UNKNOWN, scored as fail — conservative for regression, correct for gates.
• Deterministic asserts are free; use them first. Reserve model asserts (llm-rubric/llm-assert) for nuance a code check can't capture.
• is-json checks the whole output; contains-json checks for an embedded fragment. Don't use is-json on prose that merely mentions JSON.

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"Evals","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/LIFEOS/MEMORY/SKILLS/execution.jsonl