Eval System

A clean, minimal benchmark system that runs production code against test cases and writes results to a co-located markdown artifact (eval.md). The eval.md is a judgment surface: it exists so a strong external model or a human can read the real outputs and judge quality, not so the eval can auto-declare a winner.

---

Core Principles & Rules

1. Import Production Logic Directly

• ALWAYS import and invoke the actual production function, prompt constants, and schemas from the feature file.
• NEVER duplicate prompts, system messages, or JSON schemas inside eval.ts.
• Why: Re-implementing prompts in the eval causes prompt drift. When production changes, a duplicated eval becomes a false benchmark testing obsolete behavior.

2. Benchmark Real Runs — Never Estimate

• ALWAYS execute every candidate model to capture ground-truth latency, tokens, and cost.
• NEVER estimate or hallucinate latency, token counts, or pricing in a report without running the models.
• Why: Models cannot predict real streaming latency, throughput, or cost. Only direct execution records the truth.

3. "Passed" (✅) Means Schema Valid — NOT High Quality

• ALWAYS treat the status column as execution correctness (did the call succeed and conform to the Zod schema?).
• NEVER assume a "Passed" status implies the output is accurate, relevant, or good.
• Why: A model can return 100% valid JSON conforming to a schema while outputting completely wrong or noisy content.

4. Never Self-Score Quality — Surface Raw Outputs for Independent Judgment

• NEVER put the generating model's own relevance/quality/confidence score — or a count derived from it (e.g. "candidates scored 80+") — in the summary as if it were a quality metric.
• ALWAYS render each model's full raw output verbatim so quality can be judged from the output itself, not from the model's self-assessment.
• Why: A model scoring its own output rewards quantity and confidence, not correctness. A model that emits 7 loosely-relevant items looks like it "beats" one that emits 4 precise ones, when the opposite is true. Self-scores are biased and not comparable across models. Real quality only emerges when a strong external judge (or human) reads the actual outputs side-by-side.

5. Summary Table = Objective Machine Metrics Only

• ALWAYS limit the summary table to metrics the machine measures directly: latency, input/output tokens, cost, and pass/fail (schema/execution status).
• ALWAYS caption any raw output count (e.g. "Items") as a count, never as "Relevant", "Quality", or "Score".
• Why: Objective metrics are trustworthy and comparable; quality labels sneak the model's biased self-judgment back into the leaderboard.

6. Control Output Constraints for Fair Comparison

• ALWAYS enforce identical output constraints (e.g. top_k, length limits, schema bounds) across candidate models when benchmarking.
• Why: Unconstrained models that generate 1,500 tokens of verbose fluff will appear unfairly slow and expensive compared to concise models emitting 100 tokens, distorting latency and cost comparisons.

7. Track Real Billed Costs (Including Reasoning Tokens)

• ALWAYS calculate cost using total billed tokens from API response metadata, including hidden reasoning/thinking completion tokens.
• Why: Modern reasoning models (e.g., Gemini Flash thinking or OpenAI o-series) consume reasoning tokens that don't show up in visible text length but directly impact API billing.

8. Support Model Selection & Filtering

• ALWAYS support running the production model only by default, all candidates via --matrix, or a subset via --models=a,b.
• Why: Running the full suite on every iteration wastes time and API credits when a developer only needs 2–3 target models.

9. Output to Co-located Markdown (eval.md)

• ALWAYS compile output into a string buffer and write to path.join(__dirname, 'eval.md').
• NEVER output results exclusively to stdout or an external database.
• Why: Co-located eval.md files diff cleanly in git, render in the editor, and persist benchmark history with zero infrastructure.

10. Direct Terminal Progress

• ALWAYS log progress lines during the run, prefix success with ✅ and failure with ❌, and log the final eval.md path.
• Why: CLI feedback tells the developer whether the run worked without tailing files mid-execution.

---

Decision Flowchart: Eval Workflow

Are you creating or running an evaluation?
├── Running an existing eval?
│   ├── Production model only: `bun path/to/eval.ts`
│   ├── All candidates:        `bun path/to/eval.ts --matrix`
│   └── Specific models only:  `bun path/to/eval.ts --models=model-a,model-b`
└── Creating a new eval?
    ├── Does an eval file exist in the target module folder?
    │   ├── Yes → Edit `TEST_CASES` / `CANDIDATE_MODELS` in existing `eval.ts`
    │   └── No  → Identify evaluation target type:
    │       ├── AI / LLM feature (single or multi-model) → Pattern A
    │       ├── Automated quality scoring needed         → Pattern B (independent judge)
    │       └── Sync / async utility function            → Pattern C

One file per feature. Keep a single eval.ts that runs the production model by default and expands to a matrix via flags — do not split into eval.ts + eval.matrix.ts. One script and one eval.md keeps test cases in a single source of truth and stops agents guessing which file to run.

---

Implementation Patterns

Pattern A: AI Feature Evaluation (eval.ts)

Default runs the production model; --matrix runs all candidates; --models= runs a subset. The summary table stays objective; raw outputs carry the quality signal.

import fs from 'fs/promises'
import path from 'path'
import { fileURLToPath } from 'url'
import { calculateModelCost } from '../client'
import { FEATURE_MODEL, runProductionFeature } from './generate-feature'

const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)

const TEST_CASES = [
  { name: 'TestCase1', input: { /* production function params */ } },
]

// Production default first, then comparison candidates
const CANDIDATE_MODELS = [
  FEATURE_MODEL,
  'google/gemini-3.6-flash',
  'openai/gpt-5.6-luna',
]

// Flags: --matrix (all candidates) | --models=a,b (subset) | none (production only)
const isMatrix = process.argv.includes('--matrix')
const requested = process.argv.find(a => a.startsWith('--models='))
  ?.split('=')[1]?.split(',').map(m => m.trim().toLowerCase())

const TEST_MODELS = requested
  ? CANDIDATE_MODELS.filter(m => requested.some(r => m.toLowerCase().includes(r)))
  : isMatrix
    ? CANDIDATE_MODELS
    : [FEATURE_MODEL]

interface ResultRow {
  model: string
  caseName: string
  timeMs: number
  cost: number
  inputTokens?: number
  outputTokens?: number
  itemCount?: number
  error?: string
  output?: string
}

async function runEval() {
  console.log(`Running evaluation across ${TEST_MODELS.length} model(s)...\n`)
  const results: ResultRow[] = []

  for (const model of TEST_MODELS) {
    for (const testCase of TEST_CASES) {
      const start = Date.now()
      try {
        const result = await runProductionFeature(testCase.input, { model })
        const timeMs = Date.now() - start
        results.push({
          model,
          caseName: testCase.name,
          timeMs,
          cost: calculateModelCost(model, result.usage),
          inputTokens: result.usage?.inputTokens,
          outputTokens: result.usage?.outputTokens,
          itemCount: Array.isArray(result.data) ? result.data.length : undefined,
          output: JSON.stringify(result.data, null, 2),
        })
        console.log(`✅ ${model} — ${testCase.name} (${timeMs}ms)`)
      } catch (err) {
        results.push({
          model,
          caseName: testCase.name,
          timeMs: Date.now() - start,
          cost: 0,
          error: err instanceof Error ? err.message : String(err),
        })
        console.log(`❌ ${model} — ${testCase.name}: ${err}`)
      }
    }
  }

  const md: string[] = []
  md.push('# Multi-Model Comparison Evaluation\n')

  // Summary: objective machine metrics ONLY. "Items" is a raw count, not a quality score.
  md.push('## Summary (objective metrics — judge quality from the raw outputs below)\n')
  md.push('| Model | Test Case | Latency (ms) | Tokens (In/Out) | Cost ($) | Items | Status |')
  md.push('| :--- | :--- | ---: | ---: | ---: | ---: | :--- |')
  results.forEach(r => {
    const tokens = r.inputTokens !== undefined ? `${r.inputTokens} / ${r.outputTokens}` : 'N/A'
    const items = r.itemCount !== undefined ? String(r.itemCount) : '—'
    md.push(`| \`${r.model}\` | ${r.caseName} | ${r.timeMs} | ${tokens} | $${r.cost.toFixed(5)} | ${items} | ${r.error ? '❌' : '✅'} |`)
  })

  // Raw outputs grouped by test case so models sit side-by-side for independent judgment.
  md.push('\n## Raw Outputs\n')
  for (const testCase of TEST_CASES) {
    md.push(`## ${testCase.name}\n`)
    results.filter(r => r.caseName === testCase.name).forEach(r => {
      md.push(`### \`${r.model}\` — ${r.timeMs}ms, $${r.cost.toFixed(5)}`)
      md.push(r.error ? `\n**Error**: ${r.error}\n` : '\n```json\n' + r.output + '\n```\n')
    })
    md.push('---\n')
  }

  const outputPath = path.join(__dirname, 'eval.md')
  await fs.writeFile(outputPath, md.join('\n'), 'utf-8')
  console.log(`\n✅ Results written to ${outputPath}`)
}

runEval().catch(console.error)

Pattern B: Independent Judge (optional, for automated quality scoring)

Only when you need a repeatable quality number. Run a SEPARATE strong judge model that reads all candidate outputs against a rubric. The judge must never be the model that generated the output.

import { generateObject } from 'ai'
import { z } from 'zod'
import { getModel } from '../client'

const JUDGE_MODEL = 'openai/gpt-5.6-sol' // strong model, distinct from candidates

// `outputs` = the raw candidate outputs collected in Pattern A
async function judge(testCasePrompt: string, outputs: { model: string; output: string }[]) {
  const { object } = await generateObject({
    model: getModel(JUDGE_MODEL),
    schema: z.object({
      rankings: z.array(z.object({
        model: z.string(),
        score: z.number().min(0).max(10),
        precision: z.string(),  // are the items directly relevant, not just plausible?
        rationale: z.string(),
      })),
    }),
    // Include the rubric: reward precision, directness, ranking quality; penalize
    // broad/redundant/wrong items. Do NOT reward emitting more candidates.
    prompt: `Task prompt:\n${testCasePrompt}\n\nCandidate outputs:\n${JSON.stringify(outputs, null, 2)}`,
  })
  return object.rankings
}

Pattern C: Pure Function / Utility Evaluation (eval.ts)

For algorithms, parsers, formatters, throughput, or accuracy assertions.

import fs from 'fs'
import path from 'path'
import { productionUtility } from './index'

const TEST_INPUT = 'example-input'
const ITERATIONS = 100_000

function runEval() {
  console.log('Running utility evaluation...\n')
  let md = `# Utility Evaluation\n\n`

  const actual = productionUtility(TEST_INPUT)
  const expected = 'expected-output'
  if (actual !== expected) {
    console.error(`❌ Assertion failed: expected '${expected}', got '${actual}'`)
    process.exit(1)
  }
  console.log('✅ Correctness assertions passed')

  const start = performance.now()
  for (let i = 0; i < ITERATIONS; i++) productionUtility(TEST_INPUT)
  const totalMs = performance.now() - start
  const perCallMs = totalMs / ITERATIONS

  md += `Total time: ${totalMs.toFixed(2)}ms\n`
  md += `Time per call: ${perCallMs.toFixed(6)}ms\n`
  md += `Calls per second: ${(1000 / perCallMs).toFixed(0)}\n`
  md += `Sample Output: '${actual}'\n`

  const outputPath = path.join(__dirname, 'eval.md')
  fs.writeFileSync(outputPath, md, 'utf-8')
  console.log(`\n✅ Results written to ${outputPath}`)
}

runEval()

---

Execution Checklist

1. Production model only: bun <path-to-module>/eval.ts.
2. Full matrix: bun <path-to-module>/eval.ts --matrix.
3. Specific candidates: bun <path-to-module>/eval.ts --models=gemini-3.6-flash,gpt-5.6-luna.
4. Read the raw outputs in eval.md (or run Pattern B's judge) to assess quality — never rank models by their own self-reported scores.
5. Inspect git diff <path-to-module>/eval.md to review latency, cost, and output changes before committing.