Refresh the data tables in optimal-blog.md from master.db. Checks the data for integrity problems FIRST, then runs the generator that picks per-language winners and splices every GEN-marked table, then reconciles the surrounding prose. Use after new experiment results land, or when the optimal-blog numbers are stale.
The English summary is being prepared. Showing the original description.
See how to installReview the instructions the agent will receive before you install.
optimal-blog.md records what to run today, per language and task size. Its data
tables are not hand-written — they are generated from master.db by
the retort report optimal subcommand (code in
src/retort/reporting/optimal.py) and live between
<!-- GEN:<key> START/END --> markers. This skill is the safe procedure for refreshing
them: check the data before you trust it, regenerate, verify the round-trip, then fix
any prose whose numbers moved.
The order matters. master.db does not record the full stack/config (see the health gaps below), so a blind regenerate can silently publish wrong numbers. Always run the health check first and stop if it reports anything new.
A single cross-language reliability number is misleading — it blends a stack's strong languages with its weak ones (local Qwen passes Python/Go but fails Rust; Opus 4.8 dips on Java). The generator's centrepiece is the per-language success-rate matrix; the leading-stacks routine aggregate is explicitly labelled the least-useful number. Keep it that way — do not "promote" an aggregate back into the recommendation.
retort report optimal --health
Compare against the known, accepted gaps (already documented in the blog's Keeping this current section):
max_context_tokens unpopulated — the qualified config is
curated in FEATURED_STACKS, not filtered from data.model (local provenance bug) — attributed by experiment slug.experiment-11, experiment-29 not ingested.You MUST STOP and surface to the user if the report shows anything beyond those:
FEATURED_STACKS) or legacy
(add to KNOWN_NONFEATURED) before regenerating. Publishing without deciding would drop
it silently.src/retort/playpen/runner.py
stack_metadata(); if new blanks appear, that path is being bypassed.)retort report optimal --write optimal-blog.md
This splices every GEN:* block (leading stacks, per-language matrix, per-language
winners, prompt method). It only touches text between markers.
retort report optimal --write optimal-blog.md # run twice
git diff --stat optimal-blog.md
A second --write MUST produce no further change — the tables are idempotent. If
git diff shows table cells changed, that's the real new data; if it shows nothing, the
blog was already current.
The generator owns the tables, not the sentences around them. After a refresh, read the diff and fix any prose that quotes a number that moved:
†/‡ footnotes — reconcile its picks against the generated matrix and winner
table. A cell that flipped qualified↔unqualified (e.g. a language gaining local support)
changes the recommendation.You MUST NOT edit numbers inside the GEN markers by hand — re-run the generator instead.
Tell the user: which table cells changed, any prose you reconciled, and — if step 1 found anything — what you stopped on. If nothing changed, say the blog was already current.
--health and clear it against the known gaps before --write.GEN:* markers; the generator is the source of truth.--health reports an unmapped model — add it to FEATURED_STACKS (with a models
list and short column name) if it should appear in the blog, or to KNOWN_NONFEATURED
if it's legacy/a serving variant. Re-run health until only the accepted gaps remain.
Blank-model count grew — the harness stopped recording model for some runs. Check
stack_metadata() is still called by every runner's provision()
(local_runner, metaharness_runner, docker_runner); a new runner or a bypass would
reintroduce the original bug. Do not paper over it in the generator.
A local language's number looks too low — confirm the curated selection in
FEATURED_STACKS still points at the tuned-config experiments, not an all-experiment
average (which includes early bad-config runs and understates the tuned reality).
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Use when working in a repository that uses bd or Beads for durable project task tracking, issue dependencies, blocker management, multi-session handoff, or shared work memory. Trigger when the user asks to find ready work, claim or close tasks, create follow-up work, inspect blockers, recover project context, or choose between local planning and persistent project tracking.
The English summary is being prepared. Showing the original description.
Compare evaluated runs in a retort experiment along factor dimensions. Surfaces effects of each factor, aggregates across replicates, and highlights cells that diverge qualitatively — complementing (not replacing) retort's ANOVA analysis.
The English summary is being prepared. Showing the original description.
Determine the TRUE cause of a failed retort run before attributing it. Ground-truth every failure (run its tests, read its agent logs, inspect its workspace) and classify it as an infrastructure false-fail, a genuine model miss, or an environment issue — never trust the gate verdict or a log signature alone. Use whenever a run is recorded failed (test_coverage=0 / gate fail), a pass rate looks low, or you're deciding whether a "failure" is real before reporting or proceeding.
The English summary is being prepared. Showing the original description.
Evaluate a single retort experiment run. Score the generated code against the task's TASK.md requirements, run its build and tests, compute metrics, and emit a structured evaluation report plus a machine-readable findings file.
The English summary is being prepared. Showing the original description.
Aggregate a retort run's findings.jsonl into a machine-readable assessment.json summary with severity counts, penalty score, requirement coverage, and top findings.
The English summary is being prepared. Showing the original description.
Summarize the architecture of code generated by a single retort run. Produces module-level structure, interfaces, and control flow in a form suitable for cross-run comparison — not a full codebase-summary.
The English summary is being prepared. Showing the original description.