Review real images or text outputs from an evaluation, score each one, and add notes about your preferences. The skill measures how often the judge agrees with you, builds a cited taste profile, and can rescore outputs using that profile.
Calibrates any eval against the principal's own taste: takes a task in plain words, finds the eval suites and targets that cover it and offers them as options, then shows N real outputs (images or text, default 16, any count like 4, 24 or 64) on a local review page, collects a 0–10 score per output plus optional typed notes, measures how well the judge agrees, distills a taste profile the judge reads from then on, and rescores to prove the change. USE WHEN calibrate, calibrate 16, calibrate 64, calibrate social posts, calibrate the art judge, calibrate this eval, show me outputs to rate, rate these images, teach the eval what I like, tune the eval to my taste, what does the judge get wrong about my taste, calibration session. NOT FOR measuring a suite's judge against existing labels without a session (Assay Calibrate), recording one verdict on one delivered artifact (ArtDirection Calibrate), or producing the artifacts themselves (Art).
Run score Not tested
See how to installGiven a task in plain language, the skill finds related evaluation suites or configured targets and offers options. For the target you choose, it displays real images or text outputs in a local review page, where you can score each item from 0 to 10 and add notes. For text, you can also see the prompt used to generate it. The skill measures how closely the existing judge matches your ratings, then summarizes your preferences in a cited Taste.json profile. The judge can use this profile in later evaluations, and the skill can rescore outputs so you can review the difference.
Use it when you want your preferences reflected in how images, posts, or other outputs are evaluated. It can show which outputs the judge rates highly, where its ratings differ from yours, and what changes after your taste profile is applied.
Initial setup requires configuration such as Config.md. Only ratings you submit through the review page are recorded as evaluation labels. Rescoring may run multiple vision judgments for each image. This skill does not create the items being evaluated or measure a judge against existing labels without a calibration session.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.
AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).
Not counted in the score. Rated by AI (OpenAI Codex) as of Oct 28, 2026. Treat it as a guide and read the original before installing.
Review the instructions the agent will receive before you install.
Contents: Ideal State · Intent Routing · Tool · How taste reaches the judge · Data & Config · Gotchas · Examples
A judge that disagrees with the principal is grading against someone else's taste. Calibrate closes that gap for anything that has an eval. Every Assay suite is a target with no setup: its stored run outputs are the pool and its judged asserts are the judge. Image intents join through a configured target. The principal names a task in plain words, discover ranks the targets and suites that cover it, and the principal picks one.
Outputs are shown blind (no judge scores on the page), and text outputs show the prompt that produced them. The principal scores each one 0–10 on a slider, with a note when it matters; a score at or above the target's likeAt (default 7) counts as a like for pass/fail agreement, and the raw score is kept. The labels answer three things: how often the current judge agrees, where its cut-off should sit to match the principal, and what the principal actually values. The last becomes Taste.json, which the judge reads on every later run.
LIFEOS/TOOLS/lib/Calibration.ts.| Intent | Trigger | File |
|---|---|---|
| Run | "calibrate", "calibrate 24 social posts", "show me outputs to rate" | Intents/Run.md |
| Report | "how well does the judge match me", "calibration report" | Intents/Report.md |
| Distill | "update the taste profile", "what have I told you I like" | Intents/Distill.md |
| AddTarget | "calibrate a new image intent", "add a calibration target" | Intents/AddTarget.md |
| File | Read when |
|---|---|
Tools/Calibrate.help.md | the full CLI contract behind the command table below |
Setup.md | first-time setup: the optional Config.md and PREFERENCES.md keys and a proof it works |
bun skills/Calibrate/Tools/Calibrate.ts <command> from the config root. Full contract: Tools/Calibrate.help.md.
| Command | What it does |
|---|---|
discover "<task>" | Rank configured targets and every eval suite against a plain-words task; unusable ones (no stored outputs) rank last |
targets | List configured targets |
pool --target <t> | Pool size, scored share, labeled share, score bands |
session --target <t> --count N --open | Serve the review page on loopback, block until submit (exit 0) or timeout (exit 3) |
report --target <t> [--session <id>] [--rescored] [--json] | Judge-vs-principal agreement |
taste-input --target <t> | Every current label with notes, for Distill |
taste --target <t> --write <draft.json> | Validate citations, stamp provenance, write Taste.json |
rescore --target <t> [--session <id>] [--repeats 1] [--dry-run] | Re-run the target's judge with the current taste |
--target takes a configured target or any eval suite name.
EvalRunner appends the rendered profile to every judged assert of a suite whose name (or calibration: field) matches a target with a Taste.json. calibration: none opts a suite out. A new profile changes the rubric text, so that suite's Assay calibration record goes stale until re-measured, which is intended.Critique.ts adds the profile to its rubric prompt whenever the brief carries "calibration": "<target>".LIFEOS/TOOLS/lib/CalibrateTaste.ts because both skills read it.Targets, labels, sessions and taste profiles live at <configRoot>/LIFEOS/USER/CUSTOMIZATIONS/SKILLS/Calibrate/ (configRoot is $CLAUDE_CONFIG_DIR when set, else ~/.claude). If the directory does not exist, the AddTarget intent creates it.
Under LIFEOS/USER/CUSTOMIZATIONS/SKILLS/Calibrate/Targets/<target>/ (override the root with LIFEOS_CALIBRATE_DIR; stored eval outputs are read from LIFEOS/MEMORY/STATE/Evals-Results/, override with LIFEOS_EVALS_RESULTS_DIR):
target.json: only for configured targets. Name, title, the question shown on the page, the owning skill and intent, pools, the judge brief and bar, plus optional selection (newest or spread), count (the session default, 16 when unset) and likeAt. Pool kinds are evals (suite: that suite's stored outputs with its judge's verdicts), frontmatter (published documents whose frontmatter names an image: the real published outputs; needs imageField and localRoot), evalround (EvalRound.json records, which carry the art panel's scores) and files (a glob of images or .txt/.md; groupPattern is a regex whose first capture names the source round). An eval suite used directly needs no target.json.Brief.json: the ArtDirection brief used by rescore on image targets. A target with neither a brief nor "judge": "evals" is labels-only: rescore skips it with that reason, and report measures the labels against the judge snapshot its pool carried.labels.jsonl: one principal label per line, append-only. Only the page's submit writes it.sessions/<id>.json, Taste.json, rescore.jsonl: tracked. artifacts/ (content-addressed copies of labeled outputs), rescore/ and hash-cache.json are local-only.Beside Targets/, two optional files hold the install's own values:
Config.md: frontmatter browser, the app session --open opens the page in (open -a <app>; default for the system default browser). With no Config.md the page opens in the system default browser and the tool warns, naming the file; a Config.md without the key stops the open with an error naming the file and key.PREFERENCES.md: the principal's standing notes on what they score first, read by AddTarget when it writes a target's question and brief. Labels and Taste.json stay the evidence.The installer puts a starter for both in LIFEOS/USER/CUSTOMIZATIONS/SKILLS/Calibrate/; if one is missing, bun ~/.claude/skills/LifeOS/Tools/ScaffoldUser.ts --apply copies any missing starter file into place and never overwrites one you have edited. First-time setup is in Setup.md.
Agreement floors come from LIFEOS/TOOLS/lib/Calibration.ts, the one definition the Assay skill uses too.
label_source.kind is always principal and always came from the page.artifacts/, and labels point at the copy, so rescoring still works after the source is gone.History.md when present).report --rescored prints which taste version judged each output and leaves out every label that version was distilled from.--dry-run prints the count first.Example 1: Any task with an eval
User: "calibrate 20 story explanations"
→ discover ranks storyexplanation-cse first; one clear match, so it is used and named
→ page shows 20 stored outputs, each with the prompt that produced it
→ rescore, report, then Distill; the suite's judge reads the taste on its next run
Example 2: An ambiguous task
User: "calibrate the voice stuff"
→ discover returns several voice suites; AskUserQuestion offers the top four with output counts
Example 3: Checking the judge
User: "how well does the art judge match me now?"
→ Report: agreement, kappa, earned or not, best cut, per-vendor, disagreements with notes
Example 4: A new image intent
User: "calibrate the YouTube thumbnails too"
→ AddTarget: target.json with the thumbnail pools, question and brief; pool count; then Run
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Find and fetch vendor cybersecurity reports, then compare their findings on topics such as ransomware, breaches, cloud security, and AI. Analyses identify the reports consulted and cite each figure by vendor, year, and report, including where sources disagree.
Compare a component's own design needs with the goals of the larger system through three structured passes. The result identifies conflicts, recommends an appropriate scope, and assesses whether the proposed design fits the whole.
Find quotes that fit a text’s themes, with attribution and a short reason for each suggestion. Add sourced quotes, research thinkers, and organize a quote collection by theme and usage history to avoid repetition.
Create blog illustrations, technical diagrams, comparisons, comics, and icons through workflows tailored to each format. The skill guides composition, color, and model selection, then stages generated images for review before they are used in a project.
Assess the visual quality of images, films, slides, and site captures against a brief and craft criteria. Two vision judges score the work and rank defects, guiding bounded revision; delivery is gated on a passing critique, with human calibration labels recorded separately.
Find arXiv papers by topic, research category, or paper ID. Retrieve titles, authors, abstracts, and links, check new submissions, and add AlphaXiv's AI-generated overviews when available to help decide which papers to read.