384 件 ・ 関連度順
概要と使いどころ
Judge board subskill for simmer. Dispatches a panel of judges with different lenses, runs one deliberation round where they challenge each other's scores, then synthesizes consensus scores + single ASI. Drop-in replacement for simmer-judge that produces identical output format. Do not invoke directly — dispatched by the simmer orchestrator when JUDGE_MODE is board.
日本語の概要は準備中です。原文の説明を表示しています。
Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
日本語の概要は準備中です。原文の説明を表示しています。
Evidence-first pull request judge that reviews a PR and posts one consolidated GitHub review with inline comments via the gh CLI. Runs the repo's own deterministic checks first, researches current official docs before any claim about external libraries or APIs, then reviews correctness, security, structural quality (code judo, spaghetti growth, file-size limits), and AI slop including useless code comments. Every finding must carry evidence, every comment passes a deterministic noise gate before posting, and the verdict (APPROVE, COMMENT, REQUEST_CHANGES) is weighed by findings. Use when asked to review a PR, judge this PR, review this branch or diff before merge, run the-judge, "revise esse PR", "faca o code review", or "julgue esse PR". Do NOT use for reviewing prose or documents, fixing CI failures, resolving merge conflicts, writing the fix itself, or responding to review comments (use gh-address-comments).
日本語の概要は準備中です。原文の説明を表示しています。
Adversarial verification of finished work. Treats any "done" as a set of claims, then re-runs the claimed verifications, diffs what actually changed, detects weakened tests and false completion claims, and delivers an evidence-based verdict (VERIFIED / VERIFIED WITH CAVEATS / REFUTED). Use after any agent or model claims work is complete - "/fable-judge", "judge this work", "verify what it did", "did that actually work?". Also runs the fable-method trap suite against a skill or model via "/fable-judge suite TARGET".
日本語の概要は準備中です。原文の説明を表示しています。
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
日本語の概要は準備中です。原文の説明を表示しています。
Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
日本語の概要は準備中です。原文の説明を表示しています。
Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.
日本語の概要は準備中です。原文の説明を表示しています。
Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
日本語の概要は準備中です。原文の説明を表示しています。
Use when a diff needs correctness review — null-safety, edge cases, off-by-one, races, error handling — dispatched by /review-changes, /do-and-judge, /judge, even without 'judge'.
日本語の概要は準備中です。原文の説明を表示しています。
LLM-judge stage of an alpha-mining pipeline: take a miner's deterministic candidate list, keep only real signal, drop noise, and propose additions to that miner's curated note. Use after any zero-token detector run. Reads only the candidate digest and the home note, never the corpus. Triggers: "/alpha-judge <miner>", "judge the candidates".
日本語の概要は準備中です。原文の説明を表示しています。
AIエージェントの反復作業を設計・点検し、機械で判定できる完了条件、変更禁止の境界、独立した検証、再試行上限を定めて空回りや暴走を防ぐスキル。
Work on one hard mathematics problem in rounds: each round a judge writes a few self-contained questions, independent workers answer them, and the judge keeps a ledger of what is proved, refuted and open, until the judge concludes or the rounds run out, and then a proof.md says plainly what is and is not proved. A run takes hours and dozens of worker runs. Usage: /math-proof:siege [NAME=value settings] <the problem, stated in full, or the path of a file holding it>.
日本語の概要は準備中です。原文の説明を表示しています。
Score a game's combat skills and spells out of 10 with blind judges, then raise the weakest to a bar, for timing and readability (does a skill read as itself, does the blow land when it lands) and for art (detail, light, trails, impact, art direction, judged against a reference such as Diablo IV). Covers filming casts on a stopped, stepped clock, contact strips and close-ups, anonymised two-judge rounds, the levers that moved scores (hero props made from a painted concept with image-to-3D, own schedules and keyed moves, painted additive textures, camera-facing ribbons, light outside the body), the traps that cost points, asset and load budgets, and shipping each skill as its own version. Use when the user asks to score, review, improve, polish or rebuild combat skills, spells or attacks; says a skill "doesn't make sense", "looks bad" or should look like another game; names a target score; or starts a "next skill" thread. Project specifics (review hooks, harness settings, where each skill's code lives, scores so far) go in an optional local references/<project>.md next to this skill.
日本語の概要は準備中です。原文の説明を表示しています。
Launch a meta-judge then a judge sub-agent to evaluate results produced in the current conversation
日本語の概要は準備中です。原文の説明を表示しています。
Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.
日本語の概要は準備中です。原文の説明を表示しています。
Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).
日本語の概要は準備中です。原文の説明を表示しています。
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.
日本語の概要は準備中です。原文の説明を表示しています。
Prepare a lawyer for an adversarial proceeding the way Neal Katyal's "Harvey" prepared him for the SCOTUS tariffs argument: profile the tribunal from their prior opinions and questions, predict the specific questions you will face, map narrow "escape routes" each judge can walk through without abandoning prior commitments, and spar adversarially until only the strongest answers survive. The model is the sparring partner. The human still wins the case. Use when: preparing for oral argument, appellate argument, motion hearing, evidentiary hearing, deposition (taking or defending), arbitration, mediation, or any proceeding where a known decision-maker will fire questions at you in real time. Also useful for stress-testing a brief before filing. Proactively suggest when: the user mentions an upcoming hearing/argument with named judges or arbitrators, drafts a brief and wants it pressure-tested, asks "what will Justice X / Judge Y ask me," or talks about preparing for a specific bench.
日本語の概要は準備中です。原文の説明を表示しています。
Design well-structured agent loops with best-practice coaching and cross-model review gates before you run them. Use when the user wants to design, build, or set up an agent loop, iterative agent workflow, self-review loop, LLM-as-judge loop, multi-model council, reviewer/judge gate, or goal-driven looping process. Guides goal refinement, typed verification criteria, reviewer/judge selection, privacy boundaries, termination guards, and observability, then emits a RUN_IN_SESSION.md handoff prompt plus portable loop.yaml, loop.resolved.json, LOOP.md, and run-loop.py.
日本語の概要は準備中です。原文の説明を表示しています。
Use when adding, changing or reviewing Anabasis's Main Judge: the battery review, its recorded disagreements and the advice they feed. Keeps the Judge outside verification, evidence-bound, separately reported, and unable to alter pass, acceptance, claimability or adoption.
日本語の概要は準備中です。原文の説明を表示しています。
Independent LLM judge evaluates task completion separately from the executing agent, catching false success claims by reviewing task goal, actions taken, final state, and evidence. Produces PASS/FAIL with confidence score and reasoning.
日本語の概要は準備中です。原文の説明を表示しています。
Use when user says "simmer this", "refine this", "hone this", "iterate on this", or asks to improve a specific artifact over multiple rounds. Runs an iterative refinement loop with investigation-first judges that read the code, understand the problem, and propose evidence-based improvements. Auto-selects single judge or multi-judge board based on complexity. Works on any artifact type: documents, prompts, specs, emails, creative writing, API designs, pipelines, codebases. Supports multi-file workspace targets, runnable evaluators, and open-ended optimization (model selection, pipeline topology, prompt tuning).
日本語の概要は準備中です。原文の説明を表示しています。
Judge subskill for simmer. Scores a candidate artifact against user-defined criteria on a 1-10 scale and produces ASI (highest-leverage direction) for the next generator round. Supports judge-only, runnable evaluator, and hybrid evaluation modes. Do not invoke directly — dispatched as a subagent by the simmer orchestrator.
日本語の概要は準備中です。原文の説明を表示しています。
Use when orchestrating implementer/judge subagents — form gate + nine modes (do-and-judge ±two-stage, steps/parallel/worktrees, competitively, debate, live-app-judge, adversarial-council).
日本語の概要は準備中です。原文の説明を表示しています。