404건 · 관련도순
Judge board subskill for simmer. Dispatches a panel of judges with different lenses, runs one deliberation round where they challenge each other's scores, then synthesizes consensus scores + single ASI. Drop-in replacement for simmer-judge that produces identical output format. Do not invoke directly — dispatched by the simmer orchestrator when JUDGE_MODE is board.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Review a pull request using repository checks, verified code references, and current official documentation. Assess correctness, security, structure, and unnecessary code, then post one consolidated GitHub review with inline findings and an evidence-based verdict.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Review and improve game attacks and spells using recorded footage and two independent blind judges. Separate scores for readability and art guide changes to animation timing, effects, lighting, and target reactions, with performance costs tracked alongside visual quality.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Adversarial verification of finished work. Treats any "done" as a set of claims, then re-runs the claimed verifications, diffs what actually changed, detects weakened tests and false completion claims, and delivers an evidence-based verdict (VERIFIED / VERIFIED WITH CAVEATS / REFUTED). Use after any agent or model claims work is complete - "/fable-judge", "judge this work", "verify what it did", "did that actually work?". Also runs the fable-method trap suite against a skill or model via "/fable-judge suite TARGET".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use when a diff needs correctness review — null-safety, edge cases, off-by-one, races, error handling — dispatched by /review-changes, /do-and-judge, /judge, even without 'judge'.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
LLM-judge stage of an alpha-mining pipeline: take a miner's deterministic candidate list, keep only real signal, drop noise, and propose additions to that miner's curated note. Use after any zero-token detector run. Reads only the candidate digest and the home note, never the corpus. Triggers: "/alpha-judge <miner>", "judge the candidates".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
AI 에이전트의 반복 작업을 설계·점검합니다. 기계로 판정할 완료 조건, 변경 금지 경계, 독립 검증과 재시도 상한을 정해 헛돌거나 통제를 벗어나는 일을 방지합니다.
Investigate one difficult mathematics problem through repeated rounds of independent agent work and review. Track proved, refuted, and unresolved claims, check proposed proofs, and produce a self-contained proof.md that clearly states what remains unproved.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Design systems that use language models to judge AI responses. Covers scoring criteria, comparisons between two answers, bias reduction, confidence calibration, and checks against human judgments to make automated quality assessments more consistent and traceable.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Launch a meta-judge then a judge sub-agent to evaluate results produced in the current conversation
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Prepare a lawyer for an adversarial proceeding the way Neal Katyal's "Harvey" prepared him for the SCOTUS tariffs argument: profile the tribunal from their prior opinions and questions, predict the specific questions you will face, map narrow "escape routes" each judge can walk through without abandoning prior commitments, and spar adversarially until only the strongest answers survive. The model is the sparring partner. The human still wins the case. Use when: preparing for oral argument, appellate argument, motion hearing, evidentiary hearing, deposition (taking or defending), arbitration, mediation, or any proceeding where a known decision-maker will fire questions at you in real time. Also useful for stress-testing a brief before filing. Proactively suggest when: the user mentions an upcoming hearing/argument with named judges or arbitrators, drafts a brief and wants it pressure-tested, asks "what will Justice X / Judge Y ask me," or talks about preparing for a specific bench.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Design well-structured agent loops with best-practice coaching and cross-model review gates before you run them. Use when the user wants to design, build, or set up an agent loop, iterative agent workflow, self-review loop, LLM-as-judge loop, multi-model council, reviewer/judge gate, or goal-driven looping process. Guides goal refinement, typed verification criteria, reviewer/judge selection, privacy boundaries, termination guards, and observability, then emits a RUN_IN_SESSION.md handoff prompt plus portable loop.yaml, loop.resolved.json, LOOP.md, and run-loop.py.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Independent LLM judge evaluates task completion separately from the executing agent, catching false success claims by reviewing task goal, actions taken, final state, and evidence. Produces PASS/FAIL with confidence score and reasoning.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.