「llm judge」の検索結果

Use when the user wants to test their LLM/agent application for safety and security vulnerabilities — jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.

日本語の概要は準備中です。原文の説明を表示しています。

Use when the user wants to compare or benchmark multiple LLMs/agents arena-style but it's unclear which specific workflow fits — a general-purpose win-rate comparison on a custom task, or a benchmark specifically about reference/citation hallucination rate. Also use when the user mentions model arena, agent arena, pairwise model comparison, win-rate ranking, or comparing models on a task and hasn't specified whether that task is generic or about citation accuracy. This skill is the entry router for the arena-eval suite: it asks one diagnostic question when needed, then recommends the workflow or workflows needed to cover the request.

日本語の概要は準備中です。原文の説明を表示しています。

Build a repeatable eval loop that grades agent output with an LLM judge, so prompt/skill changes get scored against a baseline instead of eyeballed. Reuses loopkit's verifier subagent as the grader — do not build a new one.

日本語の概要は準備中です。原文の説明を表示しています。

Evaluate WooAIAssistant against a structured scenario suite with hard invariants + LLM-as-judge rubric scoring. Runs live against the demo store + gpt-5.1 via the woo-mobile-ai backend wrapper, writes a JSONL run record, compares against stored baselines, and surfaces regressions. Always delegated to a subagent so the main context only sees the markdown report.

日本語の概要は準備中です。原文の説明を表示しています。

Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel, inter-rater reliability targets and LLM-as-judge versus human adjudication.

日本語の概要は準備中です。原文の説明を表示しています。

Prompt engineering regression test matrix register: baseline outputs, semantic drift thresholds, judge evaluations, and golden dataset.

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

日本語の概要は準備中です。原文の説明を表示しています。

Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движки", "сколько стоит час записи", "какую модель взять для пайплайна", "benchmark the pipeline", "compare ASR engines".

日本語の概要は準備中です。原文の説明を表示しています。

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

日本語の概要は準備中です。原文の説明を表示しています。

Suede AI eval design and coverage audit: AI-SPEC, failure-mode rubric with severity scoring, concrete pass/fail eval cases, coverage and infrastructure scores, and mechanical acceptance gates. Use when a change ships LLM, RAG, agent, classifier, prompt, or generated-media behavior, or when asked to write evals for an AI feature, design test cases for a model surface, audit existing eval coverage, or judge whether AI behavior is safe to ship. No AI-SPEC means no eval plan, and no eval plan holds the recommended ship verdict. NOT FOR: reviewing or grading the implementation code behind the AI surface (use suede-code); wiring a passing suite into CI as a required check (use suede-ci-gate); UAT of the built feature beyond the eval suite (a private Suede Labs companion, not in this pack).

日本語の概要は準備中です。原文の説明を表示しています。

Designs LLM, agent, responsible-AI, and multimodal evals. Use when measuring quality, fairness, grounding, safety, or judge reliability. Not for test harnesses (qa-agent-testing).

日本語の概要は準備中です。原文の説明を表示しています。

Use when the user wants web research: gathering cited, multi-angle evidence on a specific question. Triggers on: "research X for me", "do web research on", "look up sources for", "find citations for", "gather evidence on", "what does the web say about X". Also invoked programmatically by other beagle skills (prfaq-beagle Ignition, brainstorm-beagle reference points, strategy-interview context grounding) via the companion contract. Does NOT trigger on codebase lookups ("find this function", "search the repo"), local file search, LLM-as-judge evaluation, or paywalled/auth-gated scraping. Produces a written plan, parallel-subagent findings, and a cited synthesis report on disk — never inline prose, never unsourced claims.

日本語の概要は準備中です。原文の説明を表示しています。

Expert guide for synthetic dataset generation, LLM-as-a-judge filtering, QLoRA fine-tuning (Unsloth), DPO alignment, and GGUF/Ollama export for local SLMs / Panduan ahli generasi data sintetis, fine-tuning QLoRA, DPO, dan ekspor GGUF/Ollama.

日本語の概要は準備中です。原文の説明を表示しています。

Find the places in Anabasis where TypeSafe's Jev, a cheap calibrated System One classifier (Noul, Choice, Score over a text state), could replace or sit beside an LLM call, an embedding classifier, a prose regex, a lexical ranking or a blind-judge labelling loop, and prove each one in shadow against recorded labels before anything moves. Sweeps the tree for the five shapes, puts every candidate through the authority filter (what owns the decision, can it touch a pass, a score, a denominator or a gate) and the jev-1.13 fit filter, then writes a surface card with a runnable Bun experiment. Use when asked where Jev, TypeSafe or a System One model fits, what it could replace, or to run the autoresearch feature-discovery loop over this repository's records.

日本語の概要は準備中です。原文の説明を表示しています。

Create and maintain AgentV YAML evaluation files for testing AI agent performance. Use this skill when creating new eval files, adding eval cases, or configuring custom evaluators (code validators or LLM judges) for agent testing workflows.

日本語の概要は準備中です。原文の説明を表示しています。

Set up LLM-as-judge evaluation for ORC workflows

日本語の概要は準備中です。原文の説明を表示しています。

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

日本語の概要は準備中です。原文の説明を表示しています。

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

日本語の概要は準備中です。原文の説明を表示しています。

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimi...

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

日本語の概要は準備中です。原文の説明を表示しています。

Mine any Telegram channel or chat for signal in one command: incremental zero-token scrape, deterministic detector shortlist, LLM judge for what is genuinely valuable to this owner, then linked vault notes, a database row, a map-of-content entry and reindex. Triggers: "/mine-channel <@channel|id>", "alpha from <channel>".

日本語の概要は準備中です。原文の説明を表示しています。

Run one full mining pass over an imported community-chat corpus: deterministic detector, LLM judge, harvest, review screen. Works on any imported corpus through a per-source config and is safe to re-run, being idempotent by message id. Triggers: "/community-alpha <source> [month]", "mine alpha from <community>".

日本語の概要は準備中です。原文の説明を表示しています。

Prompt engineering regression test matrix register: baseline outputs, semantic drift thresholds, judge evaluations, and golden dataset.

日本語の概要は準備中です。原文の説明を表示しています。