Results for “evaluation”

採用・人材戦略の専門スキル。職務記述書（JD）作成、採用計画策定、コンピテンシーモデル設計、 面接評価基準設計、オンボーディング計画をサポート。採用要件の明確化から内定後のフォローまで 一貫した人材獲得プロセスを支援。日英両言語のテンプレートを提供し、グローバル採用にも対応。 Use when: creating job descriptions, designing competency models, planning recruitment strategies, developing interview evaluation criteria, or creating onboarding plans. Triggers: "JD作成", "採用計画", "面接評価", "オンボーディング", "コンピテンシー", "人材獲得", "job description", "recruitment plan", "interview evaluation", "hiring", "talent acquisition"

The English summary is being prepared. Showing the original description.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

The English summary is being prepared. Showing the original description.

Evaluate recommender system quality for the CSX4207 Vinyl Record Store. Use whenever you must compute or report recommender metrics (Precision@k, Recall@k, HitRate@k, MRR, MAP@k, NDCG@k, coverage, diversity, novelty, serendipity, personalization), design an evaluation protocol/split, run a baseline comparison, or write the evaluation section of a course deliverable. Covers formulas, JavaScript reference implementations, and the reporting checklist.

The English summary is being prepared. Showing the original description.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

The English summary is being prepared. Showing the original description.

Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.

The English summary is being prepared. Showing the original description.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

The English summary is being prepared. Showing the original description.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

The English summary is being prepared. Showing the original description.

Rigorous benchmarking of multi-agent coordination systems: experiment design, statistical analysis, human evaluation protocols, and reproducible reporting. NOT FOR ML model evaluation (use llm-evaluation-harness), A/B testing for web products, survey design, or general data science.

The English summary is being prepared. Showing the original description.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

The English summary is being prepared. Showing the original description.

Create and run automated tests that score LLM answers against expected outputs or judging criteria. Monitor progress, inspect individual results, compare models on the same tests, and use score thresholds in CI, an automated checking workflow.

• Creating tests with expected answers
• Monitoring evaluation progress
• Inspecting failed samples and scores

Design evaluation tasks that check whether agents actually complete work. Define requests, simulated user behavior, starting conditions, and verifiable success criteria, then plan difficulty levels, training-data leakage checks, and ongoing maintenance.

• Creating evaluation tasks for real business workflows
• Reviewing success criteria and verification logic
• Designing checks for evaluation data leakage into training

Plan and review PufferLib reinforcement learning workflows, where agents learn through rewards. Covers version-specific environment checks, parallel execution, training and evaluation plans, plus checkpoint inspection before loading saved models.

• Choose APIs for a specific PufferLib version
• Check environment behavior before training
• Create training and evaluation plans with separate seeds

Prepare and check research documents for clinical decision-support evaluation, including evidence profiles, cohort tables, survival plans, and model assessments. Work with aggregate or synthetic inputs, record sources and review status, and keep outputs outside patient care.

• Preparing research model evaluation documents
• Checking evidence profiles and source links
• Reviewing survival analysis plans

Implement .NET workflows for Azure AI Foundry projects, including agent creation, resource connections, dataset uploads, model deployment inspection, search index management, and evaluations. Examples also cover authentication, asynchronous operations, and resource cleanup.

• Creating and running agents from a .NET application
• Inspecting resource connections and model deployments
• Uploading datasets and managing search indexes

Implement Java workflows for Azure AI Foundry projects using SDK examples for authentication, connected resources, datasets, search indexes, and model evaluations. The skill also covers client setup, paginated resource lists, and error handling.

• List connected Azure resources
• Create or update search indexes
• Implement access to model evaluations

Compare prompt revisions on a fixed set of test cases to identify regressions, meaning behavior that gets worse after a change. Record model settings, case-level evidence, missing results, costs, and evaluation limits for a reviewable decision.

• Comparing original and revised prompts
• Checking required JSON fields after prompt changes
• Reviewing tool selection and task completion

Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.

• Build an automated evaluation pipeline for AI responses
• Compare responses after changing a prompt or model
• Define shared scoring criteria for reviewers

Explains how PluginEval combines structural checks and experimental model assessments into quality scores. Helps interpret low scores, identify improvement areas, set marketplace thresholds, and explain what evaluation badges actually represent.

• Investigating why a skill received a low score
• Improving descriptions that define when a skill should run
• Setting score thresholds for marketplace listings

Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.

• Build an evaluation set from agent traces
• Create separate checks for each failure type
• Calibrate an LLM judge against human labels

Turn graded execution traces and production logs into model training examples and preferred-versus-rejected response pairs. Select useful records, remove sensitive information, exclude evaluation examples, and preserve links to the original runs.

• Convert high-scoring successful runs into training examples
• Use expert-corrected failures as training examples
• Pair passing and failing responses to the same task

Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.

• Setting up quality checks for Python AI applications
• Creating realistic scenarios for known failure modes
• Scoring full application runs with real language-model calls

Create and manage Arize datasets for evaluation and experiments using the ax CLI. Import records from files, append examples, inspect dataset versions, and export a chosen version for offline analysis, with checks for field consistency and export completeness.

• Creating evaluation datasets from files
• Adding examples to an existing test set
• Inspecting dataset metadata and versions

Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.

• Adding self-review and revision loops to AI outputs
• Separating generation from evaluation
• Refining generated code using test results

Plan technology strategy and engineering team growth with frameworks for technical debt, hiring, architecture decisions, and vendor evaluation. Use Python tools and decision templates to prioritize improvements, structure teams, and track engineering performance.

• Prioritizing technical debt reduction
• Planning hiring and engineering team structure
• Comparing technologies and vendors