'eval' 검색 결과

Build repeatable DeepEval tests for AI agents, chatbots, and applications that retrieve documents to answer questions. Reuse or generate evaluation datasets, inspect failures, and improve prompts, tools, or retrieval through repeated evaluation runs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add repeatable quality tests for an AI agent
• Evaluate chatbot behavior across multiple conversation turns
• Generate evaluation datasets from documentation

Add DeepEval tracing to Python AI applications so execution steps appear in Confident AI. Choose a supported integration or manual @observe instrumentation, then capture inputs, outputs, step types, tags, and metadata.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Trace execution steps in a Python AI application
• Select tracing integrations for an existing framework
• Add manual tracing with @observe

Set up a realtime evaluation folder with a suitable shared harness, the framework that runs evaluations. Prepare prompts, tool definitions, realistic starter data, and a README, then validate the setup with evaluation and test runs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Start a single-turn text-to-speech evaluation
• Set up evaluations using saved audio
• Build multi-turn simulations with mock tools

Evaluate code generation models with HumanEval, MBPP, and other benchmarks. Test generated programs, compare models across programming languages, and record pass@k scores, which estimate the chance that at least one of k solutions passes the tests.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing code generation models on shared benchmarks
• Measuring correctness across programming languages
• Evaluating models trained to follow instructions

Evaluate code generation models with HumanEval, MBPP, and multilingual benchmarks. The skill guides test execution and model comparisons using pass@k, an estimate of whether at least one of k generated solutions passes the tests.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare code generation models on shared benchmarks
• Measure code generation performance across languages
• Evaluate instruction-tuned coding models

Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Run benchmark evaluations on a local GPU
• Choose an evaluation framework and inference engine
• Check a small sample before a full evaluation

Add team-specific evaluation criteria to plugin-eval by designing a metric pack, a set of custom checks and measurements. Create its manifest and JSON-producing script, then run it through the evaluator using a consistent output structure.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add team-specific evaluation criteria
• Define custom checks and numerical metrics
• Build a metric-pack manifest and script

Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate AI agents by preparing test cases, scoring responses and tool use, investigating failures, and comparing results after changes. Supports built-in checks and custom scoring rules, including evaluations performed by a language model.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create test cases for an AI agent
• Check response quality and tool use
• Investigate failed evaluation cases

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models using the same benchmark tests
• Run large evaluations on a Slurm cluster
• Evaluate safety and image understanding

Evaluate language models on knowledge, math, coding, safety, and image understanding benchmarks. The skill guides reproducible runs in containers using local Docker, Slurm clusters, or cloud execution, with shared configurations for model comparison and result export.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing models on the same benchmark tasks
• Running a small evaluation sample before a full benchmark
• Running large evaluations on a Slurm cluster

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Audit a completed AI development phase to check whether its planned evaluations were implemented. Produce EVAL-REVIEW.md with a score, verdict, identified gaps, and a plan to address missing coverage.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Review evaluation coverage after an AI phase is complete
• Check implemented evaluations against AI-SPEC.md
• Document evaluation gaps and remediation steps

Create Vally evaluation specifications for skills, custom agents, and workflow packages. Design distinct test requests, prepare fixture files, define automated checks and outcome rubrics, and size comparisons to provide meaningful evidence without favoring the skill’s wording.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create an eval.yaml for a skill or custom agent
• Define outcome-based checks and judging rubrics
• Prepare fixture files that reproduce specific failures

Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Find JavaScript that executes strings as code, including eval(), new Function(), and string-based timers. Review how it runs in the browser and suggest alternatives for parsing data, choosing operations, and scheduling functions to reduce code injection risks.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Finding eval() and other string execution
• Replacing eval() used to parse JSON
• Changing string-based timers to function references

Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Check answers for unsupported claims
• Score experiment results for correctness
• Continuously evaluate new production records

Configure OpenTelemetry to send AI application traces, records of execution steps, to Confident AI. Reuse existing exporters, choose the regional endpoint, and add AI-specific attributes without requiring the deepeval Python package.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Send existing AI application traces to Confident AI
• Configure tracing for a non-Python AI application
• Choose the US or EU endpoint for an API key

Improve an evaluated Codex skill by turning plugin-eval findings into prioritized changes. Refine activation descriptions, shorten SKILL.md, move lengthy details into supporting files, fix structural issues, and compare evaluation results before and after the rewrite.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Prioritizing fixes after a skill evaluation
• Shortening SKILL.md and reducing token use
• Clarifying when a skill should activate

공통 과제로 대규모 언어 모델의 지식·추론·코드 생성을 평가합니다. 모델 비교표와 학습 단계별 성적 추이를 만들고 평가 결과와 개별 답변을 저장합니다.

• MMLU와 GSM8K로 모델 평가
• 여러 모델의 성적 비교표 작성
• 학습 중 모델의 성적 추적

목적에 맞게 Paperclip의 Runner 평가와 제품 전체 E2E 평가를 선택하고 증거, 실행 이력, 비용, 실패 분류를 바탕으로 검증과 결과 해석, 보고를 진행합니다.

• Runner 평가와 제품 E2E 평가를 선택합니다
• 인증 정보 없이 평가 목록을 확인합니다
• 증거와 실행 기록으로 실패를 해석합니다

INVOKE THIS SKILL for LLM-as-judge evaluation workflows on Arize: creating/updating evaluators, running evaluations on spans or experiments, tasks, trigger-run, column mapping, and continuous monitoring. Use when the user says: create an evaluator, LLM judge, hallucination/faithfulness/correctness/relevance, run eval, score my spans or experiment, ax tasks, trigger-run, trigger eval, column mapping, continuous monitoring, query filter for evals, evaluator version, or improve an evaluator prompt.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Plan AI application work across six areas: prompt evaluation, context allocation, retrieval-based answer generation, agent security, evaluation pipelines, and product strategy. This overview explains the approaches; the individual workflow packages, test cases, and CI templates are not included.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Planning prompt evaluation before deployment
• Allocating context capacity and designing retrieval-based answers
• Planning an authorized AI agent security assessment