'evals' 검색 결과

Author and run offline evals for Agentforce agents: fixture format, scoring rubrics, hallucination checks, regression baselines, CI integration, prompt-change safety. Trigger keywords: agent eval, hallucination, doesn't hallucinate, agentforce offline evals, rubric for agent response. NOT for deciding which layers of testing the agent needs — use agentforce/agentforce-testing-strategy. NOT for Testing Center, AiEvaluationDefinition or sf agent test runs — use agentforce/agent-testing-and-evaluation.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Claude API (platform.claude.com) のプロンプトエンジニアリング・評価リファレンス。 prompt engineering, XML タグ, few-shot, system prompts, モデル別プロンプティング（Fable 5.1 / Fable 5 / Opus 5.5 / Opus 5 / Sonnet 5.5 / Sonnet 5 / Haiku 5.5）。 evals, grading（LLM-graded 等）, jailbreak・prompt leak 対策, hallucination 低減, classification, ticket routing, commerce agent 等ユースケース実装。

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design a self-improvement loop for a shipped product's AI agents — production traces feed evals, evals feed improvement proposals (prompts, playbooks, retrieval configs), and a human approval gate promotes changes with rollback. Technique-agnostic: chooses per project between ACE-style evolving playbooks, GEPA/MIPROv2 offline optimization, or simple eval-driven iteration via references/techniques.md. Load when the user asks to make my product's agents self-improving, learn from production traces, add a learning loop, evolve prompts or playbooks safely, promote agent improvements, or GEPA-style optimization. NOT harness-evolution (that improves the coding agent), NOT experimentation (product A/B tests), NOT agent-run-retro (dev-phase manual retros — this skill is the production-scale continuation).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

AI 개발 전에 성공 조건을 정하고 코드·규칙·모델·사람의 평가로 새 기능과 기존 기능을 확인합니다. 여러 차례 시도한 결과에서 신뢰성을 기록합니다.

• AI 구현 전에 성공 조건 정의
• 프롬프트나 에이전트 변경 후 회귀 확인
• 여러 차례 시도로 신뢰성 측정

반복해서 요청하는 작업을 재사용할 스킬로 만들고 기존 스킬을 개선·평가합니다. 출력 비교와 사용자 의견을 바탕으로 지침과 호출 조건을 조정합니다.

• 반복 작업을 재사용할 스킬로 정리
• 사용자 의견으로 기존 스킬 개선
• 스킬 사용 여부에 따른 결과 비교

공식 자료를 조사해 Codex 설정과 요금, OpenAI API·제품 사용법을 출처와 함께 안내합니다. 모델 선택과 이전, 프롬프트 조정, 문제 원인 파악에도 활용합니다.

• Codex 설정 확인 및 문제 원인 파악
• 모델 비용·속도·품질 비교
• 모델 이전과 프롬프트 조정 검토

Paperclip Runner의 동작 평가에 판정 조건, 실행 대상, 보고서 검증 데이터를 추가하고 증거와 실행 이력을 남겨 검증합니다.

• Runner 동작 평가를 추가합니다
• 정상 및 잘못된 증거로 판정을 검증합니다
• 실행 대상 목록과 지속 평가를 갱신합니다

반복하는 작업 절차를 스킬로 만들고 테스트와 피드백으로 개선합니다. 기존 스킬 수정, 적용 전후 결과 비교, 적절한 요청에 호출되도록 설명 조정도 지원합니다.

• 반복 작업을 재사용할 스킬로 만듭니다
• 피드백을 바탕으로 기존 스킬을 개선합니다
• 스킬 적용 전후의 출력을 비교합니다

Create and improve agent skills by defining tasks, writing instructions, and testing realistic requests. Compare outputs, success rates, time, and token use, then refine the skill and its description so it triggers appropriately.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turn a recurring workflow into a reusable skill
• Improve an existing SKILL.md using test feedback
• Compare results with and without a skill

Get guidance on Codex, OpenAI APIs, and ChatGPT Work using official documentation with citations. Use it to investigate settings and errors, compare models, plan migrations, or work on prompts and API integrations.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Check Codex settings and troubleshoot errors
• Compare models for a particular task
• Plan a model migration and prompt changes

Evaluate BMad skills using your agent's configured runner. Compare results with the model alone or another version, grade outputs against criteria, and test whether descriptions trigger correctly, preserving transcripts and evidence for each run.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare a skill with the model alone
• Compare an edited skill with a previous version
• Grade outputs against defined criteria

Build and maintain BMad skills, agents, and modules through a guided conversation. Turn prompts or session logs into reusable skills, review existing tools, and prepare modules for distribution, with approval before files are written.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create a skill from a prompt or session logs
• Build an agent with a defined persona
• Fix or review an existing skill

Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Build an automated evaluation pipeline for AI responses
• Compare responses after changing a prompt or model
• Define shared scoring criteria for reviewers

Build repeatable evaluations for AI agent responses, tool choices, and multi-step tasks. Use reference datasets and scoring criteria to compare changes, detect quality regressions, and set automated checks that must pass before deployment.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare agent quality before and after prompt changes
• Test tool selection and multi-step task execution
• Evaluate resistance to prompt injection

Turn recurring tasks into reusable agent skills, or improve skills you already have. Draft instructions, compare test outputs with and without a skill, review quality and resource use, and refine descriptions so skills trigger for appropriate requests.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turn a recurring workflow into a reusable skill
• Improve an existing skill using test results
• Compare output quality, runtime, and token use

Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turning observed failures into evaluation criteria
• Building code-based or language-model evaluators
• Evaluating retrieval and answer faithfulness in RAG

Investigate LLM application failures with Phoenix CLI by reviewing execution records, recording observations, and grouping mistakes into counted categories. Inspect datasets and experiments to identify which failure types deserve evaluation tests first.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Investigating failures in LLM applications
• Grouping observed mistakes into failure categories
• Choosing which evaluation tests to build first

Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Setting up quality checks for Python AI applications
• Creating realistic scenarios for known failure modes
• Scoring full application runs with real language-model calls

Build and operate LLM applications with guidance on RAG, which uses retrieved documents to inform answers. Covers document indexing, vector database selection, prompt improvement, response evaluation, caching, and estimates of model usage costs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Building an assistant that answers from documents
• Choosing a vector database for document search
• Comparing answer quality after prompt changes

Validates skill files and registration consistency, then checks project tests and system health. Compares failures with recent changes to distinguish regressions, outdated tests, intermittent failures, and infrastructure issues, producing a report that tracks results over time.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Validating definitions and registration for new skills
• Running daily tests and system health checks
• Classifying failures after code changes

Build, edit, or debug backend AI agents with eve, a framework that represents agents as directories of files. Work on instructions, tools, connections, schedules, and related components using documentation bundled with the installed version.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create a backend AI agent with eve
• Update agent instructions, tools, and connections
• Debug schedules and subagent configurations

Explore explanations for poorly understood phenomena through repeated hypothesis generation and selection. Compare causal mechanisms, test what happens when components change, and retain distinct candidates with predictions, rival comparisons, and clearly stated unresolved questions.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Developing hypotheses for an unexplained phenomenon
• Comparing competing causal explanations
• Identifying predictions that distinguish theories

Develop varied ideas for names, writing angles, and problem solving by exploring five candidates internally and presenting the strongest. It also expands small example collections into diverse datasets for evaluation, training, or testing.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Finding a distinctive product name
• Developing a fresh angle for a blog post
• Exploring solutions to a technical problem

Strengthen existing tests with fast-check by checking properties across generated inputs. Find bugs that example tests miss and reduce failing inputs to small, reproducible cases. Property testing is available; the other listed workflows remain planned.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Checking parser behavior across generated inputs
• Testing serialization and deserialization round trips
• Finding pure functions covered only by example tests