Results for “evals”

Author and run offline evals for Agentforce agents: fixture format, scoring rubrics, hallucination checks, regression baselines, CI integration, prompt-change safety. Trigger keywords: agent eval, hallucination, doesn't hallucinate, agentforce offline evals, rubric for agent response. NOT for deciding which layers of testing the agent needs — use agentforce/agentforce-testing-strategy. NOT for Testing Center, AiEvaluationDefinition or sf agent test runs — use agentforce/agent-testing-and-evaluation.

The English summary is being prepared. Showing the original description.

Claude API (platform.claude.com) のプロンプトエンジニアリング・評価リファレンス。 prompt engineering, XML タグ, few-shot, system prompts, モデル別プロンプティング（Fable 5.1 / Fable 5 / Opus 5.5 / Opus 5 / Sonnet 5.5 / Sonnet 5 / Haiku 5.5）。 evals, grading（LLM-graded 等）, jailbreak・prompt leak 対策, hallucination 低減, classification, ticket routing, commerce agent 等ユースケース実装。

The English summary is being prepared. Showing the original description.

Design a self-improvement loop for a shipped product's AI agents — production traces feed evals, evals feed improvement proposals (prompts, playbooks, retrieval configs), and a human approval gate promotes changes with rollback. Technique-agnostic: chooses per project between ACE-style evolving playbooks, GEPA/MIPROv2 offline optimization, or simple eval-driven iteration via references/techniques.md. Load when the user asks to make my product's agents self-improving, learn from production traces, add a learning loop, evolve prompts or playbooks safely, promote agent improvements, or GEPA-style optimization. NOT harness-evolution (that improves the coding agent), NOT experimentation (product A/B tests), NOT agent-run-retro (dev-phase manual retros — this skill is the production-scale continuation).

The English summary is being prepared. Showing the original description.

Define success criteria before AI coding, then assess new capabilities and check that existing behavior still works using code, rules, models, or human review. Track repeated-attempt success metrics and reports to compare prompt changes, agent changes, or model versions.

• Defining success criteria before AI implementation
• Checking existing behavior after prompt or agent changes
• Measuring reliability across repeated attempts

Create reusable agent skills from recurring tasks, or improve existing ones through test prompts and user feedback. Compare outputs with and without a skill, measure performance, and refine descriptions so skills are invoked for the right requests.

• Turning a recurring workflow into a reusable skill
• Improving an existing skill using feedback
• Comparing results with and without a skill

Find guidance on Codex settings, pricing, and OpenAI products and APIs using current official documentation. Get cited answers for setup, troubleshooting, model comparisons, migration planning, and prompt changes, with uncertainty stated when sources are inconclusive.

• Check Codex settings and troubleshoot problems
• Compare models for cost, speed, and quality
• Plan a model migration and adjust prompts

Add or extend Paperclip Runner evaluations that test real sessions against a mock control service. Define observable pass conditions, validate cases and run selections, and retain source revisions, evidence, and costs for authorized live runs.

• Adding a focused Runner protocol evaluation
• Testing grading rules with valid and invalid evidence
• Updating evaluation selections and campaign membership

Turn a repeatable workflow into a skill, or improve an existing one through test prompts and feedback. Compare results with and without the skill, review performance measurements, and refine its description so it triggers appropriately.

• Turn a recurring workflow into a reusable skill
• Improve an existing skill using feedback
• Compare outputs with and without a skill

Create and improve agent skills by defining tasks, writing instructions, and testing realistic requests. Compare outputs, success rates, time, and token use, then refine the skill and its description so it triggers appropriately.

• Turn a recurring workflow into a reusable skill
• Improve an existing SKILL.md using test feedback
• Compare results with and without a skill

Get guidance on Codex, OpenAI APIs, and ChatGPT Work using official documentation with citations. Use it to investigate settings and errors, compare models, plan migrations, or work on prompts and API integrations.

• Check Codex settings and troubleshoot errors
• Compare models for a particular task
• Plan a model migration and prompt changes

Build and maintain BMad skills, agents, and modules through a guided conversation. Turn prompts or session logs into reusable skills, review existing tools, and prepare modules for distribution, with approval before files are written.

• Create a skill from a prompt or session logs
• Build an agent with a defined persona
• Fix or review an existing skill

Evaluate BMad skills using your agent's configured runner. Compare results with the model alone or another version, grade outputs against criteria, and test whether descriptions trigger correctly, preserving transcripts and evidence for each run.

• Compare a skill with the model alone
• Compare an edited skill with a previous version
• Grade outputs against defined criteria

Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.

• Build an automated evaluation pipeline for AI responses
• Compare responses after changing a prompt or model
• Define shared scoring criteria for reviewers

Build repeatable evaluations for AI agent responses, tool choices, and multi-step tasks. Use reference datasets and scoring criteria to compare changes, detect quality regressions, and set automated checks that must pass before deployment.

• Compare agent quality before and after prompt changes
• Test tool selection and multi-step task execution
• Evaluate resistance to prompt injection

Turn recurring tasks into reusable agent skills, or improve skills you already have. Draft instructions, compare test outputs with and without a skill, review quality and resource use, and refine descriptions so skills trigger for appropriate requests.

• Turn a recurring workflow into a reusable skill
• Improve an existing skill using test results
• Compare output quality, runtime, and token use

Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.

• Turning observed failures into evaluation criteria
• Building code-based or language-model evaluators
• Evaluating retrieval and answer faithfulness in RAG

Investigate LLM application failures with Phoenix CLI by reviewing execution records, recording observations, and grouping mistakes into counted categories. Inspect datasets and experiments to identify which failure types deserve evaluation tests first.

• Investigating failures in LLM applications
• Grouping observed mistakes into failure categories
• Choosing which evaluation tests to build first

Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.

• Setting up quality checks for Python AI applications
• Creating realistic scenarios for known failure modes
• Scoring full application runs with real language-model calls

Build and operate LLM applications with guidance on RAG, which uses retrieved documents to inform answers. Covers document indexing, vector database selection, prompt improvement, response evaluation, caching, and estimates of model usage costs.

• Building an assistant that answers from documents
• Choosing a vector database for document search
• Comparing answer quality after prompt changes

Validates skill files and registration consistency, then checks project tests and system health. Compares failures with recent changes to distinguish regressions, outdated tests, intermittent failures, and infrastructure issues, producing a report that tracks results over time.

• Validating definitions and registration for new skills
• Running daily tests and system health checks
• Classifying failures after code changes

Build, edit, or debug backend AI agents with eve, a framework that represents agents as directories of files. Work on instructions, tools, connections, schedules, and related components using documentation bundled with the installed version.

• Create a backend AI agent with eve
• Update agent instructions, tools, and connections
• Debug schedules and subagent configurations

Explore explanations for poorly understood phenomena through repeated hypothesis generation and selection. Compare causal mechanisms, test what happens when components change, and retain distinct candidates with predictions, rival comparisons, and clearly stated unresolved questions.

• Developing hypotheses for an unexplained phenomenon
• Comparing competing causal explanations
• Identifying predictions that distinguish theories

Strengthen existing tests with fast-check by checking properties across generated inputs. Find bugs that example tests miss and reduce failing inputs to small, reproducible cases. Property testing is available; the other listed workflows remain planned.

• Checking parser behavior across generated inputs
• Testing serialization and deserialization round trips
• Finding pure functions covered only by example tests

Develop varied ideas for names, writing angles, and problem solving by exploring five candidates internally and presenting the strongest. It also expands small example collections into diverse datasets for evaluation, training, or testing.

• Finding a distinctive product name
• Developing a fresh angle for a blog post
• Exploring solutions to a technical problem