5 skills · sorted by relevance
Plan a paid evaluation with predefined success criteria, comparison groups, decoy items, and stopping rules. Rehearse without spending, then run a three-item paid pilot and use its spending table to inform approval.
Scope a Claude Managed Agent through six questions about its job, trigger, inputs, actions, success criteria, and recurrence. Produce a validated build sheet with configuration components, deferred features, and an evaluation plan, without requiring an API key.
Prepare an AI-SPEC.md design agreement for an AI development phase. The skill coordinates framework selection, documentation and domain research, and evaluation planning so the phase has a defined technical direction and assessment strategy.
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
The English summary is being prepared. Showing the original description.
Builds an eval plan for an AI feature - error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a launch bar. Use when you ask how do we eval this, how do I test an AI feature, our LLM outputs are inconsistent, are the evals good enough, LLM-as-judge, hallucinations, or what accuracy is good enough to launch. Draws on 41 Lenny's Podcast guests incl. Hamel Husain, Shreya Shankar and Sander Schulhoff.
The English summary is being prepared. Showing the original description.