4.5K skills · sorted by relevance
Turns supplied AI agent evaluation results into reports that distinguish autonomous success, human-assisted success, failure, timeout, and invalid runs. Makes metric denominators, retries, timing, costs, and comparison conditions explicit to support evidence-based decisions.
Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.
Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.
Run repeated change-and-evaluation cycles for one predefined improvement mission within an explicit runtime limit. Save each evaluation as JSON and each decision in readable Markdown logs, with support for periodic reruns through Claude Code native cron.
Use LangSmith to record LLM application inputs, outputs, and timing, investigate failures, evaluate answers against test datasets, and monitor production usage. It also guides automated quality checks and analysis of token usage and costs.
Evaluate LLM application quality using automated scores, human reviews, and another language model as a judge. Compare models or prompts, establish baseline results, and detect quality drops through repeatable tests and statistical comparisons.
Use Phoenix to record execution traces, detailed records of an AI application's operations, and investigate problems. Evaluate answers on datasets, compare prompts and models, and set up monitoring through a self-hosted server.
Turn a feature into an agent skill or improve an existing one. Define purpose-specific quality criteria, review outputs with models from three providers, then add tests and check that user requests reach the intended skill.
Improve prompts through test cases and baseline comparisons, measure how well AI answers use retrieved sources, and check agent configurations. Includes Python tools for prompt analysis, retrieval quality reports, and token estimates, with costs calculated from supplied rates.
Evaluate a Claude Managed Agent against a written rubric, a set of checkable criteria, and guide improvements within a fixed attempt limit. After a version passes, prepare separate evaluation cases to check performance beyond the examples used for revision.
Evaluate Google Cloud workloads against built-in best practices or custom organizational rules using Workload Manager. Use public client libraries or the REST API to run checks, review violations and scanned resources, and export results to BigQuery.
Improve code, prompts, skills, or agent definitions through repeated changes and evaluation. Measure numerical targets such as latency or size, or use an LLM to judge pass-or-fail criteria, keeping changes that improve the result.
Train sentence-transformers models for search, similarity, and reranking, which reorders search results by relevance. Choose a model type, prepare data, configure training, compare results against a baseline, and publish the trained model to Hugging Face Hub.
Use Phoenix to record execution traces, detailed records of an AI application's operations, and investigate problems. Evaluate answers on datasets, compare prompts and models, and set up monitoring through a self-hosted server.
Use LangSmith to record LLM application inputs, outputs, and timing, investigate failures, evaluate answers against test datasets, and monitor production usage. It also guides automated quality checks and analysis of token usage and costs.
Train sentence-transformers models for search, similarity, and reranking, which reorders search results by relevance. Choose a model type, prepare data, configure training, compare results against a baseline, and publish the trained model to Hugging Face Hub.
Audit AGENTS.md, rules, skills, and their references for broken paths or commands. Optional independent agent reviews assess redundant instructions and practical usefulness, producing reports that identify what to retain, review, or consider trimming.
Evaluate code against a product requirements document, scoring implementation and tests separately for each acceptance criterion. Produce an evidence-backed final grade, identify gaps, and compare implementations using a fixed checklist and scoring method.
Evaluate staged Git changes before committing using an LLM, a language model acting as a reviewer. Receive correctness, completeness, and safety scores, plus issues by severity, to decide whether to fix, commit, or stop.
Guide development of a LangChain or LangGraph project created with agents-cli. It explains tool additions, local runs, evaluation, and serving while identifying the required graph structure and commands that differ from ADK projects.
Manage crypto asset cost basis in TRES Finance, from choosing how acquisition costs are assigned to reviewing realized gains and losses. Recalculate results, investigate missing prices, manage revaluations, and export cost basis reports.
Develop a broad research interest into evaluated project options. Specialized agents generate ideas, assess feasibility and competition, and draft abstracts and conclusions, helping researchers decide what to pursue, postpone, or drop and which assumption to test first.
Evaluate papers, research proposals, and literature reviews using structured criteria. Produces evidence-based feedback on research questions, methods, data, analysis, and writing, with optional scores and prioritized recommendations tailored to the work's stage and purpose.
Compare text and metadata extracted by two Tika builds from the same file collection. Generate reports to investigate content differences, encoding changes, new exceptions, and missing embedded documents, then repeat the evaluation after fixes.