Skip to content
cccskills
Official · GoogleFreePublished on GitHubEnglish overview

agent-platform-autorater-customization

Builds evaluation metrics for an AI agent from its real interaction traces and your criteria. Review grades against human labels, refine rubrics and thresholds, and prepare approved metrics for registration in the Agent Platform evaluation service.

OriginalShow the original description

Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).

google/skills24,365 stars(GitHub 24,365 + cccskills 0)

Run score Not tested

See how to install

Good for

  • Tailoring an evaluator to an agent’s behavior
  • Turning failed interaction traces into grading rubrics
  • Checking why an existing evaluator disagrees with human ratings
  • Comparing metric grades with human labels

Overview

What it does

This skill uses real agent traces and the behaviors you care about to build evaluation criteria and metrics. It can propose checks for answer correctness, tool choice, instruction following, and responses to errors or requests for clarification, using evidence in the traces. It then shapes each criterion into a suitable form, such as a code check or an LLM checklist. Results from the evaluation service are collected on a review page, where you can compare them with human labels and adjust rubrics, weights, and pass thresholds. After approval, the metrics can be registered, and the handoff includes the criteria, definitions, and results.

When it helps

Use it when an existing grader does not score your agent as expected, or when you want reusable criteria based on failed traces. It also supports grading multi-turn conversations and traces with tool calls. The work focuses on bringing automated judgments closer to human decisions.

Example requests

  • “Create criteria for evaluating answer accuracy from these traces.”
  • “Find cases where the existing grader disagrees with people and adjust the criteria.”

Notes

This skill uses the Google Cloud Agent Platform evaluation service. It requires the Python SDK, gcloud CLI, a project with billing enabled, API activation, authentication, and the necessary permissions. Evaluation calls are billed. Confirm that traces may be sent to the service and check costs before running an evaluation. Metric registration also requires approval.

This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.

Document check

AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).

SafetyAre sending data, deleting files, and running commands disclosed?
AGoodTrace submission, billing, and approval before metric registration are explicit.
Easy setupDoes it need few tools or API keys, and are they documented?
BPartly lackingIt lists the SDK, gcloud, billing, API, authentication, and IAM needs.

Not counted in the score. Rated by AI (OpenAI Codex) as of Nov 14, 2026. Treat it as a guide and read the original before installing.

Disagree? Request a re-evaluation

Tell us which point is wrong and why. No login needed. We will check it against the SKILL.md, redo the evaluation, and show the result here.

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

Files included (12)

  • SKILL.md14.1 KB
  • references/attribution.md1.9 KB
  • references/converting_trajectories.md1.9 KB
  • references/metric_templates.md24.0 KB
  • references/multi_turn.md2.0 KB
  • references/noise_and_agreement.md1.2 KB
  • references/sdk_gotchas.md2.7 KB
  • references/writing_checks.md1.4 KB
  • scripts/agreement.py6.9 KB
  • scripts/build_review_page.py10.2 KB
  • scripts/render_agent_data.py3.3 KB
  • scripts/run_metrics.py6.2 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

agent-platform-access-audit

Official · GoogleFreeSummary

Use this skill to inspect an AI agent’s Google Cloud access through audit logs queried with BigQuery. It summarizes accessed resources and services, identifies originating principals, and helps distinguish VPC Service Controls blocks from IAM permission denials.

  • Reviewing which resources and services an agent accessed
  • Finding the principals that invoked or delegated to an agent
  • Investigating PERMISSION_DENIED errors in audit logs
google/skills24.2KUpdated Nov 4, 2026

agent-platform-alert-configuration

Official · GoogleFreeSummary

Create Terraform files that define alerts for AI agent latency, errors, token use, safety, security, and response quality. Check telemetry and existing policies first, then validate the generated configuration. Quality alerts require Vertex AI.

  • Configure alerts for agent latency and error rates
  • Monitor rapid increases in token usage
  • Monitor safety triggers and permission denials
google/skills24.3KUpdated Oct 31, 2026

agent-platform-architecture

Official · GoogleFreeSummary

Design and validate enterprise Gemini platform architectures across projects, networks, identity, and governance. Plan Agent Gateway and egress patterns, policy controls, agent identity, and safety guardrails, then map them to deployment configurations and post-deployment checks.

  • Planning a multi-project Gemini platform
  • Choosing agent network egress and gateway patterns
  • Designing identity and governance controls
google/skills24.4KUpdated Nov 18, 2026

agent-platform-deploy

Official · GoogleFreeSummary

Deploy Model Garden models to Agent Platform endpoints, the addresses used for model requests. Check supported hardware and estimated hourly costs, inspect deployment progress, test predictions, and remove deployed resources after use.

  • Choose a deployable model and supported hardware
  • Estimate hourly deployment costs before approval
  • Check deployment progress and test predictions
google/skills24.4KUpdated Nov 18, 2026

agent-platform-endpoint-management

Official · GoogleFreeSummary

Manages Agent Platform endpoints, the stable URLs used to serve online model predictions. It supports listing, inspecting, creating, updating, and deleting endpoints, plus troubleshooting permission errors, quota limits, and resources that are busy.

  • Listing endpoints and checking their configuration
  • Creating an endpoint before model deployment
  • Updating endpoint display names or labels
google/skills24.4KUpdated Nov 18, 2026

agent-platform-eval-flywheel

Official · GoogleFreeSummary

Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.

  • Build evaluation datasets from conversation logs
  • Generate simulated user scenarios
  • Choose scoring criteria for responses and tool use
google/skills24.4KUpdated Nov 3, 2026

See all skills by google

Report a problem with this skill