Skill evaluator

Discover every skill in scope, classify its distribution profile, score it on eight criteria, and keep structural validity separate from routing evidence. Every score states the host and profile it applies to.

When to use

• Before committing or publishing a skill
• After bulk-importing skills from another project or library
• When a skill triggers too often, too rarely, or only on one host
• When deciding whether a skill can ship to Claude Code, Codex, or an Agent Skills upload

Read every skill in the selected scope. Do not infer collection-wide quality from a sample. When an argument names a path, audit only that path.

---

Discovery

Claude Code locations

Location	Path	Audit note
Personal	~/.claude/skills/<name>/SKILL.md	Skip .trash/. synced/ holds skills downloaded from claude.ai: report them, never edit them
Project	.claude/skills/ in the start directory and every parent up to the repository root	Commit-shared
Nested	<subdir>/.claude/skills/<name>/SKILL.md	Loads once Claude works on files in <subdir>. On a name clash both load and the nested one is /<subdir>:<name>
Additional directory	.claude/skills/ in a directory passed with --add-dir	Session-scoped
Command file	.claude/commands/**/*.md	Legacy format. Same frontmatter except name and paths
Plugin	<plugin>/skills/<name>/SKILL.md or a plugin-root SKILL.md	Namespaced as /<plugin>:<name>
Enterprise	.claude/skills/ in the managed settings directory	Report only

Reserved folder names: a folder named synced (any case) is skipped in the enterprise, personal, and project locations. Outside a plugin, a folder or command file named anthropic-skills or starting with anthropic-skills: does not load.

Codex locations

Scope	Path
Repository	.agents/skills in every directory from the working directory up to the repository root
User	~/.agents/skills
Admin	/etc/codex/skills
System	Bundled with Codex

Codex follows symlinked skill folders. A skill disabled with a [[skills.config]] entry (enabled = false) in ~/.codex/config.toml is reported as disabled, not scored as broken.

find .claude/skills .agents/skills -name SKILL.md -not -path '*/.trash/*' 2>/dev/null
find .claude/commands -name '*.md' ! -name 'README*' 2>/dev/null

Done when: every file in scope is listed with its host, location, and the discovery rule that loads it.

---

Distribution profiles

Classify each skill before scoring it. The profile decides which fields are valid.

Profile	Where it runs	Frontmatter it may use
Claude Code native	Claude Code at any local level, including plugin skills	Every field in the Claude Code table below
Agent Skills spec	claude.ai uploads, the Skills API, packaging with package_skill.py	name, description, license, compatibility, metadata, allowed-tools. Any other key fails the upload with a hard error
Codex	Codex CLI, IDE extension, ChatGPT desktop app	name and description are required. Display metadata, invocation policy, and tool dependencies go in agents/openai.yaml

A skill published to several profiles follows their intersection. Keeping host-specific fields inside metadata to record provenance is valid for a portable skill and is not a defect.

Done when: each skill has one declared or inferred profile, with the evidence for the inference.

---

Claude Code frontmatter reference

Claude Code reads frontmatter only when the opening --- is the file's first line. It ignores an unknown field without reporting an error. When the YAML does not parse, the skill loads with no fields set: /name still works, but Claude cannot match the description. Detect parse failures with claude plugin validate <skills-dir> (Claude Code v2.1.233 or later). Boolean fields accept yes, no, on, off, 1, and 0 in any case from v2.1.218.

Field	Notes
name	In a personal or project skill directory, sets the command shown in / and typed to invoke it; the directory name also invokes it. Defaults to the directory name. In a plugin it sets the last segment after the plugin prefix. Command files take their name from the file
description	What the skill does and when to use it. If omitted, the first non-empty body line is used
when_to_use	Appended to description in the listing and counted in the same cap
argument-hint	Autocomplete hint such as [issue-number]
arguments	Named positional arguments for $name substitution. Space-separated string or YAML list
disable-model-invocation	true removes the description from context: only the user can invoke. Also blocks preloading into subagents and scheduled-task invocation
user-invocable	false hides the skill from /; Claude can still invoke it
allowed-tools	Pre-approves the listed tools for the turn that invokes the skill. It does not restrict tools: every tool stays callable. Space- or comma-separated string, or YAML list. Workspace trust does not gate it
disallowed-tools	Removes tools from the pool while the skill is active. Cleared on the next message
model	Model for the rest of the current turn, or inherit. A value excluded by availableModels is not used
effort	low, medium, high, xhigh, or max. Available levels depend on the model
context	fork runs the body as the prompt of a new subagent that does not see the conversation
agent	Subagent type used with context: fork
background	With context: fork, false waits for the result in the invoking turn. Default true (v2.1.218 or later)
hooks	Registered when the skill is invoked and kept for the rest of the session. once: true removes a hook after its first successful run
paths	Glob patterns that limit automatic loading. YAML list or comma-separated string
shell	bash (default) or powershell for inline command injection
metadata	Free-form map read by your own tooling. A non-map value is dropped. Do not reuse frontmatter field names as keys
license	Accepted, not acted on
compatibility	String up to 500 characters. Accepted, not acted on

Fields outside this table, such as tags, category, keywords, usage, or args, have no effect in Claude Code and break an Agent Skills upload. Flag them.

String substitutions

Placeholder	Meaning
$ARGUMENTS	Full argument string. When no placeholder receives the arguments, Claude Code appends ARGUMENTS: <value>
$ARGUMENTS[N]	Argument at 0-based index N
$N	Shorthand for $ARGUMENTS[N]. Valid without an arguments: field
$name	Named argument declared in arguments:. Expands to an empty string when the argument is missing
${CLAUDE_SESSION_ID}	Current session ID
${CLAUDE_EFFORT}	Active effort level
${CLAUDE_SKILL_DIR}	Directory containing this SKILL.md. Also substituted in allowed-tools Bash rules
${CLAUDE_PROJECT_DIR}	Project root (v2.1.196 or later). Also substituted in allowed-tools Bash rules
${CLAUDE_PLUGIN_ROOT}, ${CLAUDE_PLUGIN_DATA}	Plugin installation and persistent data directories. Plugin skills only

An indexed placeholder with no matching argument stays in the text unchanged. Escape a literal $ before a digit, ARGUMENTS, or a declared name with a single backslash, as in \$1.00. Placeholders such as ${ARGS} or %ARGUMENTS% are passed as literal text. The Codex skills documentation does not describe argument substitution: record Codex behavior for these placeholders as UNKNOWN rather than assuming it.

Listing budget

• Claude Code: the skill listing gets 1% of the model's context window. Each entry's combined description and when_to_use is capped at 1,536 characters, configurable with skillListingMaxDescChars. skillListingBudgetFraction or SLASH_COMMAND_TOOL_CHAR_BUDGET raise the budget. On overflow, descriptions of the least-invoked skills are dropped first. skillOverrides set to "name-only" frees budget. /skill-doctor (v2.1.252 or later) reports cost and usage per skill.
• Codex: the initial list uses at most 2% of the context window, or 8,000 characters when the window is unknown. Codex shortens descriptions first and may omit skills with a warning.

On both hosts, front-load the key use case so a shortened description still matches.

---

Scoring Criteria (25 points per skill)

#	Criterion	Max	What is checked
1	identity	2	name present with lowercase letters, digits, and hyphens (1), required by Codex; no unresolved name collision on the same host (1)
2	context pointer	4	Description identifies the capability (1), names distinct trigger branches (1), leads with discriminating terms (1), and fits the listing cap without synonym stuffing (1)
3	profile fit	3	Profile declared or inferable (1); every field valid for that profile (1); tool grants proportionate to the workflow (1)
4	invocation and effort	2	Invocation control matches side effects (1); effort, when declared, matches the inferred level and the target model supports it (1)
5	workflow contract	4	Purpose or boundary is explicit (1), ordered work is actionable (1), each step has an observable completion criterion (1), and no placeholder text remains (1)
6	resource closure	3	Required local links resolve (1), required scripts and templates exist (1), and no resource escapes the skill root without an explicit portable contract (1)
7	instruction economy	3	One source of truth per rule (1), no instruction merely restates model defaults (1), and branch-only material is disclosed instead of always loaded (1)
8	routing evidence	4	Corpus has at least 8 positives and 2 close negatives (1), uses realistic language and boundary cases (1), Claude Code evidence recorded (1), Codex evidence recorded (1)

Thresholds, as a share of applicable points:

• ✅ Good: 80% or more (20 to 25 of 25)
• ⚠️ Needs work: 60% to 79% (15 to 19 of 25)
• ❌ Fix: under 60% (14 or fewer of 25)

An explicitly invoked command wrapper may omit automatic routing evidence. Record it as explicit-only, verify that each requested host can discover it, and score it out of 21: Good 17 or more, Needs work 13 to 16, Fix 12 or fewer.

Identity review

Resolve collisions with each host's rule. Claude Code: enterprise over personal over project; a skill over a command file; a local skill replaces a bundled command but not its aliases; a nested skill and a root skill both load; plugin skills are namespaced. Codex does not merge skills that share a name: both can appear in selectors, which is a collision to report.

Context pointer review

Treat the description and when_to_use text as the pointer that decides whether the skill loads. List branches, not synonym piles. A branch is a different kind of request handled by different instructions. If two descriptions can both claim the same prompt, record the collision as a correctness defect.

Profile fit review

• Claude Code native: allowed-tools is a pre-approval, so an unscoped Bash entry lets every shell command run without a prompt during that turn, even in an untrusted repository. For a read-only workflow, pre-approve only the commands it runs, such as Bash(git status *) or Bash(${CLAUDE_SKILL_DIR}/scripts/check.sh *). Use disallowed-tools when an autonomous skill must never call a tool, such as AskUserQuestion in a background loop. A native skill with no allowed-tools is not penalized: the user's permission settings apply.
• Agent Skills spec: any field outside the six allowed keys fails the criterion.
• Codex: name and description present; agents/openai.yaml, when present, parses and its dependencies.tools entries name available tools.
• Portable across hosts: host-enforcement fields are absent or kept inside metadata. Do not recommend adding allowed-tools or effort to the frontmatter of a portable skill.

Invocation and effort review

A workflow with side effects, such as deploy, publish, send, or delete, sets disable-model-invocation: true for Claude Code and policy.allow_implicit_invocation: false in agents/openai.yaml for Codex, or states why automatic invocation is safe. Missing effort is not a defect. A declared effort must match the inference below and be supported by the target model: for example, the guide lists no effort parameter for Haiku 4.5.

Workflow completion review

For each ordered step, identify the condition that permits the next step to begin. Accept commands run, files produced, checks passed, decisions recorded, or explicit reports of blocked evidence. Reject criteria such as "analysis complete" or "understanding reached" when the agent cannot test them.

Resource closure review

Resolve local Markdown links relative to the file containing them. Also verify scripts, templates, fixtures, and references declared in prose or code examples when execution depends on them. Bundled scripts should be referenced through ${CLAUDE_SKILL_DIR} in Claude Code rather than a path that depends on the working directory. A missing required resource is a blocker even when the frontmatter and prose score well.

Instruction economy review

Apply three checks sentence by sentence:

1. Would removing this instruction change likely behavior? If not, flag a no-op.
2. Is the same rule authoritative elsewhere? If yes, keep one source and point to it.
3. Does every invocation need this material? If not, move it behind a branch-specific pointer.

---

Producing routing evidence

Never award a routing point without a recorded run. Record UNKNOWN for a host that was not tested.

• Baseline comparison (both hosts): run each prompt in a fresh session with the skill available and again with it disabled (skillOverrides set to "off" in Claude Code, [[skills.config]] with enabled = false in Codex). A fresh session matters because context left from authoring the skill masks gaps.
• Claude Code plugin skills: claude plugin eval runs each prompt in an isolated session with and without the plugin; a tool_used: Skill grader measures triggering and the command exits non-zero below its threshold.
• Claude Code description tuning: the official skill-creator plugin from claude-plugins-official generates should-trigger and should-not-trigger prompts and measures the hit rate.
• Codex: the Codex skills documentation describes no dedicated evaluation command. Replay each prompt in a fresh Codex session and record whether the skill was selected.

Done when: each routed skill has a result per host, or UNKNOWN with the reason.

---

Effort level inference

Infer a level from the description and body. Report it as a recommendation for Claude Code native skills only.

• low: mechanical execution with no design decision. Template instantiation, formatting, syncing, fetching. Short sequential workflow, no subagents.
• medium: bounded analysis and categorization. Reviewing one diff, triaging issues, producing a structured table. At most one or two scoped subagents.
• high: design decisions, adversarial or cross-system reasoning. Security audits, architecture reviews, scoring against trade-offs. Several subagents or explicit uncertainty sections.
• xhigh or max: exhaustive long-running synthesis. Fan-out across many parallel agents, dynamic workflows, multi-repository scope.

Ultracode is a separate toggle in /effort since Claude Code v2.1.284 and no longer implies an effort level. Do not treat the word ultracode in a skill body as an effort signal.

If a declared effort differs from the inference, flag it:

⚠️ Effort mismatch: declared low, inferred high. Skill spawns four subagents and performs security analysis.

---

Execution instructions

Step 1: Discover

List every skill file in scope with its host and location, using the discovery tables above. Done when: the inventory is complete and each entry names its discovery rule.

Step 2: Classify and parse

For each skill:

1. Read the full file and extract the frontmatter between the first two --- lines.
2. Record a parse failure when the YAML does not parse, and confirm it with claude plugin validate <skills-dir> when Claude Code v2.1.233 or later is available.
3. Assign the distribution profile.
4. Flag fields that are unknown or invalid for that profile.
5. Check substitution placeholders against the table.
6. Resolve required local links, scripts, templates, fixtures, and references.
7. Record ordered steps and the observable completion criterion for each one.
8. Read evals/scenarios.json when present and count positives and negatives.

Done when: every skill has a parsed record, a profile, and a resource list with each item resolved or missing.

Step 3: Score and infer

Apply the eight criteria, infer effort, and compare it with any declared value. Check context: fork skills for an actionable task body and an agent field, and note that a backgrounded fork runs with the narrower background tool set. Done when: every criterion has a score and a one-line reason.

Step 4: Report

# Skills Audit: [project name or path]
Date: [today] | Scanned: N skills | Hosts: Claude Code, Codex

## Summary
| Status | Count |
|--------|-------|
| ✅ Good (>=80%) | N |
| ⚠️ Needs work (60-79%) | N |
| ❌ Fix (<60%) | N |
| explicit-only | N |

Profiles: N Claude Code native · N portable · N Codex only
Routing evidence: N both hosts · N one host · N UNKNOWN

---

## Per-Skill Results

### [skill-name] ([score]/25) [✅/⚠️/❌] [profile]

| Criterion | Score | Notes |
|-----------|-------|-------|
| identity | ✅ 2/2 | ok |
| context pointer | ⚠️ 2/4 | Overlaps a neighboring skill |
| profile fit | ⚠️ 2/3 | Unscoped Bash pre-approval for a read-only workflow |
| invocation and effort | ✅ 2/2 | No side effects, no effort declared |
| workflow contract | ⚠️ 2/4 | Two steps have no observable completion criterion |
| resource closure | ❌ 0/3 | Required template does not exist |
| instruction economy | ⚠️ 2/3 | One branch-only reference is always loaded |
| routing evidence | ❌ 0/4 | No corpus; Claude Code and Codex UNKNOWN |

**Effort inference**: `high`. Security analysis with adversarial reasoning.

**Priority fixes** (ordered by impact):
1. Restore the missing required template
2. Separate the overlapping trigger branch from the neighboring skill
3. Scope `allowed-tools` to the commands the workflow runs
4. Add a corpus and record results for both hosts

Done when: every scanned skill appears once, and the summary counts match the per-skill results.

---

Fix summary format

End with a copy-paste block of recommended effort values for Claude Code native skills only, then one count line:

## Recommended effort fields (Claude Code native skills)

skill-name-1: effort: low     # mechanical scaffold
skill-name-2: effort: high    # security analysis, spawns agents

N broken resources · N name collisions · N trigger collisions · N invalid fields for profile · N steps without completion criteria · N routing gaps · N effort mismatches