Skip to content
cccskills
FreePublished on GitHub

skill-counterfactual-audit

Audit a skill by running a paired probe — the same task once with the skill loaded and once without — segment both traces into goal-directed phases, align phases, and emit a SIP report (surface anchoring, template copy, excess planning, task recovery, off-task artifact). Use whenever a skill is created, modified, or proposed for retirement. Pass-rate is BLIND to most skill effects: CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. Triggers: "audit this skill", "is this skill helping", "retire skill", "before shipping skill", "behavioural impact of skill X", or any skill review event.

The English summary is being prepared. Showing the original description.

See how to install

Files included (1)

  • SKILL.md4.2 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Skill Counterfactual Audit

Why

When you ship a skill, pass-rate is the easy metric — and it's blind to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and compare the behaviours directly rather than the headline metric.

How

For each skill under audit:

  1. Curate a probe-task bank (3-5 small tasks per skill domain). Each task should have:
    • A clear goal (a single instruction)
    • An expected category of behaviour
    • A "phase decomposition" (rough segmentation rules)
  2. Run paired — same prompt, same task, run A with skill loaded, run B without.
  3. Segment each trace — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
  4. Align phases — match B phases to A phases; mark new/dropped/displaced phases.
  5. Classify SIP (Skill Influence Pattern) — for each aligned phase, label the effect:
    • surface-anchoring — A copies vocabulary/structure from the skill itself
    • template-copy — A reproduces a skill template literally without adaptation
    • excess-planning — A adds extra planning steps absent in B
    • task-recovery — A recovers from drift that B falls into
    • off-task-artifact — A produces artifacts unrelated to the goal
  6. Emit report to .planning/patterns/skill-audits/<skill>-<timestamp>.md with phase comparison table + SIP distribution + retire/refine/keep recommendation.

Recommendation rule of thumb

Dominant SIPVerdict
task-recovery + correct artifactskeep (genuine value)
surface-anchoring with no task-recoveryrefine (skill is leaking style, not capability)
template-copy on most tasksrefine (lower template specificity)
off-task-artifact on >25% of probesretire (skill is causing distraction)
excess-planning with no completion deltarefine (skill adds ceremony)

When to skip

  • The skill is a workflow guide that doesn't directly shape output (e.g., loop, schedule) — audit dominant SIPs would always be "task-recovery" trivially.
  • No probe-task bank exists for the domain (build one first; defer audit).
  • The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

Integration

  • done-retirement — required gate before retiring a skill.
  • gsd-cleanup — sweeps audits on a milestone-close cadence.
  • Companion to simplify, code-review — those audit code; this audits skill behaviour.

Cross-references

  • Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
  • College: agent-systems / integration-evaluation / agent-paired-trace-audit
  • Related skills: skill-integration (manages the skill lifecycle that this audits)

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

Provides web accessibility best practices for semantic HTML, ARIA, keyboard navigation, color contrast, and screen reader patterns. Use when building UI components, reviewing accessibility, or when user mentions 'a11y', 'accessibility', 'ARIA', 'screen reader', 'keyboard navigation', 'WCAG'.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

Active listening techniques for effective communication. Covers attending behaviors, paraphrasing, reflective listening, clarifying questions, empathic response, barriers to listening, listening in conflict, and cross-cultural listening. Use when building listening skills, improving understanding in conversation, mediating disputes, or analyzing communication breakdowns.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

Adversarial spec-compliance PR review — cross-references diffs against approved specs, verifies runtime claims against source, detects competing PRs, audits scope/convention compliance. Use before merging.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

Provides best practices for AI agent orchestration including MCP servers, A2A protocol, multi-agent coordination, and swarm architectures. Use when designing agent systems, configuring MCP servers, setting up agent teams, or when user mentions 'MCP', 'A2A', 'agent orchestration', 'multi-agent', 'swarm', 'agent team', 'LangGraph', 'CrewAI', 'AutoGen'.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

Symbolic manipulation, equation solving, and algebraic structures for mathematical reasoning. Covers distributive law, factoring, completing the square, linear through polynomial equation solving, systems of equations (substitution, elimination, Gaussian elimination, matrix methods), algebraic structures (groups, rings, fields), modular arithmetic, polynomial theory, and inequalities. Use when solving equations, simplifying expressions, working with algebraic structures, or performing symbolic manipulation.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

Understanding how algorithmic systems shape what users see, know, and do -- from recommendation feeds to search ranking to credit scoring to hiring software. Covers the mechanics of recommendation systems, algorithmic bias and its sources, personalization's effects on information diets, opacity and accountability, AI limitations (hallucination, confident wrongness), and the human-in-the-loop question. Use when a learner needs to think critically about why particular content reached them.

The English summary is being prepared. Showing the original description.

Tibsfox/gsd-skill-creator71Updated Jul 20, 2026

See all skills by Tibsfox

Report a problem with this skill