本文へ移動
cccskills
無料GitHub で公開

skill-counterfactual-audit

Audit a skill by running a paired probe — the same task once with the skill loaded and once without — segment both traces into goal-directed phases, align phases, and emit a SIP report (surface anchoring, template copy, excess planning, task recovery, off-task artifact). Use whenever a skill is created, modified, or proposed for retirement. Pass-rate is BLIND to most skill effects: CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. Triggers: "audit this skill", "is this skill helping", "retire skill", "before shipping skill", "behavioural impact of skill X", or any skill review event.

インストール方法を見る

含まれるファイル(1)

  • SKILL.md4.2 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

Skill Counterfactual Audit

Why

When you ship a skill, pass-rate is the easy metric — and it's blind to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and compare the behaviours directly rather than the headline metric.

How

For each skill under audit:

  1. Curate a probe-task bank (3-5 small tasks per skill domain). Each task should have:
    • A clear goal (a single instruction)
    • An expected category of behaviour
    • A "phase decomposition" (rough segmentation rules)
  2. Run paired — same prompt, same task, run A with skill loaded, run B without.
  3. Segment each trace — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
  4. Align phases — match B phases to A phases; mark new/dropped/displaced phases.
  5. Classify SIP (Skill Influence Pattern) — for each aligned phase, label the effect:
    • surface-anchoring — A copies vocabulary/structure from the skill itself
    • template-copy — A reproduces a skill template literally without adaptation
    • excess-planning — A adds extra planning steps absent in B
    • task-recovery — A recovers from drift that B falls into
    • off-task-artifact — A produces artifacts unrelated to the goal
  6. Emit report to .planning/patterns/skill-audits/<skill>-<timestamp>.md with phase comparison table + SIP distribution + retire/refine/keep recommendation.

Recommendation rule of thumb

Dominant SIPVerdict
task-recovery + correct artifactskeep (genuine value)
surface-anchoring with no task-recoveryrefine (skill is leaking style, not capability)
template-copy on most tasksrefine (lower template specificity)
off-task-artifact on >25% of probesretire (skill is causing distraction)
excess-planning with no completion deltarefine (skill adds ceremony)

When to skip

  • The skill is a workflow guide that doesn't directly shape output (e.g., loop, schedule) — audit dominant SIPs would always be "task-recovery" trivially.
  • No probe-task bank exists for the domain (build one first; defer audit).
  • The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

Integration

  • done-retirement — required gate before retiring a skill.
  • gsd-cleanup — sweeps audits on a milestone-close cadence.
  • Companion to simplify, code-review — those audit code; this audits skill behaviour.

Cross-references

  • Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
  • College: agent-systems / integration-evaluation / agent-paired-trace-audit
  • Related skills: skill-integration (manages the skill lifecycle that this audits)

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Provides web accessibility best practices for semantic HTML, ARIA, keyboard navigation, color contrast, and screen reader patterns. Use when building UI components, reviewing accessibility, or when user mentions 'a11y', 'accessibility', 'ARIA', 'screen reader', 'keyboard navigation', 'WCAG'.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Active listening techniques for effective communication. Covers attending behaviors, paraphrasing, reflective listening, clarifying questions, empathic response, barriers to listening, listening in conflict, and cross-cultural listening. Use when building listening skills, improving understanding in conversation, mediating disputes, or analyzing communication breakdowns.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Adversarial spec-compliance PR review — cross-references diffs against approved specs, verifies runtime claims against source, detects competing PRs, audits scope/convention compliance. Use before merging.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Provides best practices for AI agent orchestration including MCP servers, A2A protocol, multi-agent coordination, and swarm architectures. Use when designing agent systems, configuring MCP servers, setting up agent teams, or when user mentions 'MCP', 'A2A', 'agent orchestration', 'multi-agent', 'swarm', 'agent team', 'LangGraph', 'CrewAI', 'AutoGen'.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Symbolic manipulation, equation solving, and algebraic structures for mathematical reasoning. Covers distributive law, factoring, completing the square, linear through polynomial equation solving, systems of equations (substitution, elimination, Gaussian elimination, matrix methods), algebraic structures (groups, rings, fields), modular arithmetic, polynomial theory, and inequalities. Use when solving equations, simplifying expressions, working with algebraic structures, or performing symbolic manipulation.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Understanding how algorithmic systems shape what users see, know, and do -- from recommendation feeds to search ranking to credit scoring to hiring software. Covers the mechanics of recommendation systems, algorithmic bias and its sources, personalization's effects on information diets, opacity and accountability, AI limitations (hallucination, confident wrongness), and the human-in-the-loop question. Use when a learner needs to think critically about why particular content reached them.

日本語の概要は準備中です。原文の説明を表示しています。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Tibsfox のスキルをすべて見る

このスキルの問題を報告する