跳到正文
cccskills
免费在 GitHub 公开

skill-counterfactual-audit

Audit a skill by running a paired probe — the same task once with the skill loaded and once without — segment both traces into goal-directed phases, align phases, and emit a SIP report (surface anchoring, template copy, excess planning, task recovery, off-task artifact). Use whenever a skill is created, modified, or proposed for retirement. Pass-rate is BLIND to most skill effects: CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. Triggers: "audit this skill", "is this skill helping", "retire skill", "before shipping skill", "behavioural impact of skill X", or any skill review event.

中文简介正在准备中。当前显示原文说明。

查看安装方法

包含的文件(1)

  • SKILL.md4.2 KB

SKILL.md(原文)

安装前可以查看交给 AI 代理的指令内容。

Skill Counterfactual Audit

Why

When you ship a skill, pass-rate is the easy metric — and it's blind to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and compare the behaviours directly rather than the headline metric.

How

For each skill under audit:

  1. Curate a probe-task bank (3-5 small tasks per skill domain). Each task should have:
    • A clear goal (a single instruction)
    • An expected category of behaviour
    • A "phase decomposition" (rough segmentation rules)
  2. Run paired — same prompt, same task, run A with skill loaded, run B without.
  3. Segment each trace — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
  4. Align phases — match B phases to A phases; mark new/dropped/displaced phases.
  5. Classify SIP (Skill Influence Pattern) — for each aligned phase, label the effect:
    • surface-anchoring — A copies vocabulary/structure from the skill itself
    • template-copy — A reproduces a skill template literally without adaptation
    • excess-planning — A adds extra planning steps absent in B
    • task-recovery — A recovers from drift that B falls into
    • off-task-artifact — A produces artifacts unrelated to the goal
  6. Emit report to .planning/patterns/skill-audits/<skill>-<timestamp>.md with phase comparison table + SIP distribution + retire/refine/keep recommendation.

Recommendation rule of thumb

Dominant SIPVerdict
task-recovery + correct artifactskeep (genuine value)
surface-anchoring with no task-recoveryrefine (skill is leaking style, not capability)
template-copy on most tasksrefine (lower template specificity)
off-task-artifact on >25% of probesretire (skill is causing distraction)
excess-planning with no completion deltarefine (skill adds ceremony)

When to skip

  • The skill is a workflow guide that doesn't directly shape output (e.g., loop, schedule) — audit dominant SIPs would always be "task-recovery" trivially.
  • No probe-task bank exists for the domain (build one first; defer audit).
  • The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

Integration

  • done-retirement — required gate before retiring a skill.
  • gsd-cleanup — sweeps audits on a milestone-close cadence.
  • Companion to simplify, code-review — those audit code; this audits skill behaviour.

Cross-references

  • Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
  • College: agent-systems / integration-evaluation / agent-paired-trace-audit
  • Related skills: skill-integration (manages the skill lifecycle that this audits)

评价

还没有评价。欢迎分享你的使用感受。

评论

无需登录即可发表。欢迎分享使用感受、提问,或说明在哪些环境下无法运行。

还没有评论。来写第一条吧。

提交后会立即公开。请勿填写邮箱、电话等个人信息。URL 最多 2 个。

0 / 1000 字

此表单受 reCAPTCHA 保护,适用 Google 的隐私权政策和服务条款。

同一仓库的技能

Provides web accessibility best practices for semantic HTML, ARIA, keyboard navigation, color contrast, and screen reader patterns. Use when building UI components, reviewing accessibility, or when user mentions 'a11y', 'accessibility', 'ARIA', 'screen reader', 'keyboard navigation', 'WCAG'.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Active listening techniques for effective communication. Covers attending behaviors, paraphrasing, reflective listening, clarifying questions, empathic response, barriers to listening, listening in conflict, and cross-cultural listening. Use when building listening skills, improving understanding in conversation, mediating disputes, or analyzing communication breakdowns.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Adversarial spec-compliance PR review — cross-references diffs against approved specs, verifies runtime claims against source, detects competing PRs, audits scope/convention compliance. Use before merging.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Provides best practices for AI agent orchestration including MCP servers, A2A protocol, multi-agent coordination, and swarm architectures. Use when designing agent systems, configuring MCP servers, setting up agent teams, or when user mentions 'MCP', 'A2A', 'agent orchestration', 'multi-agent', 'swarm', 'agent team', 'LangGraph', 'CrewAI', 'AutoGen'.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Symbolic manipulation, equation solving, and algebraic structures for mathematical reasoning. Covers distributive law, factoring, completing the square, linear through polynomial equation solving, systems of equations (substitution, elimination, Gaussian elimination, matrix methods), algebraic structures (groups, rings, fields), modular arithmetic, polynomial theory, and inequalities. Use when solving equations, simplifying expressions, working with algebraic structures, or performing symbolic manipulation.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

Understanding how algorithmic systems shape what users see, know, and do -- from recommendation feeds to search ranking to credit scoring to hiring software. Covers the mechanics of recommendation systems, algorithmic bias and its sources, personalization's effects on information diets, opacity and accountability, AI limitations (hallucination, confident wrongness), and the human-in-the-loop question. Use when a learner needs to think critically about why particular content reached them.

中文简介正在准备中。当前显示原文说明。

Tibsfox/gsd-skill-creator712026年7月20日 更新

查看 Tibsfox 的全部技能

报告此技能的问题