Skip to content
cccskills
FreePublished on GitHub

ab-test-setup

Plan, design, and implement A/B tests and experiments with statistical rigor. Covers hypothesis frameworks, sample sizing, metrics, and result analysis.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3 stars(GitHub 3 + cccskills 0)

Run score Not tested

See how to install

Files included (4)

  • SKILL.md3.1 KB
  • references/sample-size-guide.md6.9 KB
  • references/test-templates.md6.1 KB
  • scripts/sample_size_calculator.py12.3 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

A/B Test Setup

Expert experimentation and A/B testing — design tests that produce statistically valid, actionable results.

Before Starting

If .claude/product-marketing-context.md exists, read it first. Then understand: test context (what to improve, what change), current state (baseline conversion, traffic), constraints (complexity, timeline, tools).

Core Principles

  1. Start with a hypothesis — Specific prediction based on data, not "let's see what happens"
  2. Test one thing — Single variable per test
  3. Statistical rigor — Pre-determine sample size, don't peek and stop early
  4. Measure what matters — Primary metric tied to business value, secondary for context, guardrail to prevent harm

Hypothesis Framework

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Test Types

TypeDescriptionTraffic Needed
A/BTwo versions, single changeModerate
A/B/nMultiple variantsHigher
MVTMultiple changes in combinationsVery high

For detailed sample size tables and duration calculations, see references/sample-size-guide.md.


Metrics Selection

  • Primary: Single metric that matters most, tied to hypothesis
  • Secondary: Support interpretation (e.g., time on page, plan distribution)
  • Guardrail: Things that shouldn't get worse (e.g., support tickets, refund rate)

Running the Test

Pre-Launch Checklist

  • Hypothesis documented
  • Primary metric defined
  • Sample size calculated
  • Variants implemented and QA'd
  • Tracking verified

During: Monitor for technical issues, check segment quality, document external factors. Do NOT peek, change variants, or add new traffic sources.


Analyzing Results

  1. Reach sample size? If not, result is preliminary
  2. Statistically significant? (95% confidence = p < 0.05)
  3. Effect size meaningful? Compare to MDE
  4. Secondary metrics consistent?
  5. Guardrail concerns?
  6. Segment differences? (mobile vs desktop, new vs returning)
ResultAction
Significant winnerImplement variant
Significant loserKeep control, learn why
No differenceNeed more traffic or bolder test

Documentation

Document every test: hypothesis, variants (with screenshots), results, decision, learnings. For templates, see references/test-templates.md.


Related Skills

  • page-cro — Ideas for what to test
  • analytics-tracking — Measurement infrastructure
  • campaign-analytics — Fold results into campaign attribution
  • marketing-context — Align hypotheses with ICP and positioning

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

Web/mobile accessibility decision guide: WCAG 2.2 AA, ARIA, keyboard nav, focus management, contrast, testing. Use for: a11y, accessibility, wcag, aria, screen reader, keyboard, contrast.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

Generate, iterate, and scale ad creative for paid advertising across Google, Meta, LinkedIn, and X. Covers platform specs, creative frameworks, and quality validation.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

Musica interactiva para juegos: layering vertical, re-secuenciacion horizontal, stingers, reglas de transicion con sync points, y mapeo de estados de gameplay a musica. Usar para: musica adaptativa, sistema de musica dinamica, transiciones musicales, combat music.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

Design multi-agent systems, create agent architectures, define communication patterns, and build autonomous agent workflows with structured evaluation frameworks.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

Inter-agent communication protocol for C-suite agent teams. Defines invocation syntax, loop prevention, isolation rules, response formats, and quality loops.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

Design production-grade multi-agent workflows with pattern selection, handoff contracts, failure handling, and cost/context controls for sequential, parallel, router, orchestrator, and evaluator topologies.

The English summary is being prepared. Showing the original description.

SebastianLuser/claude-code-arcane3Updated Oct 27, 2026

See all skills by SebastianLuser

Report a problem with this skill