Skip to content
cccskills
FreePublished on GitHub

ab-test-setup-and-analysis

Plan and analyze A/B tests for Google and Meta advertising. Estimate required traffic and duration, track results, and assess whether observed differences are statistically significant, meaning unlikely to be explained by chance alone.

OriginalShow the original description

Designs statistically valid split tests for ads, audiences, landing pages, or bid strategies. Calculates required sample sizes before you start, monitors results during the test, and calls winners when statistical significance is actually reached — not when you feel like one is winning. Platform: Google and Meta.

Run score Not tested

See how to install

Good for

  • Planning a test for a new ad headline
  • Estimating traffic and time needed for a landing page test
  • Checking whether interim results justify choosing a winner
  • Deciding what to roll out or test next

Files included (1)

  • SKILL.md2.9 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

28/ A/B Test Setup and Analysis — Google + Meta

What it does

Designs statistically valid split tests for ads, audiences, landing pages, or bid strategies. Calculates required sample sizes before you start, monitors results during the test, and calls winners when statistical significance is actually reached — not when you feel like one is winning.

How it works

Claude takes your test hypothesis, current baseline metrics, and the minimum detectable effect you care about, then calculates how much traffic and time the test needs to produce a reliable result. During the test, it tracks results and tells you whether differences are statistically significant or just noise. It prevents premature test calls that waste the effort.

Practical example

You want to test a new headline variant against your current best performer. Your current ad gets a 2.8% CTR with 1,200 daily impressions. Claude calculates that to detect a 15% improvement (3.22% CTR) with 95% confidence, you need approximately 12,400 impressions per variant — about 10 days at current traffic levels. After 7 days, the new variant shows 3.1% CTR vs 2.7% for control. Claude tells you it's trending positive but not yet significant (p=0.14, need p<0.05) — keep running for 4 more days before making a call.

What you get back

  • Test design with clear hypothesis, control, variant, and success metric
  • Required sample size and estimated test duration before you start
  • Daily monitoring with current results, confidence level, and days remaining
  • Winner call only when significance is reached, with effect size and confidence interval
  • Post-test recommendations: roll out winner, iterate further, or test something else

When to use it

  • Before launching any creative, audience, or landing page test
  • When someone on the team wants to call a winner after 2 days of data
  • For structured testing programs where you need a pipeline of tests running sequentially
  • When clients ask "is this result real or just noise" and you need a definitive answer

Data access (Ryze MCP)

This skill works best with live account data. Connect the free Ryze MCP once and Claude reads your Google Ads, Meta Ads, GA4 and Search Console directly:

  • claude.ai / Claude Desktop: Settings → Connectors → Add custom connector → https://connector.get-ryze.ai/mcp
  • Claude Code: claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp
  • Cursor: Settings → MCP → add the same URL

Setup guide: https://www.get-ryze.ai/how-to-connect-claude-to-google-meta-ads-mcp

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

ab-test-analyzer

FreeSummary

Analyze A/B test results to assess whether observed differences are likely due to chance. Check test duration and sample size, compare audience segments, distinguish statistical findings from practical value, and generate hypotheses for follow-up tests.

  • Evaluating results from a headline A/B test
  • Estimating sample sizes before starting a test
  • Comparing results by device or traffic source
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

Review Google and Meta campaign structures against goals, budgets, and conversion data. Identify excessive segmentation, overlapping targeting, and fragmented budgets, then get consolidation recommendations, an explained account structure, and a migration plan.

  • Reviewing an account inherited from another team or agency
  • Running a quarterly advertising account health check
  • Investigating structural causes of stalled performance
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

Analyze successful Google and Meta ads to find useful patterns in headlines, body text, and calls to action. Generate new copy variants that preserve those patterns, with platform formatting, test priorities, and suggested pairings for comparison.

  • Refreshing ad copy when existing creatives lose effectiveness
  • Starting an ad copy brainstorming session
  • Testing messaging angles for a new audience
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

Review Google Ads extensions, the extra links and details shown with ads, across campaigns. Find missing coverage, weak performance, and outdated information, then get replacement copy based on successful ads and landing pages, plus rollout priorities.

  • Checking extension coverage before launching campaigns
  • Quarterly reviews of outdated promotional details
  • Updating extensions after website changes or product launches
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

Analyze spending, revenue, and conversions across advertising channels to recommend budget shifts. Compare overall efficiency with the revenue gained from additional spending, identify saturation, and produce a gradual reallocation plan with monitoring priorities.

  • Reviewing budget allocation across advertising channels
  • Finding channels with declining returns on extra spending
  • Balancing proven campaigns with new experiments
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

Build a question set to check whether AI assistants mention your brand. Analyze the answers you collect, compare visibility against competitors, trace cited sources, and receive a prioritized list of opportunities to address gaps.

  • Establishing a baseline for brand visibility in AI answers
  • Comparing brand mentions and citations with competitors
  • Tracking visibility changes each quarter
irinabuht12-oss/marketing-skills5.6KUpdated Oct 23, 2026

See all skills by irinabuht12-oss

Report a problem with this skill