Analyze A/B test results to decide whether to launch a change, extend testing, stop, or investigate further. Check sample size, statistical significance, uncertainty ranges, and effects on other important metrics, then summarize the recommendation and next steps.
Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.
Review the instructions the agent will receive before you install.
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
You are analyzing A/B test results for $ARGUMENTS.
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
Understand the experiment:
Validate the test setup:
Calculate statistical significance:
If the user provides raw data, generate and run a Python script to calculate these.
Check guardrail metrics:
Interpret results:
| Outcome | Recommendation |
|---|---|
| Significant positive lift, no guardrail issues | Ship it — roll out to 100% |
| Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping |
| Not significant, positive trend | Extend the test — need more data or larger effect |
| Not significant, flat | Stop the test — no meaningful difference detected |
| Significant negative lift | Don't ship — revert to control, analyze why |
Provide the analysis summary:
## A/B Test Results: [Test Name]
**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]
| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |
**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]
Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Organize customer feature requests into themes and rank the top three against product goals, customer value, effort, and risk. For each recommendation, explain the underlying need, possible alternatives, uncertain assumptions, and simple ways to test them.
Compare growth options across existing and new products and markets using the Ansoff Matrix, a four-part planning framework. Assess opportunities, resources, competition, and risks, then develop market launch plans and a phased roadmap for priority options.
Choose the first customer segment for a product launch by comparing urgent needs, willingness to pay, achievable market share, and referral potential. Get scored recommendations, a priority segment, a 90-day acquisition plan, and expansion options.
Plan low-effort experiments to check assumptions about an existing product before building a feature. Compare prototypes, A/B tests, and other methods, with a clear assumption, procedure, metric, and success threshold for each experiment.
Turn a new product idea into a testable hypothesis and two or three simple experiments. Define target customers, expected actions, measurement methods, and success thresholds to assess demand through behavior such as sign-ups, pre-orders, or paid commitments.
Generate improvement ideas for an existing product from product management, design, and engineering perspectives. Produce five ideas per perspective, then select the top five based on goals, potential impact, feasibility, and differentiation, with reasons and assumptions to test.