Interpret A/B test results without fooling yourself

You are an experienced experimentation analyst. Help me interpret this A/B test honestly.

Hypothesis: [e.g. "A shorter checkout form will increase completed purchases"]
Primary metric: [e.g. "conversion rate = purchases / visitors who reached checkout"]
Secondary and guardrail metrics: [e.g. "average order value", "refund rate", "page load time"]
Planned design: [e.g. "50/50 split, planned sample of 20,000 per variant, 2 weeks, two-sided test, alpha 0.05"]
What actually happened: [e.g. "ran 16 days; we checked results daily"]

Results:
| Variant | Users | Conversions | [OTHER METRICS] |
| Control | [N] | [N] | |
| Treatment | [N] | [N] | |

Please:
1. Calculate the conversion rate per variant, the absolute and relative difference, and a 95% confidence interval for the difference. Show the formula and the numbers so I can verify them.
2. Check for pitfalls: sample ratio mismatch (is the split close to plan?), peeking or stopping early, multiple metrics or segments tested, novelty effects, too short a run to cover weekly cycles.
3. Look at guardrail metrics for harm.
4. Give a recommendation: ship, do not ship, or run longer, with the main reason.
5. Write a 3-sentence summary for non-technical stakeholders, including uncertainty.

Do not claim certainty the data cannot support. If a calculation needs information I did not give, say so.

Example (excerpt):
Control: 1,840 / 20,110 = 9.15%. Treatment: 1,992 / 20,050 = 9.94%.
Difference: +0.79 percentage points (+8.6% relative). 95% CI: about +0.21 to +1.36 pp.
Sample ratio: 50.07% / 49.93%, consistent with the planned 50/50 split.
Pitfall: results were checked daily; if you had stopped at the first significant day, the false positive rate would be higher than 5%. You ran to the planned sample, so the final test is still valid.
Recommendation: ship, and monitor refund rate for 4 weeks.

When to use it

Use this prompt after an experiment ends and before you announce a winner. It is useful for product managers, marketers and analysts who run tests in tools such as feature flag platforms or email software, and who want a second check on the numbers and the reasoning. It works best for simple two-variant tests on rates or averages.

How to customize it

• [HYPOTHESIS] and [PRIMARY METRIC] should be the ones you set before the test started. Changing them afterwards is a common source of false wins.
• [GUARDRAIL METRICS] are the things that must not get worse, such as refunds, errors or unsubscribes.
• [PLANNED DESIGN] lets the model compare what you planned with what actually happened.
• [WHAT ACTUALLY HAPPENED] should be honest about early checks, stopped tests or traffic problems.
• [RESULTS] can include more rows for more variants or segments; say which comparisons were planned.

Why it works

Most A/B testing mistakes are not arithmetic errors but process errors: stopping when the result looks good, testing many metrics and reporting the one that moved, or ignoring a broken traffic split. The checklist of pitfalls makes the model look for those problems explicitly.

Asking for formulas and intermediate numbers matters because language models can make calculation errors. Showing the work lets you verify the result in a spreadsheet or statistics tool. A confidence interval communicates the size and uncertainty of an effect better than a single p-value, and the stakeholder summary keeps that uncertainty in the final message.

Tips and variations

• Recalculate the key numbers with a dedicated calculator or your testing platform; treat the model’s math as a cross-check.
• For revenue or other skewed metrics, ask about bootstrapping or trimming outliers.
• For many segments, add “Correct for multiple comparisons and label segment results as exploratory.”
• Ask for a pre-registration template before your next test so the plan is written down.
• Low traffic? Ask for a sample size calculation before launching rather than after.
• Write down the decision and the reasoning, so the next test builds on what you learned instead of repeating it.