A/B Testing Interview Questions and Answers
Last updated 2026-11-19
A/B testing questions are common in product analyst, data scientist and growth roles. They combine statistics (sample size, p-values, confidence intervals) with judgement: choosing metrics, spotting a broken experiment and deciding what to do with an inconclusive result. Interviewers often give you numbers and expect the arithmetic.
Every calculation below is worked out in full. The statistical background is in statistics for data science interview questions, and the product thinking around metrics is on the product analyst interview questions page.
| Typical rounds | Experiment design case, statistics calculations, metrics discussion |
|---|---|
| Core skills | Sample size, hypothesis tests, variance reduction, experiment pitfalls |
| Questions here | 14 worked questions |
The anatomy of an experiment
- A hypothesis tied to one primary metric, plus guardrail metrics that must not get worse.
- A randomisation unit (user, session, device or cluster) that matches the analysis unit.
- A sample size computed before launch from the baseline, the minimum detectable effect, alpha and power.
- A fixed duration that covers whole weekly cycles.
- Pre-launch checks, and a sample ratio check after launch.
The decision at the end
A result has three parts: whether it is statistically distinguishable from zero, how large the effect plausibly is (the confidence interval), and whether that size matters for the business. Ship only when the interval excludes effects you would regret and the guardrails are intact. An inconclusive test is a legitimate result, not a failure.
A/B Testing Interview Questions and Answers: questions with worked answers
1 Walk through how you would design an A/B test for a new checkout button.
mediumState the hypothesis and the primary metric, for example checkout conversion rate, with guardrails such as payment failures and average order value. Randomise by user so the same person always sees the same variant. Compute the sample size from baseline conversion, the smallest effect worth detecting, alpha of 0.05 and power of 0.8. Run for whole weeks to cover weekday patterns, check for sample ratio mismatch, then analyse with a two-proportion test and a confidence interval. Decide using effect size and guardrails, not the p-value alone.
2 Compute the sample size per group to detect a lift from 10 percent to 11 percent conversion at alpha 0.05 and 80 percent power.
hardFor two proportions, n per group = (z_alpha/2 + z_beta)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2. With 1.96 and 0.84 the squared sum is 7.84. The variance term is 0.1 x 0.9 + 0.11 x 0.89 = 0.09 + 0.0979 = 0.1879. So n = 7.84 x 0.1879 / 0.0001 = 14,731 per group, about 29,500 in total. If traffic is 3,000 users a day split evenly, that is about 10 days, which you would round up to two full weeks.
pythonfrom statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize es = proportion_effectsize(0.11, 0.10) n = NormalIndPower().solve_power(es, alpha=0.05, power=0.8, ratio=1) print(round(n)) # about 14,800 (arcsine method; the pooled formula gives 14,731)3 Variant A converts 500 of 5,000 users and variant B converts 560 of 5,000. Run the test.
mediumRates are 10.0 percent and 11.2 percent. The pooled rate is 1,060 / 10,000 = 0.106. The standard error is sqrt(0.106 x 0.894 x (1/5000 + 1/5000)) = 0.00616. z = 0.012 / 0.00616 = 1.95, giving a two-sided p-value of 0.051. At the 5 percent level this is not significant, even though B is 1.2 points higher. The honest conclusion is that the data are suggestive but inconclusive, so you would extend the test to the planned sample size, not declare a winner.
pythonfrom statsmodels.stats.proportion import proportions_ztest z, p = proportions_ztest([560, 500], [5000, 5000]) print(round(z, 2), round(p, 3)) # 1.95 0.0514 Give a 95 percent confidence interval for the difference in the previous test.
mediumUse the unpooled standard error: sqrt(0.1 x 0.9 / 5000 + 0.112 x 0.888 / 5000) = sqrt(0.000018 + 0.0000199) = 0.00616. The margin is 1.96 x 0.00616 = 0.0121, so the interval for B minus A is 0.012 plus or minus 0.0121, which runs from about -0.0001 to +0.0241, or from -0.01 to 2.41 percentage points. It barely includes zero, which matches the p-value of just over 0.05, and shows the plausible effect could be anything from nothing to a lift of over two points. The interval is more informative than the p-value alone.
5 What is a sample ratio mismatch and how do you test for it? 5,200 users in A and 4,800 in B where a 50/50 split was intended.
mediumSample ratio mismatch (SRM) means the observed split differs from the designed split by more than chance allows, which signals a bug in assignment, logging or filtering, and invalidates the results. Use a chi-square goodness-of-fit test. Expected is 5,000 each, so chi-square = 200^2 / 5000 + 200^2 / 5000 = 8 + 8 = 16 with 1 degree of freedom, p = 0.00006. That is far too unlikely under a fair split, so investigate (redirects, bot filtering, crashes in one variant) before reading any metric.
pythonfrom scipy.stats import chisquare print(chisquare([5200, 4800], f_exp=[5000, 5000])) # statistic 16.0, p about 6e-056 Why is peeking at results and stopping when p < 0.05 a problem?
mediumA p-value is valid only for a single analysis at a pre-set sample size. Each extra look gives another chance for random noise to cross the threshold, so the real false positive rate rises: checking about five times can push it to roughly 14 percent instead of 5. Fix it by fixing the sample size and analysing once, or by using sequential methods designed for continuous monitoring, such as alpha-spending boundaries or always-valid p-values, or a Bayesian analysis with a stated decision rule.
7 What are novelty and primacy effects?
easyA novelty effect is an initial lift because users are curious about something new, which fades. A primacy effect is an initial dip because existing users are used to the old design and need time to adapt. Both make short tests misleading. Run long enough to cover at least one or two full weekly cycles, look at the effect over time for returning users separately from new users, and be cautious about extrapolating a first-week lift.
8 Explain CUPED variance reduction.
hardCUPED uses a pre-experiment covariate X correlated with the metric Y to remove predictable variation. The adjusted metric is Y - theta (X - mean of X), with theta = cov(X, Y) / var(X). Because X is measured before assignment, the adjustment does not bias the treatment effect. Variance shrinks by a factor of (1 - rho^2): with a correlation of 0.7 the variance is cut by 49 percent, so you need about half the sample or can detect smaller effects. It works best for metrics that are persistent per user, such as spend or sessions.
9 You test 10 metrics and one is significant at p = 0.04. What do you conclude?
mediumWith 10 independent metrics at alpha 0.05 the chance that at least one appears significant under no true effect is 1 - 0.95^10 = 0.40, so one hit at 0.04 is weak evidence. Pre-specify one primary metric and treat the rest as secondary or guardrail. Correct for multiple comparisons: Bonferroni needs p below 0.005 for 10 tests, and Benjamini-Hochberg controls the false discovery rate. Treat a lone surprising result as a hypothesis for a follow-up experiment.
10 Simpson's paradox: the new design wins in both segments but loses overall. How is that possible?
hardSuppose the old design had 600 desktop and 100 mobile users, and the new one had 100 desktop and 600 mobile. Desktop conversion is 180 / 600 = 30 percent for old and 40 / 100 = 40 percent for new. Mobile is 5 / 100 = 5 percent for old and 130 / 600 = 21.7 percent for new. New wins in both segments. Overall, old is 185 / 700 = 26.4 percent and new is 170 / 700 = 24.3 percent. The mix of traffic differed, and mobile converts worse. In a proper randomised test the mix should match, so a difference like this signals an assignment problem; otherwise report the segmented results.
11 What if users influence each other, as in a marketplace or social network?
hardStandard A/B tests assume one user's treatment does not affect another user's outcome. In a marketplace, a treated group that books more drivers or items changes what is left for the control, so the measured difference is biased. Options: randomise by cluster (city, region or social-graph cluster), run switchback tests that alternate treatment over time blocks in a region, or use a holdback design. These need larger effects or more time because there are fewer independent units, and the analysis must use the cluster as the unit.
12 Statistical significance versus practical significance, and the minimum detectable effect.
easyWith a very large sample, a tiny difference becomes statistically significant but may be worth nothing in revenue. The minimum detectable effect (MDE) is the smallest change you would care about, and the sample size is chosen to detect it with the desired power. Decide the MDE with the business before launch, based on cost and expected benefit, and judge results by the confidence interval against that threshold: a significant lift whose entire interval is below the MDE is not worth shipping.
13 Frequentist versus Bayesian A/B testing: what differs in practice?
mediumA frequentist test reports a p-value and confidence interval under a fixed plan, answering how surprising the data are if there is no effect. A Bayesian analysis starts from a prior and reports a posterior distribution, so you can say there is a 94 percent probability that B is better, or the expected loss from choosing B. Bayesian results are easier to communicate and adapt to monitoring, but depend on the prior and still need a decision rule and enough data. Poorly chosen stopping rules can mislead under either approach.
Frequently asked questions
How much statistics do I need for A/B testing questions?
Hypothesis tests for proportions and means, confidence intervals, power and sample size, and multiple comparisons. The statistics page covers the foundations.
What metrics should an experiment have?
One primary metric tied to the goal, a few secondary metrics, and guardrail metrics such as latency, errors or revenue that must not degrade.
What if I cannot run a randomised experiment?
Use quasi-experimental methods such as difference-in-differences, matched cohorts or synthetic controls, and be explicit about the extra assumptions.
How do I explain an inconclusive test?
Report the effect estimate with its interval, the power the test had, and what range of effects it rules out. Recommend either more data or deciding on other grounds.
Related guides
- Statistics for Data Science Interview Questions and Answers
- Probability for Data Science Interview Questions and Answers
- Product Analyst Interview Questions and Answers
- Data Scientist Interview Questions and Answers
- Data Analyst Interview Questions and Answers
- Business Analyst Interview Questions and Answers
More in Role-wise Interview Questions by Job and Tech Stack
- Android Developer Interview Questions and Answers
- Angular Interview Questions and Answers (Angular 18 and Later)
- Ansible Interview Questions and Answers
- API Testing and Postman Interview Questions
- Automation Testing Interview Questions and Answers
- AWS Interview Questions and Answers
- Azure Interview Questions and Answers
- Backend Developer Interview Questions and Answers
- Business Analyst Interview Questions and Answers
- Cloud Engineer Interview Questions and Answers
- Computer Vision Interview Questions and Answers
- Cybersecurity Interview Questions and Answers