Autocoder

A/B Testing Interview Questions and Answers

Last updated 2026-11-19

A/B testing questions are common in product analyst, data scientist and growth roles. They combine statistics (sample size, p-values, confidence intervals) with judgement: choosing metrics, spotting a broken experiment and deciding what to do with an inconclusive result. Interviewers often give you numbers and expect the arithmetic.

Every calculation below is worked out in full. The statistical background is in statistics for data science interview questions, and the product thinking around metrics is on the product analyst interview questions page.

Typical roundsExperiment design case, statistics calculations, metrics discussion
Core skillsSample size, hypothesis tests, variance reduction, experiment pitfalls
Questions here14 worked questions

The anatomy of an experiment

The decision at the end

A result has three parts: whether it is statistically distinguishable from zero, how large the effect plausibly is (the confidence interval), and whether that size matters for the business. Ship only when the interval excludes effects you would regret and the guardrails are intact. An inconclusive test is a legitimate result, not a failure.

A/B Testing Interview Questions and Answers: questions with worked answers

  1. 1

    Walk through how you would design an A/B test for a new checkout button.

    medium

    State the hypothesis and the primary metric, for example checkout conversion rate, with guardrails such as payment failures and average order value. Randomise by user so the same person always sees the same variant. Compute the sample size from baseline conversion, the smallest effect worth detecting, alpha of 0.05 and power of 0.8. Run for whole weeks to cover weekday patterns, check for sample ratio mismatch, then analyse with a two-proportion test and a confidence interval. Decide using effect size and guardrails, not the p-value alone.

  2. 2

    Compute the sample size per group to detect a lift from 10 percent to 11 percent conversion at alpha 0.05 and 80 percent power.

    hard

    For two proportions, n per group = (z_alpha/2 + z_beta)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2. With 1.96 and 0.84 the squared sum is 7.84. The variance term is 0.1 x 0.9 + 0.11 x 0.89 = 0.09 + 0.0979 = 0.1879. So n = 7.84 x 0.1879 / 0.0001 = 14,731 per group, about 29,500 in total. If traffic is 3,000 users a day split evenly, that is about 10 days, which you would round up to two full weeks.

    python
    from statsmodels.stats.power import NormalIndPower
    from statsmodels.stats.proportion import proportion_effectsize
    es = proportion_effectsize(0.11, 0.10)
    n = NormalIndPower().solve_power(es, alpha=0.05, power=0.8, ratio=1)
    print(round(n))    # about 14,800 (arcsine method; the pooled formula gives 14,731)
  3. 3

    Variant A converts 500 of 5,000 users and variant B converts 560 of 5,000. Run the test.

    medium

    Rates are 10.0 percent and 11.2 percent. The pooled rate is 1,060 / 10,000 = 0.106. The standard error is sqrt(0.106 x 0.894 x (1/5000 + 1/5000)) = 0.00616. z = 0.012 / 0.00616 = 1.95, giving a two-sided p-value of 0.051. At the 5 percent level this is not significant, even though B is 1.2 points higher. The honest conclusion is that the data are suggestive but inconclusive, so you would extend the test to the planned sample size, not declare a winner.

    python
    from statsmodels.stats.proportion import proportions_ztest
    z, p = proportions_ztest([560, 500], [5000, 5000])
    print(round(z, 2), round(p, 3))    # 1.95 0.051
  4. 4

    Give a 95 percent confidence interval for the difference in the previous test.

    medium

    Use the unpooled standard error: sqrt(0.1 x 0.9 / 5000 + 0.112 x 0.888 / 5000) = sqrt(0.000018 + 0.0000199) = 0.00616. The margin is 1.96 x 0.00616 = 0.0121, so the interval for B minus A is 0.012 plus or minus 0.0121, which runs from about -0.0001 to +0.0241, or from -0.01 to 2.41 percentage points. It barely includes zero, which matches the p-value of just over 0.05, and shows the plausible effect could be anything from nothing to a lift of over two points. The interval is more informative than the p-value alone.

  5. 5

    What is a sample ratio mismatch and how do you test for it? 5,200 users in A and 4,800 in B where a 50/50 split was intended.

    medium

    Sample ratio mismatch (SRM) means the observed split differs from the designed split by more than chance allows, which signals a bug in assignment, logging or filtering, and invalidates the results. Use a chi-square goodness-of-fit test. Expected is 5,000 each, so chi-square = 200^2 / 5000 + 200^2 / 5000 = 8 + 8 = 16 with 1 degree of freedom, p = 0.00006. That is far too unlikely under a fair split, so investigate (redirects, bot filtering, crashes in one variant) before reading any metric.

    python
    from scipy.stats import chisquare
    print(chisquare([5200, 4800], f_exp=[5000, 5000]))   # statistic 16.0, p about 6e-05
  6. 6

    Why is peeking at results and stopping when p < 0.05 a problem?

    medium

    A p-value is valid only for a single analysis at a pre-set sample size. Each extra look gives another chance for random noise to cross the threshold, so the real false positive rate rises: checking about five times can push it to roughly 14 percent instead of 5. Fix it by fixing the sample size and analysing once, or by using sequential methods designed for continuous monitoring, such as alpha-spending boundaries or always-valid p-values, or a Bayesian analysis with a stated decision rule.

  7. 7

    What are novelty and primacy effects?

    easy

    A novelty effect is an initial lift because users are curious about something new, which fades. A primacy effect is an initial dip because existing users are used to the old design and need time to adapt. Both make short tests misleading. Run long enough to cover at least one or two full weekly cycles, look at the effect over time for returning users separately from new users, and be cautious about extrapolating a first-week lift.

  8. 8

    Explain CUPED variance reduction.

    hard

    CUPED uses a pre-experiment covariate X correlated with the metric Y to remove predictable variation. The adjusted metric is Y - theta (X - mean of X), with theta = cov(X, Y) / var(X). Because X is measured before assignment, the adjustment does not bias the treatment effect. Variance shrinks by a factor of (1 - rho^2): with a correlation of 0.7 the variance is cut by 49 percent, so you need about half the sample or can detect smaller effects. It works best for metrics that are persistent per user, such as spend or sessions.

  9. 9

    You test 10 metrics and one is significant at p = 0.04. What do you conclude?

    medium

    With 10 independent metrics at alpha 0.05 the chance that at least one appears significant under no true effect is 1 - 0.95^10 = 0.40, so one hit at 0.04 is weak evidence. Pre-specify one primary metric and treat the rest as secondary or guardrail. Correct for multiple comparisons: Bonferroni needs p below 0.005 for 10 tests, and Benjamini-Hochberg controls the false discovery rate. Treat a lone surprising result as a hypothesis for a follow-up experiment.

  10. 10

    Simpson's paradox: the new design wins in both segments but loses overall. How is that possible?

    hard

    Suppose the old design had 600 desktop and 100 mobile users, and the new one had 100 desktop and 600 mobile. Desktop conversion is 180 / 600 = 30 percent for old and 40 / 100 = 40 percent for new. Mobile is 5 / 100 = 5 percent for old and 130 / 600 = 21.7 percent for new. New wins in both segments. Overall, old is 185 / 700 = 26.4 percent and new is 170 / 700 = 24.3 percent. The mix of traffic differed, and mobile converts worse. In a proper randomised test the mix should match, so a difference like this signals an assignment problem; otherwise report the segmented results.

  11. 11

    What if users influence each other, as in a marketplace or social network?

    hard

    Standard A/B tests assume one user's treatment does not affect another user's outcome. In a marketplace, a treated group that books more drivers or items changes what is left for the control, so the measured difference is biased. Options: randomise by cluster (city, region or social-graph cluster), run switchback tests that alternate treatment over time blocks in a region, or use a holdback design. These need larger effects or more time because there are fewer independent units, and the analysis must use the cluster as the unit.

  12. 12

    Statistical significance versus practical significance, and the minimum detectable effect.

    easy

    With a very large sample, a tiny difference becomes statistically significant but may be worth nothing in revenue. The minimum detectable effect (MDE) is the smallest change you would care about, and the sample size is chosen to detect it with the desired power. Decide the MDE with the business before launch, based on cost and expected benefit, and judge results by the confidence interval against that threshold: a significant lift whose entire interval is below the MDE is not worth shipping.

  13. 13

    Frequentist versus Bayesian A/B testing: what differs in practice?

    medium

    A frequentist test reports a p-value and confidence interval under a fixed plan, answering how surprising the data are if there is no effect. A Bayesian analysis starts from a prior and reports a posterior distribution, so you can say there is a 94 percent probability that B is better, or the expected loss from choosing B. Bayesian results are easier to communicate and adapt to monitoring, but depend on the prior and still need a decision rule and enough data. Poorly chosen stopping rules can mislead under either approach.

Frequently asked questions

How much statistics do I need for A/B testing questions?

Hypothesis tests for proportions and means, confidence intervals, power and sample size, and multiple comparisons. The statistics page covers the foundations.

What metrics should an experiment have?

One primary metric tied to the goal, a few secondary metrics, and guardrail metrics such as latency, errors or revenue that must not degrade.

What if I cannot run a randomised experiment?

Use quasi-experimental methods such as difference-in-differences, matched cohorts or synthetic controls, and be explicit about the extra assumptions.

How do I explain an inconclusive test?

Report the effect estimate with its interval, the power the test had, and what range of effects it rules out. Recommend either more data or deciding on other grounds.

Related guides

More in Role-wise Interview Questions by Job and Tech Stack

See all in this section