An A/B test is the one rigorous instrument in the data science toolkit that licenses causal claims—this change caused that lift—and it earns that license only when the statistical mechanics and engineering foundations are flawless. Most broken experiments are broken before any formal analysis runs: the wrong randomization unit is chosen, no power analysis is conducted, or metrics are cherry-picked after peeking at the results.
In modern software development, A/B testing is not merely a method for optimizing button colors or tweaking copy; it is a fundamental architectural and product strategy. When organizations are making decisions worth millions (e.g., a change expected to yield a $1.5M revenue increase), the cost of statistical illiteracy is immense. This deep dive covers the practice: designing, sizing, monitoring, and analyzing product experiments, alongside the open-source tooling and infrastructural patterns that support them.
Three crucial decisions define an experiment’s identity, and all three must precede the collection of data.
An experiment requires exactly one Primary Metric—chosen in advance, sensitive to the change, and closely connected to business value (often called the Overall Evaluation Criterion, or OEC). In contrast, Guardrail Metrics (e.g., page latency, unsubscribe rates, revenue per user) ride along with a "do no harm" mandate.
Having a dozen "primary" metrics creates a multiple comparisons problem. If you test 20 metrics at a 5% significance level, you are mathematically guaranteed to find at least one false positive on average. If you must use multiple primary metrics, you should apply corrections (like Bonferroni or Benjamini-Hochberg), though this penalizes your statistical power heavily.
The unit of assignment is the entity that is randomized into a bucket. This is usually the user, not the session or pageview. If you randomize by session, a single user returning over multiple days might see both Variant A and Variant B. This contaminates the measurement of user-level behavioral shifts (e.g., retention or cumulative spend). Furthermore, analyzing session-level data when assignment happens at the user level violates the independence assumption—observations from the same user are highly correlated. This silently shrinks your standard errors and artificially inflates your p-values.
A hypothesis is a directional statement paired with a Minimum Detectable Effect (MDE). The MDE is not "the effect we expect to see"; it is the smallest effect that would justify the engineering maintenance cost and risk of the new feature. If rolling out a new checkout flow costs $50K in developer time, your MDE should be sized such that the expected lift exceeds that $50K threshold.
Sample size follows directly from four inputs: the baseline conversion rate, the Minimum Detectable Effect (MDE), the significance level (\alpha, conventionally 0.05), and the statistical power (1 - \beta, conventionally 0.80).
For a standard two-sample proportion test, the required sample size per variant is approximated by:
Where:
The result is usually sobering. Detecting small relative lifts on single-digit baselines requires massive samples. For example, moving a 5.0% baseline to 5.2% (a +4% relative lift) requires over 90,000 users per arm.
An underpowered test is never neutral. It will frequently miss real effects (Type II errors), and when it does achieve significance, the estimated effect magnitude is severely inflated. This phenomenon, known as the "winner's curse" or Type M (Magnitude) error, means that organizations shipping underpowered "winners" will inherit systematic optimism that never materializes in the actual topline metrics. If traffic cannot reach the required power in a reasonable time frame (e.g., 2-4 weeks), you must test bolder changes, adopt more sensitive proxy metrics, or utilize variance reduction techniques.
In practice, product managers and engineers look at experimentation dashboards daily. However, checking a fixed-horizon test continuously and stopping it the moment the p-value dips below 0.05 drastically inflates the false-positive rate. With daily peeking over a two-week period, a nominal 5% false-positive rate can easily balloon to 30% or more. Every "look" at the data is another draw from the noise distribution.
There are two honest ways to handle this:
Open-source experimentation platforms like GrowthBook ship sequential engines precisely because dashboards make peeking irresistible. It is better to use math that anticipates peeking than to pretend your organization has the discipline to ignore a live dashboard.
CUPED (Controlled Experiment Using Pre-Experiment Data) is a technique popularized by Microsoft that reduces metric variance without requiring larger sample sizes. It uses pre-experiment covariate data to explain away the predictable variance in user behavior.
The core idea is to subtract the predictable part of each user's outcome. If we let Y be the experiment metric and X be the covariate (most commonly, the exact same metric measured on that user in the weeks before the experiment), the CUPED adjusted metric is:
Where the optimal \theta that minimizes the variance of the adjusted metric is:
Because the pre-period behavior is unaffected by the treatment assignment, the adjustment remains unbiased. With typical pre- and post-experiment correlations (\rho \approx 0.5 to 0.7), the variance of the metric drops by \rho^2. This means a 30% to 50% reduction in variance, which translates to achieving the same statistical power with roughly a third to half the traffic.
CUPED is often the highest-ROI upgrade available to an experimentation program. Modern platforms generalize this idea further using regression adjustments (e.g., OLS or machine learning models) that incorporate multiple covariates simultaneously.
Before reading the results of any test, you must verify the integrity of the experiment itself. The primary tool for this is the Sample Ratio Mismatch (SRM) check.
An SRM occurs when the observed traffic split between variants deviates significantly from the intended allocation. If a 50/50 test with a million users arrives at a 50.5/49.5 split, this is a massive red alert. The standard check is a Chi-Square goodness-of-fit test:
If the p-value for the SRM check is highly significant (typically < 0.001 to avoid false alarms), the results are structurally untrustworthy, regardless of the metric p-values. SRM is usually a symptom of a deep instrumentation flaw:
A/A Tests involve running the experimentation machinery with identical variants (i.e., both Variant A and Variant A'). If you observe significant "effects" beyond the expected \alpha rate, it exposes bias in your logging pipeline or statistical logic.
Furthermore, experiments in marketplaces, social networks, or constrained-resource environments violate the SUTVA (Stable Unit Treatment Value Assumption) or the "no-interference" assumption. If treating one user cannibalizes resources (like drivers in a ride-sharing app) away from a control user, standard assignment fails. In these cases, you must move away from user-level assignment and adopt Cluster Randomization (randomizing entire cities or social graphs) or Switchback Testing (randomizing time windows within a single market).
While the Average Treatment Effect (ATE) tells you if a feature worked overall, it obscures critical nuances. Heterogeneous Treatment Effects (HTE) or Conditional Average Treatment Effects (CATE) explore whether the feature's impact varies across different segments of the user base.
For instance, a radically simplified navigation bar might drive a massive +15% conversion lift for new users but cause a -5% regression for power users who rely on hidden shortcuts. Detecting HTE requires careful stratification, either at assignment time (Stratified Sampling) or during analysis (Post-Stratification). It allows product teams to tailor rollouts—perhaps launching the simplified navigation only for accounts younger than 30 days.
At the infrastructure layer, assignment is usually handled by a feature-flag system employing deterministic hashing. A typical hashing function looks like:
bucket = hash(user_id + experiment_salt) % 100
This guarantees that users are stably bucketed across sessions without requiring a centralized database lookup.
Whatever stack you choose, there is one non-negotiable rule: you must log the assignment event server-side at the exact moment of exposure. Assignments without corresponding exposure logs are fundamentally unanalyzable, as they dilute the treatment effect with users who were assigned but never actually saw the feature.