1. Complete Separation of Experiences

First, take a look at your user randomization and group splitting module. Does it cleanly split off a portion of your users for the experimentation group? Are they experiencing only the new design (or model, or whatever)?

It’s important to cleanly and completely separate the experiences between the two groups. Suppose you are testing a new button for your website. If the button appears on every page, then make sure the same user sees the same button everywhere. It’ll be better to split by user ID (if available) or user sessions instead of individual page visits.

Also watch out for the possibility that some of your users have been permanently “trained” by the old model or design and prefer the way things were before. In their KDD 2012 paper, Kohavi et al. calls this the carryover effect. Such users carry the “baggage of the old” and may return biased answers for any new model. If you think this might be the case, think about acquiring a brand new set of users or randomizing the test buckets.

It’s always good to do some A/A testing to make sure that your testing framework is sound. In other words, perform the randomization and the split, but test both groups on the same model or design. See if there are any observable differences. Only move to A/B testing if the system passes the A/A test.

2. Which Metric?

The next important question is, on which metric should you evaluate the model? Ultimately, the right metric is probably a business metric. But this may not be easily measurable in the system. For instance, search engines care about the number of users, how long they spend on the site, and their overall market share. Comparison statistics are not readily available to the live system. So they will need to approximate the ultimate business metric of market share with measurable ones like number of unique visitors per day and average session length. In practice, short-term, measurable live metrics may not always align with long-term business metrics, and it can be tricky to design the right metric.

Backing up for a second, there are four classes of metrics to think about: business metrics, measurable live metrics, offline evaluation metrics, and training metrics. We just discussed the difference between business metrics and live metrics that can be measured. Offline evaluation metrics are things like the classification, regression, and ranking metrics we discussed previously. The training metric is the loss function that is optimized during the training process. (For example, a support vector machine optimizes a combination of the norm of the weight vector and misclassification penalties.)

The optimal scenario is where all four of those metrics are either exactly the same or are linearly aligned with each other. The former is impossible. The latter is unlikely. So the next thing to shoot for is that these metrics always increase or decrease with each other. However, you may still encounter situations where a linear decrease in RMSE (a regression metric) does not translate to a linear increase in click-through rates. (Kohavi et al. described some interesting examples in their KDD 2012 paper.) Keep this in mind and save your efforts to optimize where it counts the most. You should always be tracking all of these metrics, so that you know when things go out of whack—usually a sign of distribution drift or software and instrumentation bugs.

3. How Much Change Counts as Real Change?

Once you’ve settled on the metric, the next question is, how much of a change in this metric matters? This is required for picking the number of observations you need for the experiment. Like question #2, this is probably not solely a data science question but a business question. Pick a reasonable value up front and stick to it. Avoid the temptation to shift it later, as you start to see the results.

4. One-Sided or Two-Sided Test?

Making the wrong choice here could get you (almost) fired. One-sided (or one-tailed) tests only test whether the new model is better than the baseline. It does not tell you if it is in fact worse. You should always test both, unless you are confident it can never be worse, or there are zero consequences for it being worse. A two-sided (or two-tailed) test allows the new model to be either better or worse than the original. It still requires a separate check for which is the case.

5. How Many False Positives Are You Willing to Tolerate?

A false positive in A/B testing means that you’ve rejected the null hypothesis when the null hypothesis is true. In other words, you’ve decided that your model is better than the baseline when it isn’t better than the baseline. What’s the cost of a false positive? The answer depends on the application.

In a drug effectiveness study, a false positive could cause the patient to use an ineffective drug. Conversely, a false negative could mean not using a drug that is effective at curing the disease. Both cases could have a very high cost to the patient’s health.

In a machine learning A/B test, a false positive might mean switching to a model that should increase revenue when it doesn’t. A false negative means missing out on a more beneficial model and losing out on potential revenue increase.

A statistical hypothesis test allows you to control the probability of false positives by setting the significance level, and false negatives via the power of the test. If you pick a false positive rate of 0.05, then out of every 20 new models that don’t improve the baseline, on average 1 of them will be falsely identified by the test as an improvement. Is this an acceptable outcome to the business?

6. How Many Observations Do You Need?

The number of observations is partially determined by the desired statistical power. This must be determined prior to running the test. A common temptation is to run the test until you observe a significant result. This is wrong.

The power of a test is its ability to correctly identify the positives, e.g., correctly determine that a new model is doing well when it is in fact superior. It can be written as a formula that involves the significance level (question #5), the difference between the control and experimentation metrics (question #3), and the size of the samples (the number of observations included in the control and the experimentation group). You pick the right value for power, significance level, and the desired amount of change. Then you can compute how many observations you need in each group. A recent blog post from StitchFix goes through the power analysis in minute detail.

As explained in detail on Evan Miller’s website, do NOT stop the test until you’ve accumulated this many observations! Specifically, do not stop the test as soon as you detect a “significant” difference. The answer is not to be trusted since it doesn’t yet have the statistical power for good decision making.

7. Is the Distribution of the Metric Gaussian?

The vast majority of A/B tests use the t-test. But the t-test makes assumptions that are not always satisfied by all metrics. It’s a good idea to look at the distribution of your metric and check whether the assumptions of the t-test are valid.

The t-test assumes that the two populations are Gaussian distributed. Does your metric fit a Gaussian distribution? The common hand-wavy justification is to say, “Almost everything converges to a Gaussian distribution due to the Central Limit Theorem.” This is usually true when:

1. The metric is an average.
2. The distribution of metric values has one mode.
3. The metric is distributed symmetrically around this mode.

These are actually easily violated in real-world situations. For example, the accuracy or the click-through rate is an average, but the area under the curve (AUC) is not. (It is an integral.) The distribution of the metric may not have one mode if there are multiple user populations within the control or experimental group. The metric is not symmetric if, say, it can be any positive number but can never be negative. Kohavi et al. gives examples of metrics that are definitely not Gaussian and whose standard error does not decrease with longer tests. For example, metrics involving counts are better modeled as negative binomials.

When these assumptions are violated, the distribution may take longer than usual to converge to a Gaussian, or not at all. Usually, the average of more than 30 observations starts to look like a Gaussian. When there is a mixture of populations, however, it will take much longer. Here are a few rules of thumb that can mitigate the violation of t-test assumptions:

1. If the metric is nonnegative and has a long tail, i.e., it’s a count of some sort, take the log transform.
2. Alternatively, the family of power transforms tends to stabilize the variance (decrease the variance or at least make it not dependent on the mean) and make the distribution more Gaussian-like.
3. The negative binomial is a better distribution for counts.
4. If the distribution looks nowhere near a Gaussian, don’t use the t-test. Pick a nonparametric test that doesn’t make the Gaussian assumption, such as the Mann-Whitney U test.

8. Are the Variances Equal?

Okay, you checked and double-checked and you’re really sure that the distribution is a Gaussian, or will soon become a Gaussian. Fine. Next question: are the variances equal for the control and the experimental group?

If the groups are split fairly (uniformly at random), the variances are probably equal. However, there could be subtle biases in your stream splitter (see question #1). Or perhaps one population is much smaller compared to the other. Welch’s t-test is a little-known alternative to the much more common Student’s t-test. Unlike Student’s t-test, Welch’s t-test does not assume equal variance. For this reason, it is a more robust alternative. Here’s what Wikipedia says about the advantages and limitations of Welch’s t-test:

Welch’s t-test is more robust than Student’s t-test and maintains type I error rates close to nominal for unequal variances and for unequal sample sizes. Furthermore, the power of Welch’s t-test comes close to that of Student’s t-test, even when the population variances are equal and sample sizes are balanced.

It is not recommended to pre-test for equal variances and then choose between Student’s t-test or Welch’s t-test. Rather, Welch’s t-test can be applied directly and without any substantial disadvantages to Student’s t-test as noted above. Welch’s t-test remains robust for skewed distributions and large sample sizes. Reliability decreases for skewed distributions and smaller samples, where one could possibly perform Welch’s t-test on ranked data.

In practice, this may not make too big of a difference, because the t-distribution is well approximated by the Gaussian when the sample sizes are larger than 20. However, Welch’s t-test is a safe choice that works regardless of sample size or whether the variance is equal. So why not?

9. What Does the p-Value Mean?

As Cosma Shalizi explained in his very detailed and technical blog post, most people interpret the p-value incorrectly. A small p-value does not imply a significant result. A smaller p-value does not imply a more significant result. The p-value is a function of the size of the samples, the difference between the two populations, and how well we can estimate the true means. I’ll leave the curious, statistically minded reader to digest the blog post (highly recommended!). The upshot is that, in addition to running the hypothesis test and computing the p-value, one should always check the confidence interval of the two population mean estimates. If the distribution is close to being Gaussian, then the usual standard error estimation applies. Otherwise, compute a bootstrap estimate, which we discussed in Chapter 3. This can differentiate between the two cases of “there is indeed a significant difference between the two populations” versus “I can’t tell whether there is a difference because the variances of the estimates are too high so I can’t trust the numbers.”

10. Multiple Models, Multiple Hypotheses

So you are a hard-working data scientist and you have not one but five new models you want to test. Or maybe 328 of them. Your website has so much traffic that you have no problem splitting off a portion of the incoming traffic to test each of the models at the same time. Parallel A₁/.../A_m/B testing, here we come!

But wait, now you are in the situation of multiple hypothesis testing. Remember the false positive rate we talked about in question #5? Testing multiple hypotheses increases the overall false positive probability. If one test has a false positive rate of 0.05, then the probability that none of the 20 tests makes a false positive drops precipitously to (1 – 0.05)²⁰ = 0.36. What’s more, this calculation assumes that the tests are independent. If the tests are not independent (i.e., maybe your 32 models all came from the same training dataset?), then the probability of a false positive may be even higher.

Benjamini and Hochberg proposed a useful method for dealing with false positives in multiple tests. In their 1995 paper, “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” they proposed a modified procedure that orders the p-values from each test and rejects the null hypothesis for the smallest normalized p-values (, where q is the desired significance level, m is the total number of tests, and i is the ranking of the p-value). This test does not assume that the tests are independent or are normally distributed, and has more statistical power than the classic Bonferroni correction.

Even without running multiple tests simultaneously, you may still run into the multiple hypothesis testing scenario. For instance, if you are changing your model based on live test data, submitting new models until something achieves the acceptance threshold, then you are essentially running multiple tests sequentially. It’s a good idea to apply the Benjamini-Hochberg procedure (or one of its derivatives) to control the false discovery rate in this situation as well.

11. How Long to Run the Test?

The answer to how long to run your A/B test depends not just on the number of observations you need in order to achieve the desired statistical power (question #6). It also has to do with the user experience.

In some fields, such as pharmaceutical drug testing, running the test too long has ethical consequences for the user; if the drug is already proven to be effective, then stopping the trial early may save lives in the control group. Balancing the need for early stopping and sufficient statistical power led to the study of sequential analysis, where early stopping points are determined a priori at the start of the trials.

In most newly emergent machine learning applications, running the test longer is not as big of a problem. More likely, the constraint is distribution drift, where the behavior of the user changes faster than one can collect enough observations. (See question #12.)

When determining the length of a trial, it’s important to go beyond what’s known as the Novelty effect. When users are switched to a new experience, their initial reactions may not be their long-term reactions. In other words, if you are testing a new color for a button, the user may initially love the button and click it more often, just because it’s novel, or she may hate the new color and never touch it, but eventually she would get used to the new color and behave as she did before. It’s important to run the trial long enough to get past the period of the “shock of the new.”

The metric may also display seasonality. For instance, the website traffic may behave one way during the day and another way at night, or perhaps people buy different types of clothes in the summer versus fall. It’s important to take this into account and discount foreseeable changes when collecting data for the trial.

12. Catching Distribution Drift

We introduced the notion of distribution drift in Chapter 1. Many machine learning models make a stationarity assumption, that the data looks and behaves one way for all eternity. But this is not true in practice. The world changes quickly. Nothing lasts forever. Translated into statistical terms, this means that the distribution of the data will drift from what the model was originally trained upon.

Distribution drift invalidates the current model. It no longer performs as well as before. It needs to be updated.

To catch distribution drift, it’s a good idea to monitor the offline metric (used for evaluations during offline testing/prototyping) on live data, in addition to online testing. If the offline metric changes significantly, then it is time to update the model by retraining on new data.