Criteria for success

You might be surprised that I start with the criteria for success instead of the definition of the intervention. After all, shouldn’t we know what we’re testing first? Unfortunately, a common cause of failure is the decision to test something (often the latest management fad or something that your boss’s boss read about) without a clear sense of what we’re solving for.

What does success look like? Ideally, it should be expressed in terms of dollars of additional revenue or decreased cost. If that’s not possible, you should at least have an operational metric that seems reasonably connected to business outcomes. In the case of the 1-click button, we can obviously look at booking rates. But what about the duration of website visit? If the button shortens how long it takes someone to book a trip but otherwise doesn’t increase bookings in any way, is that a success or not? What about satisfaction with the booking experience and net promoter score? These may not translate directly into dollar figures, but it is reasonable to assume that improving them has a positive impact for your business.

A compromise you’ll often have to make at this point is to use “leading indicators”—basically causes of the variables you’re ultimately interested in. For example, sign-up can be used as a leading indicator for usage, usage for amount purchased, etc. This will allow you to report results much earlier than if you used long-term business metrics. You may care about your customers’ LTV (Life-Time Value, the total amount they’ll spend with your company) but have to settle for 3-month booking amount.

This is sometimes done informally, by picking operational metrics as targets for experimentation and assuming with some hand-waving that these will end up benefiting the company’s bottom line. Of course, armed with this book, we can do better. We can validate and measure the causal connection between a short-term operational metric and long-term business outcomes through an observational study or a dedicated experiment, as we’ll see later.

Here again, the goal is to make good business decisions. I don’t want to be a fanatic of dollar figures, because it would be unduly restrictive and would exclude a large range of business improvements. At the same time, you want to make sure that you have a measurable target metric that you’ll be able to track. There are several potential pitfalls that you should avoid here.

The first one is picking something you can’t measure reliably. What would it mean to say “the 1-click button makes the booking experience easier”? How would you measure it? Asking the product manager or the product owner to decide after the fact whether there has been an improvement is not a measurement. “Our customers rate the website as easier to navigate, as measured by a 2-question survey at the end of a visit” may be an imperfect proxy but at least it can be measured.

The second pitfall is to write down a laundry list of metrics, such as “success would be an improvement in booking rate, booking amount, customer satisfaction or net promoter score”, or even worse, to wait until after seeing the results to determine the metrics for success—e.g. you were originally thinking that the experiment would improve customer experience, but when the results come, customer experience is flat and average website session duration has improved. The problem with that approach is that it increases the risk of false positives (calling the result a success when it was a random fluke)¹. It’s ok however to have up to 2 or 3 target metrics that are clearly defined before the experiment as long as you take that into account when analyzing the results (more on that later).

Definition of intervention

Once we have our criteria for success, we can work on defining our intervention. You may remember our earlier discussion of customer satisfaction in chapter 5, and how many rather hidden assumptions were going into a CSAT score. The same problem occurs here: there can be a large gap between a business idea, so used and familiar that everyone immediately feels they know what it is, and a specific implementation.

On the face of it, what could be simpler than a “1-click booking” button? Most of us have seen it implemented on the website of an online retailer and the idea seems perfectly straightforward. But if you think about the nitty-gritty details of how it would be implemented, there are actually a lot of questions to answer, each with multiple possible answers:

• At which point in the process does the button become available?

• Where is the button located?

• What does the button look like? Is it in the same colors as the other buttons on the page or does it stand out in a bright and rich color?

• What is written on the button?

• What information do we need about a customer to make 1-click booking available and how do we make sure we have it?

• What happens after the customer clicks the button? What page are they taken to, and what, if any, actions do they still have to take?

• Etc., etc.

Let’s say for example that the website’s color theme is pastel green and blue, to hint at nature and travel, and the new button is a shiny red. If that button increases booking, it might be because of the attractiveness of 1-click booking, but it could also be because customers otherwise struggle to see the normal booking button and give up. In that case, the cause of the increase in booking is really “a more visible booking button” rather than “a 1-click booking button”, but you can’t distinguish between the two because the 1-click button was also more visible. Unfortunately, you can’t ever just test one idea, as you’re always also testing the many ways in which it is implemented.

The lesson here is that A/B testing is a powerful but narrow tool. You need to be careful not to make—or let others make—grand statements about what a specific experiment says. It is definitely easier said than done, because business partners often want answers that are broad, clear cut and without fine prints. With that said, even simply stating in your presentation that you’re testing a specific implementation and not a general idea can be useful, for example “this experiment will test the impact of a 1-click button under such and such conditions, and its results should not be interpreted as applying to booking buttons more broadly”.

A better yet alternative would be to test different implementations of the same concept in the experiment. If four slightly different 1-click button treatments have the same impact, you can more confidently draw conclusions regarding the general impact of 1-click booking; on the other hands, if they have very different impacts, then this suggests that implementation matters a lot and that you need to be very careful with your conclusions regarding how a specific implementation will generalize.

Logic for success

Once we know our criteria for success and we have defined our intervention, the last step is to connect the two through the logic for success: why and how would our intervention impact our criteria for success?

This is another surprisingly common source of failure for experiments: a problem has been identified and someone decides to implement the latest management fad that they have been thinking about recently, even though it’s unclear why it would help with this specific problem. Or someone decides that a more attractive and simpler User Interface will increase purchase amounts. To have confidence in our experiment, you need to be able to articulate a reasonable behavioral story². In the case of 1-click booking, you might hypothesize that customers identify an attractive booking but give up before completing their booking because the booking process is cumbersome; the 1-click button would impact the probability of booking by shortening and simplifying the booking process. Drawing a CD here comes handy to visualize your logic (figure 8-1).

Figure 2-1. Articulating our logic for success with a CD

Overall, articulating your logic for success has two benefits.

First, it is often testable in itself. Does a large number of customers actually drop off between starting a booking and completing it? If so, the hypothesized logic makes sense from a behavioral data perspective. But if, for example, most customers leave without having started the booking process with a specific product, e.g. because they couldn’t find something that appealed to them or they were overwhelmed with the number of options, then AirCnC is trying to solve the wrong problem; it’s unlikely that offering 1-click booking will improve their numbers.

Ask yourself: what would confirm or infirm our logic? What would the data look like if it were true or false? If you don’t have the necessary data at hand, it might be worth running a preliminary test, such as bringing in 10 people for User Experience testing—e.g. observing them trying to use your website while they’re thinking out loud. This may not fully confirm or infirm your logic, but it will probably give you some indication, at a fraction of the development cost for the solution considered. As the often-cited quote by Albert Einstein goes, “If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions”.

The second benefit of articulating the logic of your intervention is that it will generally give you a sense of the potential upside. What would the numbers look like, in the best-case scenario, if the problem at hand was solved? Assuming that all customers who drop off during the booking process would complete it with 1-click booking, how much would the booking rate increase? This is the best-case scenario from the perspective of the experiment because we’re assuming that it would fully solve the problem, which is unlikely in reality. If the increase in booking rate under this scenario would not pay for the implementation of the 1-click booking, then don’t even bother testing it.

Once you’ve validated that your best-case scenario would be profitable, you can start thinking about your most likely scenario. How much would we expect 1-click booking to improve the booking rate? There is undoubtedly a lot of subjectivity and uncertainty involved, but it can still be a worthwhile exercise, by making explicit people’s assumptions and gut feelings. Use your business sense and understanding of the processes. If most customers are dropping off at the exact same step in the process, e.g. payment, then it’s likely that there is something wrong with that specific step—people don’t all run out of patience at the exact same time. You can then compare the expected benefits with the cost of implementing the solution. Is it still worthwhile?

You can also approach this question from the other side: start by determining the break-even point of the solution, i.e. the improvement in the target metric that would make the solution profitable to implement, and then consider whether that improvement is realistic from a behavioral standpoint. From a psychological perspective, it’s better to start with the expected result than with the break-even point: if you start with the break-even point, you’re more likely to anchor on it and find reasons to justify that it’s achievable. However, in many cases, you’ll know the break-even point first, for example if it was calculated during a preliminary cost-benefit analysis; your company or business partners might also request it and refuse to think about the expected result first. Don’t worry too much about it. Regardless of whether you’re working with your expected or best-case result, we’ll need it to determine the minimum detectable effect for our experiment.

It’s important that your logic connects your proposed solution with your target metric. Don’t leave it to “this will improve the customer experience”. How will you know it? If your logic is solid, you should be able to express it in terms of a causal diagram, with at least some of the effects being observable.

A useful rule of thumb to help you articulate your logic is to break down your business metric into components. For example, revenue (or most of its variations) can be broken down into number of customers, probability/frequency of purchase, quantity purchased, and price paid. Determining which components are likely to be affected can allow you to better articulate the business case. If your business partners are concerned by a decrease in the number of customers and the proposed intervention would most likely only increase the quantity purchased, you need to clarify with them that they would still call it a win. This approach can also reduce the noise in your experiment; if the proposed intervention would most likely only increase the quantity purchased, you can focus on that metric and disregard somewhat random fluctuations in price paid.

Random assignment

The theory of random assignment could not be simpler: whenever a customer reaches the relevant page, they should be shown the current version of the page (called “control” in experimental jargon) with a 50% chance and the version with the 1-click booking button (the “treatment”) with a 50% chance.

From a coding perspective, assuming that you are not using a software that takes care of it for you, this can easily be implemented in R or Python:

• Whenever a new customer reaches the relevant page, we generate a random number between 0 and 1;

• Then we assign the customer a group based on that random number: if K is the number of groups we want (including a control group), then all individuals with a random number less than 1/K are assigned to the first group; all individuals with a random number between 1/K and 2/K are assigned to the second group, and so on.

Here, K is equal to two, which translates into a very simple formula:

## R
> K <- 2
> assgnt = runif(1,0,1)
> group = ifelse(assgnt <= 1/K, “control”, “treatment”)
## Python
K = 2
assgnt = np.random.uniform(0,1,1)
group = "control" if assgnt <= 1/K else "treatment"

There are, however, a certain number of subtleties which can trip novice experimenters.

The first one is determining the right point in time for random assignment. Let’s say that whenever a customer gets on the first page of the website, you assign them to either the control or the treatment group. Many of these customers will never reach the point of making a booking and will therefore not see your booking interface. This would dramatically reduce the effectiveness of your experiment because you would in effect experiment only on a fraction of your sample.

When determining which customers should be part of the experiment and when they should be assigned to an experimental group, you should reflect on how the treatment will be implemented if the experiment is a success. Your experimental design should include the same people who would see the treatment if it got implemented in business as usual and only them. Visitors who leave the website before booking will still not see the button, whereas any future promotion or change to the booking page would be built in addition so to speak “on top of” the button which would always be present. Therefore, non-booking visitors should be excluded but customers with a promotion should be included.

The second challenge is making sure that the random assignment is happening at the “right” behavioral level. I’ll explain what this means with an example. Let’s say that a visitor comes on the AirCnC website, starts a booking but then for whatever reason leaves the website (they got disconnected, it’s time for dinner, etc.) and comes back to it later. Should they see the same booking page? If they were offered 1-click booking the first time, should they still be offered it the second time?

The problem here is that there are really multiple levels that could potentially make sense. You could assign control or treatment at the level of a single website visit, of a booking, however many visits it takes, or at the level of a customer account (which may or may not be the same person if several people in a household use the same account). Unfortunately, there are no hard rules here, the right approach must be determined on a case-by-case basis by thinking about the conclusions you want to draw and what a permanent implementation would look like.

In many cases, it makes sense to do an assignment at the closest level you can to a human being: customer account if you can’t distinguish people in a household, or individual customer if they each have a sub-account, as is the case for example with Netflix. Human beings have persistent memories and alternating options for the same person can get confusing. Here, this would mean that our AirCnC customer should see the 1-click button for the whole duration of the experiment, regardless of how many visits and bookings they make during that time. Unfortunately, this means that you can’t just roll the dice metaphorically each time someone starts a booking on the website to determine their assignment; you need to keep track of whether they have been assigned in the past and if so to which group. For a website experiment, this can be done through cookies (assuming the customer allows them!).

Whatever level you choose, you’ll have to keep track of the assignment(s) to be able to link them to business outcomes later. This is why best-in-class companies use centralized systems that record all assignments and connect them to customer IDs in a database, so that they can serve a consistent experience to customers over time.

More broadly, what these two challenges point at is that the implementation of a business experiment is almost always a complex technical affair. A variety of vendors now offer somewhat plug-and-play solutions that hide the complexity under the hood, especially for website experimentation. Whether you rely on them or on your internal tech people, you’ll need to understand how they’re doing the random assignment to make sure that you’re getting the experiment you want.

A good way to check that the system works correctly is to start with an A/A test, where there is a random assignment but the two groups see the same version of the page. This will allow you to check that there are indeed the same number of people in the two groups and that they don’t differ in any significant way.

Sample size and experiment power

Once we know what we’re going to test and how, we need to determine our sample size. In some cases, as here with our 1-click booking experiment, we can choose our sample size: we can just decide how long we want to run the experiment. In other cases, our sample size might be defined for us. If we’re going to run a test across our whole customer or employee population, we can’t increase that population just for the sake of experimentation!

Regardless of the situation we’re in, we will look at our sample size in relation to other experimental variables, such as statistical significance, and not in a vacuum. Understanding how these variables are related is crucial to ensure that our experimental results are usable and that we’re drawing the right conclusions from them. Unfortunately, these are highly complex and nuanced statistical concepts and they have not been developed for business decisions.

In line with the spirit of this book, I’ll do my best to explain these statistical concepts and conventions in the context of business decisions. I’ll then share my reservations about the traditional conventions and offer my two cents regarding how to tweak them while remaining in the traditional framework. And finally, I’ll describe an alternative approach that has been slowly gaining momentum and which I think is superior, namely using computer simulations.

## R
> library(pwr)
> effect_size <- ES.h(0.194,0.184)
> pwr.2p.test(h = effect_size, n = NULL, sig.level = 0.05, power = 0.8, alternative = "greater")
     Difference of proportion power calculation for binomial distribution (arcsine transformation)
              h = 0.02554423
              n = 18950.14
      sig.level = 0.05
          power = 0.8
    alternative = greater
NOTE: same sample sizes

## Python
from math import asin, sqrt
import statsmodels.stats.power as ssp
effect_size = 2*asin(sqrt(0.194))-2*asin(sqrt(0.184))
ssp.tt_ind_solve_power(effect_size = effect_size, alpha = 0.05, nobs1 = None, alternative = 'larger', power=0.8)
Out[1]: 18950.818821558503

Power analysis without statistics: simulations

The logic of using simulations is very simple: we’re going to use some of our past historical data to “fake” an experiment and see what happens. By running a large number of such fake experiments, we can get a reasonable idea of what will happen in our real experiment under a variety of situations. What if the button has no effect whatsoever? What if it increases the booking rate by 1%? By 10%? What if we run the experiment with 10,000 customers? 40,000? Simulations will tell you!

Simulations at scale can be a little difficult to wrap your head around, so I’ll start with single runs that demonstrate how we can simulate false positives, and then false negatives.

Understanding simulations: false positives

First, we’ll tackle false positives, because they’re easier to understand. Let’s assume that our 1-click button has no impact on the booking rate. This means that on average our control group and our treatment group should have the same booking rate. The key phrase here is “on average”: if we take 2,000 customers and assign them at random to one of two groups, these two groups are unlikely to have exactly the same booking rate, because of random variations. Think of coins: if we toss two coins 10 times each, we expect that they’ll both have about 5 tails and 5 heads, but one might have 6 and 4, while the other has 3 and 7. Life is random! The question is how random it might get. It is possible that out of sheer luck all customers in one group will book a location and none in the other group does, but that’s extremely unlikely, so we can safely ignore that possibility. But what about a 10% difference? Could one group have a 55% booking rate and the other a 65% booking rate just out of luck? That’s where simulations come in.

We have a bit less than 3 years of historical data, and we know with certainty that the button could not affect them, because it was not in place. This is how we’ll proceed:

• Draw at random 2,000 observations from our data;

• For each one of them, change the OneClick variable to 1 with probability 50% (i.e. we randomly assign them to the treatment group);

• Run a regression and measure the coefficient and p-value for the OneClick variable.

## R
> n <- 2000
> prob_button <- runif(n)
> sim_data <- hist_data %>%
+   sample_n(K) %>%
+   mutate(oneclick = ifelse(prob_button > 0.5,1,0))
> sim_mod <- glm(booked ~ age + gender + period + oneclick,
+                family = binomial(link = "logit"), data = sim_data)
> summary(sim_mod)
Call:
glm(formula = booked ~ age + gender + period + oneclick, family = binomial(link = "logit"),
    data = sim_data)
Deviance Residuals:
    Min       1Q   Median       3Q      Max
-2.0298  -0.5282  -0.2785  -0.1116   3.0201
Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept) 10.29624    0.66458  15.493   <2e-16 ***
age         -0.31815    0.01834 -17.349   <2e-16 ***
gendermale   0.13519    0.14258   0.948   0.3430
period      -0.01731    0.00713  -2.428   0.0152 *
oneclick     0.27351    0.14280   1.915   0.0554 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
    Null deviance: 1810.8  on 1999  degrees of freedom
Residual deviance: 1276.6  on 1995  degrees of freedom
AIC: 1286.6
Number of Fisher Scoring iterations: 6

As you can see, the coefficient for the OneClick variable is not null, and actually pretty high compared to the coefficient for age (in absolute value). Our imaginary button has about the same effect as decreasing a customer’s age by 1 year! It’s also statistically significant at the 10% threshold and close to the 5% one, with a p-value of 0.055. That’s clearly a false positive, given that we flipped the OneClick variable at random without changing anything else.

The corresponding Python code is

## Python code (output not shown)
random.seed(123)
np.random.seed(123)
# Simulating the experiment with no true effect
n = 2000
prob_button = np.random.uniform(size=n)
sim_data_df = hist_data_df.sample(n).drop('oneclick', axis = 1)
sim_data_df['prob_button'] = np.random.uniform(size=n)
sim_data_df['oneclick'] = sim_data_df['prob_button'].apply(lambda x: 1 if x > 0.5 else 0)
# Running a logistic regression
import statsmodels.formula.api as smf
model = smf.logit('booked ~ age + gender + period + oneclick', data = sim_data_df)
model.fit().summary()

To determine how large (again, in absolute values) false positives can get, we can repeat the previous process, say 1,000 times and see what happens. The code is the same, except that we save our results in a list that we then convert into a data frame.

## R
> #Simulation loop
> Nloop <- 1000
> set.seed(1234)
> ptm <- proc.time()
> sim_list <- list()
> n <- 2000
> for(i in 1:Nloop){
+   sim_data <- pre_data %>%
+     sample_n(n) %>%
+     mutate(oneclick = ifelse(runif(n) > 0.5,1,0))
+   sim_mod <- glm(booked ~ age + gender + period + oneclick,
+                  family = binomial(link = "logit"), data = sim_data)
+   sim_list[[i]] <- data.frame(
+     coeff = sim_mod$coefficients['oneclick'],
+     pvalue = coef(summary(sim_mod))[5,4]
+   )
+   }
> sim_data_summary <- bind_rows(sim_list)
> proc.time() - ptm
   user  system elapsed
  15.52    0.89   16.48
> summary(sim_data_summary)
     coeff               pvalue
 Min.   :-0.412403   Min.   :0.00292
 1st Qu.:-0.100693   1st Qu.:0.24325
 Median :-0.008339   Median :0.47684
 Mean   :-0.004953   Mean   :0.48687
 3rd Qu.: 0.099778   3rd Qu.:0.73260
 Max.   : 0.390446   Max.   :0.99547
## Python version (output not shown)
nloops = 1000
sample_sizes = [2000] * nloops
#Serialized function with FOR loop
def false_positive_repeat_sim(sample_sizes):

    coeff_vector = []
    pvalue_vector = []

    for s in sample_sizes:
        sim_data_df = hist_data_df.sample(s)
        sim_data_df['prob_button'] = np.random.uniform(size=s)
        sim_data_df['oneclick'] = sim_data_df['prob_button'].apply(lambda x: 1 if x > 0.5 else 0)

        model = smf.logit('booked ~ age + gender + period + oneclick', data = sim_data_df)
        res = model.fit(disp=0)

        # Extracting the coefficients of interest from the summary
        coeff_vector.append(res.params['oneclick'])
        pvalue_vector.append(res.pvalues['oneclick'])

    sim_summary_df = pd.DataFrame({'coeff': coeff_vector,'pvalue': pvalue_vector})
    return sim_summary_df
sim_summary_df = false_positive_repeat_sim(sample_sizes)
sim_summary_df.describe()

The coefficient estimates go from strongly negative to strongly positive, and the p-values are distributed uniformly between 0 and 1. By definition of a p-value when there is no effect, about 10% of the p-values are less than or equal to 0.1. Let’s see how large these estimates are by filtering on p-values and taking the absolute values of the coefficients.

## R
> sim_data_summary %>%
   filter(pvalue <= 0.1) %>%
   mutate(coeff = abs(coeff)) %>%
   summarize(min_eff = min(coeff),
             max_eff = max(coeff),
             avg_eff = mean(coeff))
    min_eff   max_eff  avg_eff
1 0.2267798 0.4124027 0.290947
## Python (output not shown)
def f(c):
    return pd.Series([c.abs().min(), c.abs().mean(), c.abs().max()], index = ['min_eff', 'avg_eff', 'max_eff'])
sim_summary_df.loc[sim_summary_df.pvalue <=0.1].apply(f).drop('pvalue', axis=1)

There is a 10% chance that we’ll observe a coefficient that will be between 0.23 and 0.41, with an average of 0.29. Remember that these are coefficient in a logistic regression so they don’t have an immediate interpretation, but we can get a sense of their relative size by comparing with the coefficients for the other variables in the earlier regression: at 0.41, our button would have the biggest impact of all variables!

2,000 customers is not that many. What if we run the previous simulation with 10,000 of them, about a month worth of data? The code is identical so I will only provide the results: coefficients in the top 10% are now between 0.105 and 0.23, with an average of 0.13. By using a larger sample, we’ve reduced the random variation in our data and we’re now much less likely to see a large effect out of randomness. As you try out various sample sizes, you’ll get a sense of how much uncertainty you can expect in your real experiment.

Understanding simulations: false negatives

Handling false negatives (falsely concluding that there is no impact when there is one) is more complicated because it depends on the effect size we’re looking at, which we’ve set at 1%.

Dealing with logistic regression also adds some complexity because we don’t observe directly the booking probability, only whether someone booked or not. There are several ways to manage that, with various degrees of complexity and accuracy. The approach I’ll use is somewhat in the middle of the pack in both regards (the steps that are different from the statistical significance simulation are in italics):

• Calculate the probability of booking in our complete historical dataset;

• Draw at random 10,000 observations from our data;

• For each one of them, change the OneClick variable to 1 with probability 50% (i.e. we randomly assign them to the treatment group);

• For those in the treatment group, increase the probability of booking by 1%;

• For all observations, reassign the Booked variable based on the probability of booking (e.g. if the probability of booking is 0.5 for an observation, then assign it the value 1 with probability 50%, regardless of the true value)

• Run a regression and measure the coefficient and p-value for the OneClick variable.

• Repeat the process 1,000 times.

This approach preserves the observed existing statistical relationships between all of our variables while allowing us to increase the probability of booking in the treatment group. An alternative approach would be to switch 1% of the Booking values in the treatment group from 0 to 1. That second approach is simpler but less precise, so you should simulate a much larger number of “experiments” if you’re using it.

## R
> #Add predicted probability of booking to historical data
> hist_mod <- glm(booked ~ age + gender + period, family = binomial(link = "logit"), data = pre_data)
> pre_data <- pre_data %>%
   mutate(pred_prob_bkg = hist_mod$fitted.values)
>
> #Simulation loop
> Nloop <- 1000
> set.seed(1234)
> ptm <- proc.time()
> sim_list <- list()
> K <- 10000
>
> for(i in 1:Nloop){
   sim_data <- pre_data %>%
     sample_n(K) %>%
     select(-oneclick) %>%
     mutate(oneclick = ifelse(runif(K) > 0.5,1,0)) %>%
     mutate(pred_prob_bkg = ifelse(oneclick == 1, pred_prob_bkg + 0.01, pred_prob_bkg)) %>%
     mutate(booked = ifelse(pred_prob_bkg >= runif(K,0,1),1, 0))

   sim_mod <- glm(booked ~ age + gender + period + oneclick,
                  family = binomial(link = "logit"), data = sim_data)
   sim_list[[i]] <- data.frame(
     coeff = sim_mod$coefficients['oneclick'],
     pvalue = coef(summary(sim_mod))[5,4]
   )
 }
> sim_data_summary <- bind_rows(sim_list)
> proc.time() - ptm
   user  system elapsed
  48.61    0.18   48.93
> summary(sim_data_summary)
     coeff              pvalue
 Min.   :-0.15894   Min.   :0.0000066
 1st Qu.: 0.05646   1st Qu.:0.0333601
 Median : 0.09374   Median :0.1297420
 Mean   : 0.09445   Mean   :0.2345229
 3rd Qu.: 0.13140   3rd Qu.:0.3491058
 Max.   : 0.28237   Max.   :0.9974436
> sim_data_summary %>%
   summarize(power = sum(pvalue <= 0.1 & coeff > 0)/n())
  power
1 0.454

Now that we’ve simulated a “true” effect of 1%, our p-values are not uniformly distributed anymore. You can see from the data summary that the median value is a bit less than 0.13, as opposed to close to 0.5 previously. Looking at the third quartile value, we can see that 75% of p-values are less than 0.349.

What is the probability that we’ll rightly conclude that there is a positive and statistically significant effect, assuming we use the 10% threshold? About 45.4%. In other words, our simulated estimate of the statistical power of our experiment for an effect size of 1% and a sample size of 10,000 is a bit more than 45%.

The Python code is similar:

## Python
Nloop = 1000
n = 10000
coeff_vector = []
pvalue_vector = []
random.seed(123)
np.random.seed(123)
for i in range(Nloop):
    #Assigning button to 50% of sample
    sim_data_df = hist_data_df.sample(n)
    sim_data_df['prob_button'] = np.random.uniform(size=n)
    sim_data_df['oneclick'] = sim_data_df['prob_button'].apply(lambda x: 1 if x > 0.5 else 0)
    #Updating probability of booking for people with button
    sim_data_df['pred_prob_bkg'] = np.where(sim_data_df['oneclick'] == 1, sim_data_df['pred_prob_bkg'] + 0.01, sim_data_df['pred_prob_bkg'])
    #Generating binary value for booked based on new probability
    sim_data_df['prob_booked'] = np.random.uniform(size=n)
    sim_data_df['booked'] = np.where(sim_data_df['pred_prob_bkg'] >= sim_data_df['prob_booked'], 1, 0)
    model = smf.logit('booked ~ age + gender + period + oneclick', data = sim_data_df)
    res = model.fit()
    # Extracting the coefficients of interest from the summary
    coeff_vector.append(res.params['oneclick'])
    pvalue_vector.append(res.pvalues['oneclick'])
sim_summary_df = pd.DataFrame({'coeff': coeff_vector,
                                   'pvalue': pvalue_vector})
sim_summary_df.describe()
#Getting the overall empirical power of our analysis
len(sim_summary_df[(sim_summary_df['pvalue'] <= 0.1) & (sim_summary_df['coeff'] > 0)].index) / len(sim_summary_df.index)

Using simulations to determine sample size

In the previous two sections, we looked at two questions:

• For a given sample size, assuming that there’s no true effect, how large of an effect can we observe out of sheer randomness at the 10% statistical significance level?

• For a given sample size, assuming that there is a true effect of 1%, how likely are we to observe an effect at the 10% statistical significance level?

However, in the present case, what we want to determine is the sample size. Therefore, what we’ll do is run a version of the previous simulations for various sample sizes, and then check for each possible sample size the answer to the two questions above. This will allow us to check the sample size provided by the statistical formula against “real life” conditions.

The full code for this simulation is on the book’s Github, but the intuition remains the same as before: for each possible sample size, we run a large number of “experiments” of that size and then look at the distribution of the variable of interest.

Let’s look at a few summary graphs that provide insights into the questions we have (figure 8-2). If we apply tests of proportion to our simulated datasets, how large of a sample size do we need to observe a statistically significant effect 90% of the time when there is one? What about applying a logistic regression?

Figure 2-2. Empirical power for a test of proportions (left) and a logistic regression (right).

The empirical power for the test of proportions seems in line with the statistical formula. Remember that our required sample size for a power of 0.9 was 2 x 18,966 = 37,932. Our logistic regression shows a bit more oomph, with a power of 90% at 35,000, or three months and a half.

With that established, we can now run the experiment and analyze its results. The actual “running the experiment” part is very technology-driven and each situation is unique, so I won’t get into details regarding that. Just remember to rely on your experimental design: the metrics you’re tracking, the criteria and logic for success, the process for random assignment.

Test of proportions

In R, you need to provide a 2 x 2 table representing your two variables of interest for your experimental data.

## R
> tab <- with(exp_data, table(oneclick, booked))
> prop.test(tab, alternative = "greater", conf.level = 0.9)
	2-sample test for equality of proportions with continuity correction
data:  tab
X-squared = 1.8176, df = 1, p-value = 0.0888
alternative hypothesis: greater
90 percent confidence interval:
 0.0001641621 1.0000000000
sample estimates:
   prop 1    prop 2
0.9364953 0.9331240
## Python
import statsmodels.stats.proportion as sspr
exp_data_df.groupby('oneclick').agg(booked_percentage = ('booked',np.mean))
Out[2]:
          booked_percentage
oneclick
0                  0.063505
1                  0.066876
tab = exp_data_df.groupby('oneclick').agg(count_booked = ('booked',sum), count_total = ('age','count'))
sspr.proportions_ztest(count=tab.count_booked, nobs=tab.count_total, alternative='larger')
Out[3]: (-1.3683887677631428, 0.9144047908879882)

We’re only interested in determining whether the 1-click button increases the booking rate or not, so we can use a one-tailed test by setting the “alternative” parameter to “greater” in R (“larger” in Python). The confidence level is the one you have chosen earlier.

The probability of booking in the control group is 6.35 percentage points (pp) and the probability of booking in the treatment group is 6.69pp, with a difference of about 0.34pp. In other words, the measured effect size for the 1-click button is an increase of 0.34pp in absolute terms in booking probability, or around 5.3% in relative terms (0.34/6.35)⁶.

The p-value for this result is about 8.9%, weakly significant. According to conventional wisdom, this makes the result of the experiment statistically not significant because it’s less than 5%. People will often use this threshold as a strict cut-off: if the p-value is less than 5% and the difference is positive, the treatment is deemed a success and gets implemented; if the p-value is more than 5%, the effect is treated as if it was zero and the treatment does not get implemented. This interpretation is mistaken. In the absence of any other information, your best estimate of the difference is your measured effect size, which here is 0.34pp, regardless of whether the p-value is 0.00001%, 5% or even 90%. The short version of the story is that even under the most generous interpretation, your p-value tells you how confident you can be in your result, but it doesn’t tell you anything about your result itself. This is apparent here in the confidence interval for the difference, which goes from 0.016pp to 1.00. In other words, you can be confident that the 1-click button has a positive effect, and values near 0.34pp are more likely than values far away from it, the closer the likelier; but you should not be confident that the true effect is truly 0.34pp, as it could really be anything between 0 and 1. Does it matter? Well, that depends on your situation. If there is little cost and risk in implementing your treatment, you should go ahead and do that; if you want to be more confident, for example if you only want to implement the button with an effect size above 4%, then you should run another experiment to increase your confidence.

Ultimately, statistical tests do not make full use of the data available, which is why I prefer regression instead.

Logistic regression

Running a logistic regression instead of using a test of proportion is straightforward and offers multiple advantages. Because of the random assignment, we know that our coefficient for the 1-click button is unconfounded—we don’t need to control for any confounder. However, by adding other variables that are also causes of the booking probability, we can reduce noise and significantly improve the precision of our estimation.

## R
> log_mod_exp <- glm(booked ~ oneclick + age + gender,
+                    data = exp_data, family = binomial(link = "logit"))
> summary(log_mod_exp)
Call:
glm(formula = booked ~ oneclick + age + gender, family = binomial(link = "logit"),
    data = exp_data)
Deviance Residuals:
    Min       1Q   Median       3Q      Max
-2.7191  -0.3076  -0.1527  -0.0696   3.6957
Coefficients:
            Estimate Std. Error z value    Pr(>|z|)
(Intercept) 11.69282    0.22565  51.819     < 2e-16 ***
oneclick     0.15784    0.04702   3.357    0.000789 ***
age         -0.39406    0.00643 -61.282     < 2e-16 ***
gendermale   0.25420    0.04905   5.182 0.000000219 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
    Null deviance: 19358  on 40159  degrees of freedom
Residual deviance: 12949  on 40156  degrees of freedom
AIC: 12957
Number of Fisher Scoring iterations: 7
##Python code (output not shown)
import statsmodels.formula.api as smf
model = smf.logit('booked ~ age + gender + oneclick', data = exp_data_df)
res = model.fit()
res.summary()

The coefficients from a logistic regression are not directly interpretable, and I find that the recommended solution of using the odds ratio only helps marginally. My preferred rule of thumb is to generate two copies of the experimental data, one where the variable for 1-click button is set to 1 for everyone, and the other where it is set to 0. By comparing the probability of booking predicted by our logistic model for these two datasets, we can calculate an “average” effect that is very close to the effect you would observe if implementing the treatment for everyone. It’s unscientific but useful.

## R
> ### Calculating average difference in probabilities
> no_button <- exp_data %>%
   mutate(oneclick = 0) %>%
   #mutate(oneclick = factor(oneclick)) %>%
   select(age, gender, oneclick)
> button <- exp_data %>%
   mutate(oneclick = 1) %>%
   #mutate(oneclick = factor(oneclick)) %>%
   select(age, gender, oneclick)
> #Adding the predictions of the model
> no_button <- no_button %>%
   mutate(pred_mod = predict(object=log_mod_exp, newdata = no_button, type="response"))
> button <- button %>%
   mutate(pred_mod = predict(object=log_mod_exp, newdata = button, type="response"))
> #Calculating average difference in probabilities
> diff <- button$pred_mod - no_button$pred_mod
> mean(diff)
[1] 0.007129714
## Python (output not shown)
#Creating new copies of data
no_button_df = exp_data_df[['age', 'gender']]
no_button_df['oneclick'] = 0
button_df = exp_data_df[['age', 'gender']]
button_df['oneclick'] = 1
#Adding the predictions of the model
no_button_df['pred_bkg_rate'] = res.predict(no_button_df)
button_df['pred_bkg_rate'] = res.predict(button_df)
#Calculating average difference in probabilities
diff = button_df['pred_bkg_rate'] - no_button_df['pred_bkg_rate']
diff.mean()

The syntax of the code above is as follows:

• We create a dataset called no_button for which we set the variable oneclick to zero for all rows (and convert it to factor so that the predict function later will work);

• We create a dataset called button for which we set the variable oneclick to one for all rows;

• We calculate the predicted probability of booking in each case by using the predict function with our model log_mod_exp;

• We calculate the difference between the predicted probabilities.

We can see that after our average effect across our experimental population is about 0.7pp. This is not very far from the observed value of 0.34pp, but higher. Most likely, there is a slight difference between our control and treatment group that explains the remaining 0.36pp.

In the present case, the p-value for the 1-click coefficient is very low, so we can safely conclude that the result is highly statistically significant.

However, even if that was not the case, the same logic would apply here as for the T-test above: our best guess for the size of the effect is the estimated coefficient, which here translated into a 0.7pp increase. You would need to ask yourself and your business partners if this increase is economically significant. In most lines of business, a 0.7pp increase in a rate is nothing to scoff at. You would then have two options:

Run a second experiment to get more precision in your estimate.

Implement the 1-click button regardless of the poor statistical significance, if the costs or risks of doing so are low.

To cover all cases, let’s recap our decision rule (table 8-3). This way, you’ll see what to do depending on whether the observed estimated effect is statistically significant or not and economically significant (taken here to mean a 1pp increase) or not.

Table 8-3. Decision rule for 1-click booking button.