The first thing you need to figure out when you're designing an experiment on a website is what are you trying to optimize for? What is it that you really want to drive with this change? And this isn't always a very obvious thing. Maybe it's the amount that people spend, the amount of revenue. Well, we talked about the problems with variance in using amount spent, but if you have enough data, you can still, reach convergence on that metric a lot of times.

However, maybe that's not what you actually want to optimize for. Maybe you're actually selling some items at a loss intentionally just to capture market share. There's more complexity that goes into your pricing strategy than just top-line revenue.

Maybe what you really want to measure is profit, and that can be a very tricky thing to measure, because a lot of things cut into how much money a given product might make and those things might not always be obvious. And again, if you have loss leaders, this experiment will discount the effect that those are supposed to have. Maybe you just care about driving ad clicks on your website, or order quantities to reduce variance, maybe people are okay with that.

You can measure more than one thing at once too, you don't have to pick one, you can actually report on the effect of many different things:

• Revenue
• Profit
• Clicks
• Ad views

If these things are all moving in the right direction together, that's a very strong sign that this change had a positive impact in more ways than one. So, why limit yourself to one metric? Just make sure you know which one matters the most in what's going to be your criteria for success of this experiment ahead of time.