Another thing to watch out for is attributing conversions to a change downstream. If the action you're trying to drive doesn't happen immediately upon the user experiencing the thing that you're testing, things get a little bit dodgy.

Let's say I change the color of a button on page A, the user then goes to page B and does something else, and ultimately buys something from page C.

Well, who gets credit for that purchase? Is it page A, or page B, or something in-between? Do I discount the credit for that conversion depending on how many clicks that person took to get to the conversion action? Do I just discard any conversion action that doesn't happen immediately after seeing that change? These are complicated things and it's very easy to produce misleading results by fudging how you account for these different distances between the conversion and the change that you're measuring.