This scenario is one that can occur in both isolated automation teams and integrated teams where everybody works together. You have probably seen automated tests that are not totally reliable: you know, that one flickering test that occasionally fails for no obvious reason. Somebody once had a look at it and said that there was no reason for it to fail so it got ignored, and now whenever it fails again somebody says "Oh, it's that flickering test again—don't worry about it. It will be green again soon."

The thing is that we now have a problem: tests do not flicker for no reason. This test is desperately trying to tell you something and you are ignoring it. What is it trying to tell you? Well, you can't be sure until you have found out why it is flickering; it could be one of many things. Among the many possibilities, a few are the following:

• The test is not actually checking what you think it is checking
• The test may be badly written
• There may be an intermittent fault in the application that is under test (for example, there may be a race condition nobody has identified yet)
• Maybe you have some problems with a date/time implementation and it's only going red at specific times when the date implementation goes wrong (date/time implementations are something that are notoriously hard to get right and the cause of many bugs in many systems)
• There are network problems; is there a proxy getting in the way?

The point is that, while your test is flickering, we don't know what the problem is; but don't fool yourself—there is a problem. It's a problem that will, at some point, come back and bite you if you don't fix it.

Let's imagine for a moment that the software you are testing is something that buys and sells shares, and you are pushing new releases out daily because your company has to stay ahead of the game. You have a test that has been flickering for as long as you can remember. Somebody once had a look at it, said they couldn't find any problems with the code, and said that the test is just unreliable; this has been accepted and now everybody just does a quick manual check if it goes red. A new cut of code goes in and that test that keeps flickering goes red again. You are used to that test flickering and everything seems to work normally when you perform a quick manual test, so you ignore it. The release goes ahead, but there is a problem: suddenly your trading software starts selling when it should be buying, and buying when it should be selling. It isn't picked up instantly because the software has been through testing and must be good, so no problems are expected. An hour later all hell has broken loose; the software has sold all the wrong stock and bought a load of rubbish. In the space of an hour, the company has lost half its value and there is nothing that can be done to rectify the situation. There is an investigation and it's found that the flickering test wasn't actually flickering this time; it failed for a good reason, one that wasn't instantly obvious when performing a quick manual check. All eyes turn to you; it was you who validated the code that should never have been released and they need somebody to blame; if only that stupid test hadn't been flickering for as long as you can remember.

The preceding scenario is an extreme example, but hopefully you get the point: flickering tests are dangerous and something that should not be tolerated.

We ideally want to be in a state where every test failure means that there is an undocumented change to the system. What do we do about undocumented changes? Well, that depends; if we didn't mean to make the change, we revert it. If we did mean to make the change, we update the documentation (our automated tests) to support it.