21   Different Makes a Difference

“As long as the code works on your machine, that’s OK. Who cares if it works on some other platform? You don’t have one.”

When a vendor or a co-worker says those immortal words, “Oh, that won’t make a difference,” you can bet they are wrong. If something is different, odds are it will make a difference.

Venkat learned this lesson the hard way on a project. A colleague complained that Venkat’s code was failing. But oddly, the situation was the same as one of the test cases that passed on Venkat’s machine. It worked on one machine but not on the other.

They finally figured out the culprit was a difference in behavior of a .NET API on different platforms: Windows XP versus Windows 2003.^[21] The platform was different—and it made a difference.

They were lucky to discover this problem by accident; otherwise, it may have been noticed only once the product shipped. Discovering such problems late can be very expensive—imagine releasing an application and only then finding out it breaks on one of the platforms you’re supposed to be supporting.

Andy says:

But It Works on My Machine…

I once had a client who needed better performance from their OS/2 system, so one of the bolder developers decided to rewrite the OS/2 operating system scheduler from scratch, in assembly language.

And it actually worked. Sort of. It worked really well on the original developer’s machine, but they couldn’t get it to work on any other machine. They even went so far as to purchase identical hardware from the same vendor and load up the same version of operating system, database, and other tools. No luck at all.

They tried facing the machines in the same direction, at the same time of day, with a sacrificial chicken for luck (OK, I’m making that part up, but the rest is true).

The team eventually abandoned the attempt. Messing with the undocumented internals of the operating system is most likely a fragile technique, not an agile one.

You might ask your QA team to test your application on all supported platforms. But that may not be the most reliable tack if they’re testing manually. We need a more developer-oriented approach!

You’re already writing unit tests to exercise your code. Whenever you modify or refactor your code, you exercise your test cases before you check in the code. All you have to do now is exercise your test cases on each supported platform or environment.

If your application is expected to run on different operating systems (MacOS, Linux, Windows, etc.) or even on different versions of the same operating system (Windows 2000, Windows XP, Windows 2003, etc.), you need to test on all of them. If you expect your application to work on different versions of the Java Virtual Machine (VM) or .NET common language runtime (CLR), you need to test that as well.

But you probably already feel pressed for time, so how can you possibly take time to run tests on multiple platforms as well? Continuous integration^[22] to the rescue!

As we saw in ​Keep It Releasable​ a continuous integration tool periodically fetches the code from your source control system and exercises it. If any test fails, it notifies the relevant developers. The notification may be through email, a pager, an RSS feed, or other more creative approaches.

To test on multiple platforms, you simply set up a continuous integration system on each. When you or a fellow developer checks in the code, the tests will be run automatically on each of these platforms. Imagine being informed of any failures on any platform within minutes of checking in your code! That’s using your resources wisely.

Hardware for build machines costs the equivalent of only a few hours of developer time. If needed, you can reduce the hardware cost even further by using products such as VMware or Virtual PC to run different versions of operating system, VM, or CLR on a single box.

Different makes a difference. Run unit tests on each supported platform and environment combination, using continuous integration tools. Actively find problems before they find you.

What It Feels Like

It feels like unit testing, only more so—it’s unit testing across multiple worlds.

Keeping Your Balance

• Hardware is cheaper than developer time. But if you have a large number of supported platforms and configurations, you may need to be selective as to which ones you actively test in-house.

• Bugs that exist on all platforms may be spotted only because of stack layout differences, word-endian differences, etc., so even if you have fewer clients on Solaris than Linux, you still want to test on both.

• You don’t want to be bombarded with five notifications for one error (that’s like double taxation and contributes to “email fatigue”). Either lower the frequency of the integration build on all but one main platform/configuration to give you enough time to fix the main build if it breaks or roll up the errors in a single convenient report.