Extending Your Life Span

The major dangers to your system’s longevity are memory leaks and data growth. Both kinds of sludge will kill your system in production. Both are rarely caught during testing.

Testing makes problems visible so you can fix them. Following Murphy’s Law, whatever you do not test against will happen. Therefore, if you do not test for crashes right after midnight or out-of-memory errors in the application’s forty-ninth hour of uptime, those crashes will happen. If you do not test for memory leaks that show up only after seven days, you will have memory leaks after seven days.

The trouble is that applications never run long enough in the development environment to reveal their longevity bugs. How long do you usually keep an application server running in your development environment? I’ll bet the average life span is less than the length of a sitcom on Netflix. In QA, it might run a little longer but probably still gets recycled at least daily, if not more often. Even when it is up and running, it’s not under continuous load. These environments are not conducive to long-running tests, such as leaving the server running for a month under daily traffic.

These sorts of bugs usually aren’t caught by load testing either. A load test runs for a specified period of time and then quits. Load-testing vendors charge large dollars per hour, so nobody asks them to keep the load running for a week at a time. Your development team probably shares the corporate network, so you can’t disrupt such vital corporate activities as email and web browsing for days at a time.

So how do you find these kinds of bugs? The only way you can catch them before they bite you in production is to run your own longevity tests. If you can, set aside a developer machine. Have it run JMeter, Marathon, or some other load-testing tool. Don’t hit the system hard; just keep driving requests all the time. (Also, be sure to have the scripts slack for a few hours a day to simulate the slow period during the middle of the night. That will catch connection pool and firewall timeouts.)

Sometimes the economics don’t justify setting up a complete environment. If not, at least try to test important parts while stubbing out the rest. It’s still better than nothing.

If all else fails, production becomes your longevity testing environment by default. You’ll definitely find the bugs there, but it’s not a recipe for a happy lifestyle.