Other disciplines take more of a walk-before-you-run approach
to engineering and the validation of production readiness. For example, airplane aerodynamics are vetted in a wind tunnel before the first test flight. Before the full-sized bridge is built, a scale model is fabricated to assess load-bearing capacity.
Yet even today, we continue to write software that is unprepared to handle a production load, as if onesie-twosie functional testing
nearly approximated a production workload. It doesn’t. Our deliverables include the part about handling a production load.
To address this gap, you need to first understand a common misperception, one that I encourage you to discuss with your friends and coworkers. Don’t be shy about it.
Here is the misperception: Getting performance right is nearly impossible, because you must have a massive computing environment and expensive monitoring to solve problems that only performance experts can fix. We also think of many environment as having little monitoring—the Dark Environments. That Dark Environments exist is antother misperception, because as this book has shown, there are lots of plug-it-in-now tools available in every JDK in every Dark Environment.
But really, like the code samples in this book, the majority of performance defects can be reproduced and detected in just about any environment using open source or freely available monitoring tools. It just takes a little guidance from a methodical approach (like the P.A.t.h. Checklist) to help you locate the most common and pernicious defects.
Ultimately, before going into production, system performance must be vetted in a full-sized environment under the full production workload. But attempting to tune everything at once in a large environment normally results in a half-baked performance rush-job. It is an expensive and horrifically slow endeavor, because of the lack of access and high maintenance required, like code redeployments that take half a day or longer.
Contrast that to the easy access of a small environment where you can get perhaps ten fix-test tuning cycles in a single day. Finally, the early feedback you can get in a small environment helps avoid costly retreats from approaches with unacceptable performance.
Steps for Getting Started
Getting started now to tune your system is pretty straightforward. Follow these steps:
- 1.Create a Modest Tuning Environment (Chapter 2) using existing hardware. A massive amount of fast, expensive performance hardware is not required.
- 2.Let’s fix the problem of undersized databases in development. Whether you rent it or buy it, make a modest investment in extra hard disks for the various Modest Tuning Environments in development. The cost of ten single terabyte drives is small. The lack of a production-sized dataset is an obstacle to locating performance defects. It often takes a lot of data from a large disk to exacerbate GC and other problems. You’ll also need to give the team 40–80 hours to work on a repeatable solution to generate a production-sized database. This is a modest ask for a huge benefit.
- 3.Ensure good performance visibility in every single environment. The plug-it-in-now tools demonstrated in this book go a long way here. Sure, SQL performance metrics are still required, but you can easily use the open source Glowroot or similar for that.
- 4.Apply load to your code on a regular basis to identify obvious issues. Remember, the “under load” thing is part of our deliverable. Most of the examples in this book were able to demonstrate performance issues with just three threads of load and zero think time. I call this “3t0tt.” Chapters 4, 5, 6 and 7 focus on load generation.
- 5.Use the P.A.t.h. Checklist as a reminder of where to look to find the most common defects. See Chapters 9, 10, 11 and 12. Chapter 1 on the Main Performance Anti-Patterns will also help. Whether you are fighting performance fires in production or you have other development priorities, there is only so much time to troubleshoot performance. To minimize troubleshooting time, you can easily make one pass at P.A.t.h. Checklist in 30 minutes or less.
Using this Book with Large Monitoring Tools
For all of my talk about small, free monitoring tools, I have nothing against sophisticated monitoring tools— not even commercial ones. Java profilers, APM tools, network trace tools, and so on. They’re a mainstay in my day job as a performance engineer; I occasionally use one tool that has a $75k license. New, open source APM tools like Glowroot are maturing and there are a few good commercial ones, too. I’m OK with large tools that work.
But these tools are absent from perhaps half or more of all environments, and this happens because of high license cost, fear of high overhead, and other reasons.
I hope the plug-it-in-now, small-tool tuning techniques presented in this book can shed enough light on these issues in Dark Environments to keep the performance technical debt at bay.
But the load generation techniques, the Main Performance Anti-Patterns
, the P.A.t.h. Checklist and other tuning approaches in this book can provide value with any monitoring toolset in any environment, Dark or not.
For example, if stack traces in a Java Profiler or APM tool indicate during a load test that some slowdown is caused by a rarely executed business process, you’ll know to remove that process from the load test, because it’s not a challenge that will be encountered (at any significant load) in production. That’s Main Anti-Pattern 3, Overprocessing.
Here is a second example. Quality APM tools
are capable of serving up the vast sea of metrics that I outlined in Chapter 3 where I talked about the three reasons for metrics (checking whether we’ve met our performance requirements, assessing whether our resources are all used up, and identifying the component responsible for a slowdown). This vast sea of metrics can be overwhelming, knowing which ones to focus on. Well that’s easy. As a starting point, make sure you have some focus on each of the items in the P.A.t.h. Checklist. To get a little more specific, the P.A.t.h. Checklist should guide decisions on which metrics to place on APM “dashboards.” A dashboard is a user-arranged screenful of performance graphs/metrics.
The plug-it-in-now tool mentality is certainly lacking one big thing: a repository. When you are there troubleshooting a system, plug-it-in-now is a great approach. But systems go bump in the night, and its great to have an APM tool with a repository that captures great data while your not there. This lack of a repository is perhaps the Achilles heel of the plug-it-in-now toolset. The said, the plug-it-in-now is always there as a last resort for Dark Environments.
3t0tt (3 Threads of Load, Zero Think Time )
I think we are very close to the point where all code should get at least a light performance vetting before being checked in. Here is one way to accomplish this: stipulate to your developers that:
- All changes to components as big as or bigger than a SOA service should be examined under the stress of exactly three threads of load with zero think time.
When I say, “bigger than a SOA service”, I am trying to avoid the complexities of micro-benchmarking. Towards this end, 3t0tt should only include components that have at least one I/O—like one or more database hits, or one or more other network requests. It also makes sense to exclude “really fast” code from 3t0tt testing, again to avoid the complexities of micro-benchmarking. Perhaps lay out ‘required 3t0tt’, ‘optional 3t0tt’ and ‘never 3t0tt’ response time landmarks. Choose your own thresholds, but I would choose numbers like never do 3t0tt testing for 25ms and faster; 25ms to 100ms would be optional, and 100ms and higher would be required.
Why three threads of load and not 42 or 95 threads of load? The idea is that we need enough activity to reproduce issues, including multi-threaded ones, but not so much activity that it risks overwhelming modest-sized tuning environments, the most productive place to tune code.
In my experience, three threads are sufficient to flush out the vast majority of all multi-threaded issues. Applying this amount of load to a multi-threaded or re-entrant bug, you will see between 00.001% and 10.000% of all requests end in errors errors/exceptions. Ignore this “errors” metric at your own risk. The JMeter Summary Report and JMeter Plugins Synthesis Report (a better “Summary Report”) label this simply as “errors.” Also, if your code makes a regular practice of catching and ignoring exceptions, you are short-circuiting a wonderful feedback process that will
help you find fixable bugs using this 3t0tt technique.
One Last Word
Have you ever heard coworkers say these things?
- It would cost too much to create a performance environment that matches the size of production.
- The network people won’t let us load test on that network.
- Is the firewall causing performance issues?
- It will take one day to get that code redeployed.
- It will take two days to get an exception for that firewall rule.
Actually, you’d never hear that last thing because you could never get a firewall changed that quickly. But outside of that, this is just a sampling of all the daily impediments to load testing. These are all legitimate issues, but they don’t excuse us from asking developers to be accountable for their own deliverables, which should include acceptable performance under a production workload
.