Performance Assessment Workflow

Good performance analysis is about taking mountains of facts and focusing on the most important grains of truth. In the previous sections we established what to measure so you’ll know how your system behaves through different traffic patterns. There’s plenty of data to gather, should you be so inclined. In fact, there would be too much information to decipher if you decided to measure everything we’ve outlined. Remember, the name of the game is focus. It’s time to find out which processes to target for more detailed instrumentation.

Regardless of whether you are optimizing a web server, an embedded system or the tools used by your team, the journey is quite similar. Before you begin, load test a feature and compare that to your performance requirements. If it is behaving as desired, you can gladly move forward. If it is not, you will want to profile the system and identify the hotspots. Then, you can use benchmarks to compare different solutions and remove the bottleneck.

That’s the flow we will explore here, using the Phoenix application we created in the previous section as a starting point. Before we move on to load testing with specific tools, consider the following suggestions.

First, never measure only averages. Besides the average, you also want to measure at least one of the 90th, 95th, and 99th percentiles. When talking about web applications, most page loads experience 99% latencies^[114] so it is essential to include those percentiles in your measurements.

Second, quantify your gains. Since determining how fast a system or a feature should be is sometimes hard, you should try to convert that time into monetary gains, even if it is back-of-the-napkin calculus. If you can reduce the page load by 1 second, many companies report gains from 2% to 10% in conversion rates, and you can do assumptions based on existing case studies. Similarly, if you are in a team of five engineers with a slow test suite that you run on average 20 times per day and you believe you can cut the test time from 20 seconds to 10 seconds, that will save your team 8 hours every month altogether. If the speedup takes a day, you will recoup that back by the end of the month and your team will feel more confident.

Finally, avoid performance regressions. Every time you find a hotspot, you should consider feeding this information back to your metrics system. If a process is a bottleneck, track its message queue length. If a function turns out to be computationally expensive, instrument it. You should also consider setting up performance tests that you run in your CI server. That will give you confidence the server performance won’t regress as you add new features.

Now that you have some basic guidelines to work from, it’s time to collect some real data through load testing.