Most interesting software products are not built by a single person. Chances are, you are part of a team trying to build some useful and well-performing software. If you are the performance guru on your team, then there are things you can do to encourage others to be so-minded as well.

Much of the advice in this chapter presupposes an organization that understands software engineering as a true engineering effort. Unfortunately, many people find themselves in less than ideal circumstances. If this is you, do not despair. Perhaps some of the advice in this chapter can help you improve the level of engineering appreciation and competence in your company.

1. Understand the Areas of Critical Performance

You cannot optimize everything, almost by definition. This goes back to the first principles we discussed in Chapter 1 about measurement and finding the areas of critical performance. As a team, you will need to build consensus on which areas are critical and which can be left alone.

As engineers, we all should have pride in our work and make it the best it can be, and in no area of code should we completely slack off. However, the realities of business dictate limited time and human resources in which to accomplish the necessary work. Given these realities, you should take time to understand where the critical areas of the system are (Remember Measure, Measure, Measure) and make sure those receive a larger portion of careful attention to detail.

Performance is not the only metric by which you need to judge code. Maintainability, scalability, security, and other important factors must also guide your decision-making. However, of the items in that list, I suspect that performance measurement and tuning will take more of your time on a continual basis.

1. Effective Testing

This is not a testing book, but it should go without saying that having effective tests at all levels will greatly increase your confidence in making significant changes to code. If you have unit tests with high code coverage, drastically changing a central algorithm or data structure to be much more efficient should not fill you with dread.

More to the point of this book, if performance is critical to you, then you should be tracking it with the tools and techniques mentioned in this book. Just like you have functional tests, you can have performance tests. They may be as simple as tracking how many operations a component can do in a second, to as complicated as the performance across thousands of metrics between your pre-release server farm and the production server farm.

Performance test failures should be treated as seriously as functional test failures and should be ship blockers. You will likely find that building reliable, repeatable performance tests is far more difficult than functional tests. So much of performance is intertwined with the state of machines, other software running on the machine, history of the running process, and infinitely more variables. There are two basic approaches to handling this noise that will creep in:

1. Remove noise—Have clean machines, restart them before testing, control all the processes running, control for hardware differences, and more. This approach is required if you are testing on single machines.
2. Run tests at a large scale—If it is not practical to eliminate all the noise then ignore it and run your tests on a scale that is sufficient to eliminate the significant sources of noise. This can be quite expensive, especially for larger infrastructures. You may need dozens, hundreds, even thousands of machines to get a truly statistically significant result. If you cannot scale out the hardware, you can scale out in time and rerun tests hundreds of times, but this does not account for as many variables.

Either way, you will need to engage in A/B testing, that is, comparing the performance of one build against another in as ideal a scenario as you can manage.

1. Performance Infrastructure and Automation

You will probably need to build some custom infrastructure, tools, and automation support to gather performance data for you, but all the tools to read the metrics are described in this book. Thankfully, nearly all useful performance tools are scriptable in some way.

There are many ways to track performance and you will have to decide the best way for your product. Some ideas include:

• PerfMon—If all of your data is represented by performance counters and runs on a single machine, this may be sufficient.
• Performance counter aggregation—If you run on multiple machines, you probably need to aggregate the counter information into a centralized database. This has the advantage of storing performance data for historical analysis.
• Benchmarks—Your application processes a standard set of data and the resulting performance metrics are compared with historical results. Benchmarks are useful, but you must be careful with their historical validity as scenarios change. Benchmarks need occasional tuning and this can lead to invalid comparison between results.
• Automated profiling—Perform random profiling of CPU and memory allocation on either test or real data.
• Alerts that fire on performance data—For example, send off an automated alert to a support team if CPU gets too high for too long, or the number of tasks queued is increasing.
• Automated analysis of ETW events—This can get you some very nitty-gritty detail that performance counters will miss.

The things you do now to build a performance infrastructure will pay huge dividends in the future as performance maintenance becomes mostly automated. Building this infrastructure is usually far more important than fixing any arbitrary actual performance problem because a good infrastructure will be able to find and surface performance problems much earlier than any manually driven process. A good infrastructure will prevent you from being surprised by bad performance at an inconvenient time.

The most important part of your infrastructure is how much human involvement is necessary. If you are like most software engineers, there is far more work than time to do it in. Relying on manual performance analysis means it will often not get done. Thus, automation is the key to an effective performance strategy. An investment upfront will save countless hours day after day. A good time-saving device can be as simple as a script that runs the tools and generates the report for you on demand, but it will likely need to scale with the size of your application. A large server application that runs in a data center will need different kinds of performance analysis than a desktop application, and a more robust performance infrastructure.

Think of what the ideal infrastructure is for your setting and start building it. Treat it as a first-class project in every way, with milestones, adequate resourcing, and design and code reviews. Iterate it in a way that makes the infrastructure usable in some way very early on and gradually add automation features over time.

1. Believe Only Numbers

In many teams where performance is an afterthought, performance improvements are often pursued only when problems occur that are serious enough to affect the end-user. This means there is an ad-hoc process that basically boils down to:

User: Your application is too slow!

Dev: Why?

User: I don’t know! Just fix it!

Dev: <Does something to make it faster, maybe?>

You do not ever want to have this conversation. Always have the numbers available measuring whatever it is you are judged by. Have data to back up literally everything you do. People have infinitely more credibility when backed up by numbers and charts. Of course, you will want to make sure those numbers are correct before you publically rely on them!

Another aspect of numbers is ensuring that your team has official, realistic, and tangible goals. In the example above, the only “metric” was “faster.” This is an unofficial, fuzzy, and mostly worthless goal to have. Make sure you have real, official performance goals and get your leadership chain to sign off on them. Have deliverables for specific metrics. Make it known that having unofficial pressure for better performance after the fact is unacceptable.

For more information about setting good performance goals, see Chapter 1.

1. Effective Code Reviews

No developer is perfect and having multiple sets of eyes can dramatically improve the quality of anyone’s code. All code should probably go through some code review process, whether it is via a system of reviewing diffs via email, or a formal sit down with the whole team.

Recognize that not all code is equally critical to your business. While it might be tempting to say that all code should adhere to the highest standards, that might be a high bar to reach at first. You may want to consider a special category of code reviews for software that has an especially high business impact, where a mistake in functionality or performance can cost real money (or someone’s job!). For example, you could require two developers to sign off before code submission, requiring one to be a senior developer or a subject matter expert. For large, complicated code reviews, put everyone into a room with their own laptops and someone projecting and have at it. The exact process depends on your organization, resources, and culture, but develop a process and go with it, modifying as necessary.

It may be helpful to have code reviews that focus on just particular aspects of the code, such as functional correctness, security, or performance. You may ask specific people to comment only on their areas of expertise.

Effective code reviewing does not equate to nitpicking. Stylistic differences should often be ignored. Sometimes even larger issues should be glossed over if it does not truly matter and there are more important things to focus on. Just because code is different than how you would write it, does not mean it is necessarily worse. Nothing is more frustrating than going into a code review expecting to dissect some tricky multi-threaded code and instead spending all the time arguing about correct comment syntax or other trivialities. Do not tolerate such wastes of time. Set the expectations for how code reviews should run and enforce it. If there are legitimate standards violations, do not ignore them, but focus on the important things first.

On the other hand, don’t accept lame excuses like, “Well, I know that line is inefficient, but does that really matter in the grand scheme of things?” The proper response to this is, “Are you asking how bad you’re allowed to make your code?” You do need to balance overlooking minor issues with the need to create a culture of performance so that next time, the developer does the right thing automatically.

Finally, do not buy into the notion of total code “ownership.” Everyone should feel ownership for the entire product. There are no independent, competing kingdoms, and no one should be over-protective of “their” code, regardless of original authorship. Having owners for the purposes of gatekeeping and code reviews is great, but everyone should feel empowered to make improvements to any area of code. Check your ego at the door.

1. Education

A performance mindset requires training. This can be informal, from a team guru or books such as this one, or formal, with paid classes from a well-known lecturer in the area.

Keep in mind that even those who already know .NET programming will need to change their programming habits once they start acquiring a serious performance mentality.

Likewise, people who are well-versed in C or C++ will need to understand that the rules for achieving good performance are often completely different or backwards from what they thought in the native code world.

Change can be hard and most people resist it, so it is best to be sensitive when trying to enforce new practices. It is also always important to build leadership support for what you are trying to accomplish.

If you want to kick-start some performance discussions with your peers, here are some ideas:

1. Host brown bag lunch meetings to share what you are learning.
2. Start an internal or public blog to share your knowledge or discuss performance issues you have discovered in the products.
3. Pick a team member to be your regular performance-related reviewer.
4. Demonstrate benefits of improving performance with simple benchmarks or proof-of-concept programs.
5. Designate someone as a performance specialist who will stay on top of the performance, do code reviews, educate others about good practices, and stay on top of industry changes and the state of the art. If you are reading this, you have already volunteered for this.
6. Bring up areas of potential improvement. Tip: It’s best to start with your own code first!
7. Get your organization to buy copies of this book for everyone. (Shameless plug!)

1. Summary

Start small when creating a performance mindset in your team. Begin with your own code and take time to understand which areas actually matter for performance. Build an attitude that performance regressions are just as serious as functional failures. Automate as much as possible to reduce the burden on the team. Judge performance metrics on hard numbers, not gut feeling or subjective perception.

Build an effective code review culture that encourages a good coding style, a focus on things that really matter, and collective code ownership.

Recognize that change is hard and you need to be sensitive. Even those familiar with .NET will likely need to change their ways. C++ and Java veterans will not necessarily be great .NET programmers right away.

Find ways to kick-start regular performance conversations and find or create experts to disseminate this information.