© Erik Ostermueller 2017
Erik OstermuellerTroubleshooting Java Performancehttps://doi.org/10.1007/978-1-4842-2979-8_1

1. Performance Anti-Patterns

Erik Ostermueller
(1)
Little Rock, Arkansas, USA
 
Before getting into the details of performance troubleshooting and tuning, it is helpful to have a high-level understanding of how performance defects make it into code in the first place. The four anti-patterns presented in this chapter illustrate the main kinds of defects you will encounter in Java server-side software. Throughout the book, we will refer back to the performance anti-patterns described in this chapter.
The objectives of this chapter are:
  • Learn how to start with unfamiliar-looking, raw data from a performance monitoring tool and create a story (aka hypothesis) around it that describes how the code got to be so slow.
  • Recognize the striking similarities between most performance defects.
  • Commit to memory the names of the four Main Performance Anti-Patterns, or perhaps define your own anti-patterns.
  • Learn how to assess whether a given defect is a large or small processing challenge.
When you first start investigating a performance problem, you know very little—perhaps you have a scrap or two of suspicious-looking data from one of the monitoring tools mentioned in the four P.A.t.h. chapters later in this book—but there is no smoking gun, no fix-in-the-waiting. This is the problem: you are stuck with just a sketch of the slowdown and you need a detailed picture to identify the culprit.
To fill in those details and to better understand the full story behind your defect, this chapter provides dossiers on the four most common types of performance defects—I call them the Main Performance Anti-Patterns.

Main Performance Anti-Patterns

Here are the Main Performance Anti-Patterns. As you are investigating a performance problem, ask yourself which of these anti-patterns your defect most closely resembles.
  1. 1.
    Unnecessary Initialization: While initializing some larger process, the system frequently reprocesses a small result that has already been processed. The accumulated effect of the many executions is extra CPU/RAM consumption and/or slower execution that can all be avoided. Caching the small result is the main antidote.
     
  2. 2.
    Strategy/Algorithm Inefficiency: A misconfigured or poorly chosen algorithm or coding strategy is causing performance problems. A strategy is a technique used throughout a code base and an algorithm is a plan used to implement a single component.
     
  3. 3.
    Overprocessing: The system is doing unnecessary work. Removing that work provides measurable performance benefit. One example is retrieving too much data, where most of the data is discarded. Another example is including in your load test the wrong resource-hungry use case—one that is seldom used in production.
     
  4. 4.
    Large Processing Challenge: Attempting to process and conquer a massive amount of data. Very few applications legitimately have such a need, but they do exist. Querying 4 billion rows of data. Transferring 10MB of data repeatedly over a slow network, and so on.
     
Unfortunately, there is overlap between these main anti-patterns, where one performance defect might fit two or more different main anti-patterns. To get around this fuzzy categorization issue, I have sorted these four issues from the most to least common (based on my own experience). This enables you to walk down the list in order and choose the first one that has the best fit to the problem you are faced with.

Main Anti-Pattern #1: Unnecessary Initialization

This anti-pattern frequently shows up in initialization, prior to actually doing something. This is a type of inefficient, CPU- and I/O-intensive coding in which certain tasks that have already been processed are unnecessarily reprocessed, often ad nauseum, causing unnecessary overhead. For example, rereading and reparsing an XML Schema .xsd file for every schema validation is unnecessary because the data in the .xsd file rarely changes. Instead of rereading/reparsing the .xsd, store and use a cached version that is already parsed.
Well, .xslt files are static too, and they have the same issue as .xsd files—they must be read from the file system (or network) and parsed before the transformation party begins, but most systems incur the reread and reparse overhead for every single transformation.
It turns out that static data , like .xsd and .xslt files, is everywhere, and there is a performance penalty to rereading or reparsing all that data. Here are examples of application-level data with this same problem:
  • Authorization data. Which user has been granted which roles, and which roles have permissions to which resources? If you decide to cache this data to get a nice performance boost, the cache can easily be evicted-on-demand to accommodate time-sensitive security changes, like revoking privileges from your favorite disgruntled ex-employee.
  • Product data. Yes, it would be tough for Wal-Mart and Amazon to fit their modest product offerings in RAM, but many organizations whose product offerings can fit nicely into a half gigabyte of RAM can boost performance by loading product configuration into memory at system startup.
  • Organizational data, like physical plant locations, lists of countries of the world, states in the USA, and so on.
  • Processing configuration data, like various transaction codes that orchestrate the flow of debits and credits in a banking system.
Repeatedly reloading any of those kinds of static application data causes measurable and problematic overhead. However, these improvements are not limited to application data, because there are even lower-level objects that are costly to repeatedly reload, as well. For example, many XML processing APIs sure seem like 100% of their processing is done in-memory, until you discover large, heaping portions of file system reads, jar-file-unzipping, and synchronization. Part of the problem is that the majority of Internet XML how-to articles advocate coding idioms that are insensitive to performance concerns. All of this nasty I/O can be avoided by an alternative idiom to consuming the API. Lots of room for optimization here. So, try these suggestions. You can find the code for these on my eostermueller repo on github.com—it’s named crazyFastXml:
https://github.com/eostermueller/crazyFastXml
  • For XSLT, cache and reuse the output of javax.xml.transform.TransformerFactory.newTransformer()
  • For SAX parsing, cache and reuse the output of javax.xml.parsers.SAXParserFactory.newSAXParser()
  • For StAX parsing, cache and reuse the output of javax.xml.stream.XMLInputFactory.newInstance() and javax.xml.stream.XMLOutputFactory.newInstance()
  • For XPath, instead of using javax.xml.xpath.XPathFactory.newXPath(), use org.apache.xpath.CachedXPathAPI.
We will talk more about how to detect and fix these issues, especially in Chapter 11.
Caching the right objects helps avoid this first anti-pattern (Unnecessary Initialization). Exactly the same thing applies to networking: caching and reusing a network connection (aka TCP socket) is many times more efficient than shutting down and re-creating it for every request. For example:
  • Consider using the pooling in the Apache HTTP Client to avoid the overhead of re-creating the TCP Socket. An Internet search for “apache client connection management” will get you to the right place.
  • Some server-side JDBC apps still incur the overhead of using javax.sql.DriverManager.getConnection() to re-create the TCP socket for the database connection for every SQL request. Instead, consider using Hikari, C3P0 or a JDBC connection pool provided by your container, whether it is JBoss, WebSphere, Jetty, Tomcat or some other container.
Now that we have a good picture of several of the culprits, let’s talk about a few more of the common traits that will help you recognize this Unnecessary Initialization anti-pattern when you are looking at performance data:
  • Many small I/O hits. These examples are not about large I/O hits (say, 50ms or longer) but many small I/O hits (perhaps a few ms each) executed frequently by a handful of subsystems (application initialization, XML processing, HTTP requests, and so on), incurring overhead for every round trip.
  • Performance Insensitive Idiom. The four problems with the XML APIs show that Unnecessary Initialization can be caused by using the wrong idiom to consume a particular API. Some of the most popular idioms available on the Internet, in the four XML cases, just happen to be fraught with performance problems. This is a problem.
As a performance recommendation , “avoid unnecessary I/O” is a cliché and banal, but it also happens to be the root cause of this bountiful class of performance defects. Ouch. Let’s pay closer attention to the “avoid unnecessary I/O” recommendation and minimize requests to the disk and over the network.

Main Anti-Pattern #2: Strategy/Algorithm Inefficiency

This Main Anti-Pattern is about choosing an algorithm or strategy that doesn’t perform well. Also included, though, is choosing the right algorithm with the wrong algorithm configuration/parameterization. Comparing the complexity of algorithms (using “Big O” notation) is out of scope for this small book.
It is not about fawning over your favorite sorting algorithms or using “Big O” notation to compare worst-case complexity. Why not? Because most enterprise software is considerably less complex. It is more about moving data from point A to point B. This book is about all the performance trouble that we regularly get ourselves into before even attempting to tackle more complex things, like algorithms.
This anti-pattern is about algorithms (structured plans of attack for processing some data), but also strategies, like a data access strategy that is an application’s approach to sculpting SQL statements to retrieve/update data. An algorithm’s implementation is normally in a single place. A strategy, on the other hand, is much more difficult to tune and/or replace because its code is scattered throughout the system.
You can use performance testing early in the development cycle to avoid investing in an ill-performing code strategy that is cost-prohibitive to replace. For example, you could discover early on whether executing a lot (say more than 25) of database or other back-end requests will perform well (it won’t).
In Chapter 9 on Persistence, I will talk about the “Biggest Performance Problem Ever” in server-side software. This problem is so big that it stretches into anti-patterns 1 and 2. Stay tuned—it’s really big and there is a good chance you have seen this already. Can you guess what it is?

Main Anti-Pattern #3: Overprocessing

This anti-pattern is about misguided attempts to embrace resource-intensive use cases, ones that are only marginally valid. One common example is where more data is processed than a single end user could possibly make use of in a single screen-full or webpage of information. In other words, expecting 50 pages of data in a single HTML table to load in 100ms is not going to happen. But 3 pages of data with a “next page” button could perform quite nicely. Another example is expecting a particularly complex use case to perform well under high throughput, even though it will see only light throughput in production.
While assessing this anti-pattern requires some technical expertise, it falls more so on business expertise. You specifically need to know which business processes are, or will be, used in production, and it would be helpful to know the rough percentages of each.
As a convenience to both developers and QA, often test databases contain a few “super customers” configured to demonstrate a large, dizzying array of business functionality, even though customers never get that large or complicated in production. Data like this in a load test is a frequent cause of the Overprocessing anti-pattern.
Here is one way this unfortunate scenario will play out: you create a load test that stresses out your system (Chapter 4). The four chapters in the P.A.t.h. section of this book will then walk you through hooking up monitoring tools and finding the slowest parts of your code and the highest CPU consumers. This is all good and well—no anti-pattern here. The problem happens if you get eager and spend lots of time tuning a particular section of code that users rarely traverse, but just happens to be inadvertently included in your load tests. This is time wasted.
Here is another example. Consider an inadvertently oversized RDBMS result set that takes extra time to process. An overly-broad WHERE clause in a SELECT could cause this, like a misplaced wild card that retrieves all customers matching anything. But unexpected and invalid data in the database could be at fault, too. And do not forget about the quintessential example mentioned earlier, where your system wastes resources cramming huge amounts of data into a single web page, more data than a single user will be able to digest.
So remember: performance tuning is as much about sculpting code to go faster as it is about making smart choices about which business processes to load-test. It is also about trade-offs between performance and user interface design—like keeping web page sizes small to accommodate performance concerns.
This means that when your head is stuck under the hood of your system looking at questionable performance from a large result set, you have to reorient your thinking and consider more abstract things, like performance/usability trade-offs and the throughput needs of the business. This human context switching is not easy , but consider how helpful it is: some tuning challenges can be avoided altogether because the underlying use case was so unrealistic that it shouldn’t have been included in the load test in the first place.

Main Anti-Pattern #4: Large Processing Challenge

This anti-pattern describes I/O and other hardware limitations to big performance challenges, like lookups on a few billion rows of data or repeatedly pushing several megabytes over a really slow network. Enterprise software development has very few of these large, difficult challenges. But when you do encounter one, I recommend extra design and testing care, respect, and a healthy dose of trepidation. Optimizing everything else is almost trivial.
“Large Processing Challenge” is the last of the four Main Anti-Patterns. As I mentioned, anti-pattern 1 shows up the most, and anti-pattern 4 shows up the least. You can count yourself as an exception if you are on the bleeding edge and trying new things, and regularly encounter many of these big challenges. Congratulations. But for the rest of us writing enterprise web applications, there are very few big processing challenges that inspire this kind of fear. Here are a few examples of these large challenges:
  • Lookups on just a few records in a table with 200 million records.
  • Transferring a lot of data (say more than a megabyte) repeatedly over a slow network, especially over long distances.
  • Multi-threaded performance is tough to get right, but there are many readily available APIs and solutions, where someone else already got performance right.
How will you know which processing challenges are “Large?” These kinds of challenges are so enormous, they will show up regardless of
  • Language (Java, C#, C++, …)
  • Operating System
  • Selection of API (like StAX vs SAX for XML parsing)
  • Idiom your code uses to access the API, like Xalan vs. Saxon for XSLT transformation
Preparing for these large challenges takes so much work, it seems that preparations should begin back when the design document is created. If you truly fear a large processing challenge, invest the time to test its feasibility early in the project; perhaps hardware experts could contribute here.
Its common for developers to mistake an inadvertant glut of data for a large processing challenge. For example, why suffer slow responses from queries through 10 years of historical data when queries through 2 years of data would perform quite nicely, and the business side would be happy retaining a single year of data?
The troubleshooting techniques in this book will certainly identify performance issues with large processing challenges, both real and imagined. However, I must admit that creating solutions for large challenges is out of scope for this book, mostly because genuine issues of this type are rare.

Assessing the Size of a Processing Challenge

It makes perfect sense why large processing challenges (Main Anti-Pattern 4) show up on some kind of a performance radar that highlights performance problems, especially when troubleshooting online, Java server-side systems—the main topic of this book.
Because of the hard work it takes to tackle large processing challenges like this, development teams should anticipate the extra work and, in scheduling, increase the priority of the performance tuning effort. However, when smaller performance challenges show up on performance radar, they get in the way of other priorities like tuning large performance challenges. They steal time from other parts of the development lifecycle as well: design, development, QA, and so on.
Tuning small performance challenges is essentially a nuisance; in my experience, slowdowns in small performance challenges make up an unexpectedly large portion of the pool of all performance defects.
So when looking at any piece of data that might indicate a slowdown, do this:
  • Make a rough, mental assessment of whether the processing at hand is a large, medium, or small processing challenge.
Here is an important hint to help find small challenges to tune: generally, when tasked to code something like an SOA service or a web request, there is one main task to be processed. An order must be placed; the query results must be returned; the store closing must commence. Almost by definition, processing that happens before or after the “main task” is a small challenge, and any corresponding slowdown is a nuisance to be eradicated. Likewise, so called “cross-cutting concerns ” like logging, auditing, and security and even monitoring itself are likely candidates for small processing challenges.
Furthermore:
  • When a slow-looking, small processing challenge involves a publicly available API, that API’s user community is highly likely to have experience with the slowdown for you to learn from.
Take the time to seek out this performance experience—on doc pages, Internet searches, wherever. Just tack the word “performance” to the name of the API in your search criteria and search away. Ask questions in user forums. Also, when I say “publicly available API,” certainly anything available from github.com and mvnrepository.com falls in this category. I have seen a great many performance defects in my ten years of tuning experience, and the good people of the Internet always seem to beat me to the punch. When the size of the processing challenge is small and the API in use is publicly available, others have already documented the issues and discovered reasonable workarounds.

Using the Main Performance Anti-Patterns

Here is walkthrough of how to put these anti-patterns to work.
  1. 1.
    Start by collecting data using the P.A.t.h. Checklist, which begins in Chapter 8. Of all the P.A.t.h. data, choose the measurement/result that looks the most problematic, whose fix would make the most positive impact on performance.
     
  2. 2.
    Ask whether this “most problematic” measurement looks like main anti-pattern 1.
    • Does it deal with static data, or at least no dynamic data?
    • Is the static data reprocessed for every system request?
    • Is a small amount of I/O involved each time the static data is reprocessed?
    • …and so on
     
  3. 3.
    If the data seems to fit main anti-pattern 1, then congratulations, you have built a nice hypothesis for the problem. To test the hypothesis, apply a fix to your system and retest and reassess performance. This is how to “build a case” for a performance defect.
     
  4. 4.
    If the “most problematic” measurement data does not seem to fit 1, then move on to main anti-pattern 2.
     
  5. 5.
    Continue to evaluate all four main anti-patterns. It’s highly unlikely that none of them will fit.
     

Don’t Forget

A large portion of today’s server-side performance problems are generally caused by a handful of trivially small processing tasks whose performance could and should be good, but in actuality is not. These “trivially small processing tasks” mostly show up in Main Performance Anti-Patterns 1 and 2, “Unnecessary Initialization” and “Inefficient Algorithm/Strategy.” Actually, anti-pattern 1 is a special case of 2.
Keep in mind that an inefficient algorithm is a lot easier to refactor or replace than a strategy, whose details are scattered throughout the code base. The most typical example is a really chatty database strategy that executes dozens of SQL statements for each request: it performs poorly and is very costly to refactor.
Although seen less frequently, the other two Main Performance Anti-Patterns (3 and 4) are “Overprocessing” and “Large Processing Challenge.” Large processing challenges should be identified (and feared) in design. As you start to find parts of the system that might be slow, don’t forget to ask whether “overprocessing” is the cause. This is when you apply unrealistically large amounts of load to an infrequently used business process, or when your code casts a wide net and processes a large amount of data, only to discard a large portion of it.
Throughout the rest of this book, we will continually refer back to these Main Performance Anti-Patterns as a quick way to communicate and understand the root of cause of the various problems that really drag down server-side Java performance.

What’s Next

Having trouble finding the right environment to performance test in? The next chapter is just for you. The code examples that come with this book (not discussed in detail until Chapter 8) demonstrate more than a dozen very commonly found write-once-run-anywhere (WORA) performance problems. Chapter 2, coming up next, will show you how to create an inexpensive, modest-sized tuning environment where you can get started right away finding these and other performance defects, ones that would likely cause problems in production.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.128.255.24