Synthetic testing is your first answer to the question, “Could they do it?” It’s a reliable, controllable, repeatable measurement that you can use regardless of how much or how little traffic there is on your website. It also has some significant shortcomings.
Synthetic testing services don’t know how much traffic is on your site. A testing service is blissfully unaware of whether your site is experiencing a deluge of traffic or is so slow that it has idle servers. This means they’re missing an important possible cause of web latency, because the more visitors you have, the longer the site takes to respond. Alerts and thresholds on synthetic testing systems can’t take into account how busy the site is.
You should, of course, be concerned if your site becomes slow when nobody’s using it, but a synthetic testing service won’t alert you to that fact as long as the latency is within acceptable limits. Dynamic baselining, in which the service learns what “normal” latency is like at a certain time of day, is somewhat of a proxy for load, assuming your website gets the same loads at the same times of day.
Synthetic tests generate traffic. If you’re running tests on a site, exclude the synthetic tests from overall analytics before you start testing or your visitor count will be artificially high, as shown in Figure 9-31.
If you’re using a CDN to speed up traffic, you should test the performance of retrieving a page from the CDN, but also of retrieving the same page from your own servers. To accomplish this, you’ll need a second domain name that bypasses the CDN. By comparing both test results, you’ll see how much the CDN is helping with performance and whether you’re getting your money’s worth.
Most of this chapter has dealt with monitoring HTML-centric applications. The modern Web is changing that in two important ways. First, more and more applications rely on rich clients written in JavaScript, Flex, and so on. These clients do much of the presentation and display on the end user’s computer. Instead of retrieving whole pages, the client retrieves smaller nuggets of information from the server, often structured in XML or JSON. The testing service must be able to emulate these kinds of calls.
This is an area where browser puppetry works better, because there’s a real browser executing the tests. Nevertheless, rich clients have destroyed much of the standardization that we enjoyed with traditional HTML-centric, page-by-page web design, forcing us to rethink web monitoring.
In some cases, the user may also install client-side software, such as a streaming video helper or a plug-in that uses a non-HTTP protocol like the Real-Time Streaming Protocol (RTSP) or even UDP instead of TCP. If this is the case, you need to extend your synthetic testing to test these new protocols, and this will limit the synthetic testing vendors you can use.
Video monitoring is a special case that will change which metrics matter to you. Rather than response times, you’ll care about startup time (how long video takes to begin) and rebuffering ratio (what percentage of visitors saw a message saying “rebuffering...” while playing the video). You may even measure details like frame rate and effective bit rate that can only be collected within a media player. If you want to measure this synthetically, you’ll need “media player puppetry” from your testing provider.
Changes to a website are the most common cause of false alarms. Test scripts are inherently “brittle”—when the site changes, the tests stop working properly. It may be a transaction that that no longer follows the same navigation path through the site, a new URL, or a different response that the service interprets as an error.
There are no simple technical solutions to this problem. They’re human issues, and you need good processes like change control management to overcome them. Remember, however, that tests using browser puppetry are less likely to falsely report errors when you change a page: if a URL is altered but the button’s name remains the same, the test will still work.
Synthetic tests still consume resources on your servers. We’ve seen bad test plans that caused over 60 percent of all the site’s traffic via the testing itself. This is a tremendous waste of money and makes it even harder to identify real users who are having problems amidst the noise.
Synthetic testing data contains historical information that you may want to use elsewhere. When selecting a synthetic testing platform, you should be aware of how much data the system stores and for how long. This is particularly true if your organization relies heavily on data warehousing or business intelligence systems. Data should be downloadable via XML, CSV, or a real-time feed.
Synthetic tests let you keep tabs on your competitors. You can monitor their performance and availability and use it to set your own service-level targets. For SaaS companies, knowing that your competitors are slower or less reliable than you are is an important selling point.
Some synthetic testing companies offer comparative benchmarks that rank members of a particular industry segment against one another.
It’s easy to let synthetic traffic lull you into a false sense of security. Often, synthetic tests will be fine, even as users are suffering. This happens for two main reasons:
Your tests don’t simulate the network, region, bandwidth, browser, or desktop properly.
Users are doing something on the site that you aren’t testing for.
This is one of the main reasons that you should always use synthetic testing in conjunction with RUM, which we’ll cover in the next chapter.