Output Metrics

Active monitoring includes the following key metrics:

Availability

The key metric from active monitoring. Because such monitoring is typically prescheduled, the “fate” of every active test in terms of success (access to the test site) or failure is known. In contrast, passive monitoring (RUM) analyzes the response of successful visits to the site only.

Total response time

It is important to understand what the page load or response time reported by active monitoring systems actually represents. Such tooling operates by effectively listening to the network traffic associated with a web page request. Thus it records the total time for page delivery—that is, the elapsed time between the receipt of the HTTP request and the delivery of the final byte of data associated with the page. This response time and the underlying component-level data is extremely valuable in understanding comparative performance, and in the isolation of some categories of application issues. Although page response times will be longer in end-user test conditions that make use of consumer-grade devices over tertiary ISP or wireless carrier links, this metric differs from the browser fill time experienced by an actual site user. This latter metric, also known as the perceived render time or some variant thereof, is recorded by some of the more sophisticated passive monitoring tools. Figure 7-3 illustrates the difference.

ISP Testing Best Practices

When configuring ISP-based testing, you should consider the following points:

Connectivity

This class of testing is designed to be best case. It is crucial to ensure that no constraints or variations in the test environment exist, either of connectivity or test node. The major vendors of this type of monitoring will either deploy multiple nodes within tier-one ISP data centers close to the major confluences of the Web, or locate testing within LINX exchanges. The latter approach ensures that a range of ISPs are selected in successive tests, while the former enables specific peerage issues to be screened (at least among the ISPs for whom test nodes are available). Avoid tooling based in tertiary data centers where high levels of minimum bandwidth cannot be assured.

Triangulation

Undertake testing from across a variety of ISPs and/or test nodes. Doing so ensures that any performance disparities reported are effectively isolated to the test site concerned (assuming that the issue is detected across multiple nodes/tests), rather than potentially being an ISP- or test-node-specific issue—which could be the case if testing was undertaken from a single node. Ideally, you should use three or four nodes, distributing testing evenly across each.

Location

When selecting vendors, and in designing tests, test from multiple locations with a similar latency. It is not essential for all test nodes to be from the same country as the target, but take care to ensure that, if not, they are in regional locations on the primary backbone of the Internet.

Figure 7-3. Total page load time (active monitoring) versus perceived render time

Frequency

With the advent of passive monitoring (RUM), which by its nature is (or should be) continuous, the requirement for high-frequency active monitoring has declined. A distinction should be made between ping (availability, aka up/down testing) where test frequencies tend to be very high and strategic, and full object-level or use-case monitoring. The key is to have recent data and trend information available to drill into when a potential issue is flagged. Thus, a heartbeat test frequency of, say, one or two tests per hour from four nodes giving a nominal frequency of one test every 7.5 minutes should suffice. Test frequency can always be increased during active issue investigation if required.

Agent type

As a general rule, the primary browser agent you use should reflect the predominant browser in the country being tested (or the browser/version with the highest penetration among your customer base if different). Browser usage changes significantly over time, and some entrenched patterns of usage (such as that of Internet Explorer 6 in China, for example) can be expected to change rapidly following withdrawal of support and malware protection by the vendor.

Analysis

Automated analysis is a useful feature of some tooling. Even if present, it provides no more than a shortcut to some of the more common factors to consider. A structured issue isolation model should be adopted and followed.

Synthetic End-User Testing Best Practices

Synthetic end-user testing differs from backbone in that it is essentially about quality of service (errors and performance) rather than ideal, comparative external tests. It is not an alternative to active ISP-based and passive (RUM) testing, but rather a supplement. Some important differences include the following:

• Public peers have the advantage of multiple tertiary ISPs. Consider black/white listing and PC specifications.

• Private peers are useful for testing from known locations. They can be particularly useful in testing in closed intranet application situations, or from specific partner organizations, for example. Private peer-agent deployment has also been successfully used for scheduled testing from specific high-net-worth customers (e.g., in the gaming industry).

• Note that end-user response data is always inconsistent—but so is the real world. Artificially throttled testing from data center test nodes is cleaner, but does not represent the heterogeneity of actual usage conditions. It also tests from very few, fixed locations with constant conditions of ISP connectivity.

• Connectivity—mapping to typical/desired users and use cases—helps you to understand site performance in limiting conditions. It is particularly important for large-footprint sites or those delivered into regions with poor Internet infrastructure.

• Mode, or wireless connectivity, is essential to understand the performance of mobile users, both from a device and network perspective. Hardwired ISP testing of mobile users should never be used other than as a reference point, because the results cannot be extrapolated to the real world. Synthetic testing of mobile end users has limitations, however, as it typically occurs from fixed test nodes. This has the advantage of more replicable testing in terms of signal strength but may suffer from saturation of the local cellular mast, which introduces distortion into the results. Triangulation of results (as with backbone tests) is particularly important, as is careful attention to the connection speed: 3G has a maximum theoretical bandwidth of more than 6 Mbps, but connection speeds of less than 100 Kbps are not uncommon in remote areas and situations of high traffic.

• Consider the browser agent used, as it is not usually possible to read across backbone and last-mile tests measured using different browsers (particularly if one of them is IE).

• Peer size is an important factor. Due to the inherent variability of testing from end-user machines, you must test from sufficiently large clusters to enable average smoothing of the results. You must carefully consider the peer results selected for object-level analysis, and engage in progressive “pruning” by blacklisting any specific test peers that consistently produce erroneous results. Take care to operate blacklisting using rational exclusion criteria in addition to poor performance; otherwise, you may be actively removing evidence of a fault that you should be dealing with. If testing from public end-user machines, adopt a minimum peer cluster size of 7 to 10 nodes as a best practice to enable meaningful baselining. Peer groups should be selected from machines and connectivity bandwidths of similar characteristics. Results should be further filtered where possible to remove outliers due to rogue low (or high) connectivity tests.

All these caveats can lead you to think that scheduled end-user monitoring is of no value in a passive (RUM)-based world. Although continuous scheduled end-user monitoring may be of limited utility (for reasons, among others, of cost), this class of monitoring can prove invaluable in isolating or excluding unusual issues (e.g., tertiary ISP peerage, DNS resolution issues, CDN assurance). The ability to study object-level data from end users and error codes provides a useful view of quality of service to end users (see Figure 7-4), particularly in areas where base traffic is low (e.g., new markets).

How Passive Monitoring Works

The most common technique for browser clients is to place a JavaScript tag or beacon in web page headers designed to detect user traffic. In most cases, the tag permanently resides on instrumented pages, placed via a simple cut-and-paste or a global include statement. EUE toolset vendors are increasingly offering the option of dynamic injection of JavaScript into the visitor browser by the site application servers during the page request.

The code executes, writing cookies as needed to capture browser information, navigation metrics, and potentially other data such as connection speed or ISP. After the on-unload step, data is passed back using a GET request via HTTP or HTTPS to the vendor database. This data is then made available via appropriate dashboards in the product client.

Passive (RUM) tooling is typically provided either as a standalone product or as part of an integrated end-to-end visibility extension to APM tools. While the mode of action is similar, APM tooling has the advantage of associating browser-side performance metrics with those of the delivery infrastructure, whether application (often to code method level) or hardware. A rapid growth in recent years has followed the publication (by W3C) and adoption by many browser vendors of standard navigation and resource metrics. Although broadly similar from a basic technical standpoint, passive (RUM) tools do differ fairly substantially. Some key considerations are listed here:

Sophistication/coverage

As previously mentioned, many passive (RUM) products are based on the standard W3C navigation metrics, which means they are not supported in all browsers, primarily older versions and Safari. In certain cases, basic performance data is collected from these metrics to supplement the core metrics. Key aspects of sophistication include the following:

• Ability to record user journeys: Less evolved products act at the individual page level.

• Ability to capture and report individual session-level data: This includes reporting on business-relevant metrics, such as transaction abandonment and shopping cart conversion by different categories of user.

• Detailed reporting: This includes reporting on bounce rate, stickiness (time on site), abandonment, and similar.

• Ability to record above-the-line performance: Browser fill or perceived render time.

This metric is rarely estimated by active monitoring tools (the only one of which I am aware is WebPagetest). Therefore, passive (RUM) tooling supporting this metric provides a useful additional perspective into end-user satisfaction.

Real-time reporting

Tools vary in two principal ways with regard to data handling:

• How long captured data is stored. As with other monitoring, the problem for vendors storing customer data is that they rapidly become data storage rather than monitoring companies. However, the ability to view trend data over extended periods is extremely useful, so individual vendor strategies to manage that requirement are relevant. This problem is exacerbated if object-level metrics are captured.

• The frequency of customer data updates. This can vary from 24 hours to less than 5 minutes. Near-real-time updates are relevant to active operations management, while daily information has limited historic value only.

All traffic or traffic sampling

Because passive (RUM) data is inferential in nature, it is important to capture all visitor traffic rather than a sample. Some tooling offers the option of user-defined sampling, often to reduce license costs. This is unlikely to be good practice except possibly in the case of extremely high-traffic sites. Within Europe, this situation is exacerbated by EU legislation enabling individual users to opt for do-not-send headers, which restrict the transmission of tag-based data.

API access

Passive (RUM) tooling will always provide some form of output charting or graphing. You can derive additional value by integrating passive (RUM) data with the outputs from other tooling. This is particularly true for those products that do not report on session-level data, such as conversion and abandonment rates. In such cases, it may be advantageous to combine such data from web analytics with passive (RUM)-based performance metrics.

Page or object level

Although, theoretically, all products could be extended to capture object-level rather than page-delivery metrics, this is not the case with every passive (RUM) offering.

User event capture

This is the ability to record the time between two events (e.g., mouse clicks). Such subpage instrumentation is valuable in supporting design and development decisions.

Extensibility

This is the ability to capture and integrate nonperformance user data. Examples include associating user login details with session performance, and collecting details of the originating application or database server.

Reporting

This metric refers to the extent, type, and ability to customize built-in reporting.

The strength of passive monitoring (RUM) lies in the breadth of its reach in terms of browsers/versions, geography, ISP, and similar. High-level patterns of performance—for example, regional CDN performance—and national performance delivery deficits in challenging markets can be identified. At a more granular level, patterns of performance between browsers or ISPs can readily be determined, which in the case of smaller players would be unlikely through active monitoring.

Passive monitoring (RUM) has two primary weaknesses:

The inferential nature of the results

Passive (RUM) data, being based on visitors, is essentially reactive. It does not support the proactive testing of particular markets, devices, or functionality. Different patterns of usage in different markets could be a reflection of cultural differences or of functional/nonfunctional issues, requiring active monitoring to accurately determine. By definition, prospective site visitors who fail to reach the site are not visitors, so the concept (and metric) of availability is not valid.

The absence of object data (in the majority of cases)

This is extremely limiting if you are seeking to isolate the cause of a performance issue.

Active Pros

• Enables you to target monitoring to issues, browsers, location, conditions, content, services (e.g., multimedia, payment service provider)

• Offers round-the-clock (if required) visibility in periods of low base traffic

• Provides consistent comparative data (internal or competitive)

• Allows you to monitor any public site (benchmarking)

• Is easily repeatable

• Can capture host-level data

• Allows full-object capture and single-object testing

• Enables availability testing

Active Cons

• Is not real world

• Provides only a subset of true conditions

• Can be conducted only with limited frequency (depending upon a realistic budget)

Passive Pros

• Provides true visitor understanding

• Allows discovery of origin-related issues

• Provides an aggregate understanding of application components (CDN performance, for example)

Passive Cons

• Is potentially limited in new markets because visibility depends on visitor traffic

• Is inferential—that is, not based on proactive test design

• Can be counterfactual (e.g., poor quality of service producing low regional traffic volumes)

• Offers limited understanding in that results are not (usually) object based, can be limited by browser coverage, and can be skewed by cookie blocking from end users

Tool Selection Criteria

A number of professional tools exist for external monitoring. As with any other competitive market, the cost and detailed characteristics of individual tools will vary. You should make your selections with knowledge of these details, in the light of the needs and constraints of your particular business. While several tools may fit the bill, mixing more than one of the same (e.g. active testing) is not good practice. Differences in test agents, design, test location, and more can make comparing results a difficult undertaking. At the very least, you should undertake a detailed parallel test study to determine the offsets between the tools, although these may not be consistent or linear.

In selecting suitable tooling, here are the key considerations:

Technical sophistication

Ability to detect and isolate issues, and to support evolving site components such as RIA technologies (e.g., Flex); server push functionality; scripting filtering and custom coding; and validation for logins, message-based components (SMS), and multimedia stream monitoring (encryption codices, formats, and adaptive streams). Specific features within some tooling of potential benefit include the following:

• Automatic analysis and scoring against key performance factors and page components

• Alert sophistication and integration with third-party alert management tools

• Screen capture and trace route on error or success

• Ability to parse and trace individual (e.g., third-party) object performance

• Ability to test complex transactions (e.g., select random or nth product from a list, select only in-stock product)

Ease of use of product with regard to scripting, test commissioning, and reporting as well as the speed of response of vendor support and script modification.

The relative importance of these factors will depend on the time criticality and dynamic nature of the application(s) monitored and the extent of in-house skills.

API

Ability to integrate data, alerts, and so forth with external systems

Report flexibility

Ability to compare multiple perspectives (e.g., traffic, RUM, mobile, backbone) on single chart

Cost

As with all tooling, total cost of ownership should be considered—licenses, personnel, training, and support. Dependent upon the flexibility of the tooling and the licensing model, strategies to mitigate cost include the following:

• Maintenance windows (change test frequency outside core business hours)

• “Follow the sun” testing

• Continuous and active heartbeat testing

Flexibility/range of test perspectives (relevant to your particular site)

It is important to bear in mind that (a) all monitoring tools will provide a response metric of some kind, but (b) as performance analysts—or at least consumers of this data—we are required to draw business-relevant insights from them, and the measurements can and do vary wildly between vendors and across the various products offered by given providers (see Figure 7-6). The first key to effective external monitoring is to understand the question you wish to ask and select tooling that will enable you to ask it.

Figure 7-6. External monitoring—a many-headed beast

Tool selection considerations relevant to active (synthetic) and passive monitoring (RUM) differ, as we’ll discuss next.

Active Monitoring Tooling

There are a relatively small number of active monitoring solutions to choose from. The predominance of web technology and the relative ease of passive monitoring (RUM) deployment makes RUM the easy option. As we will discover, active (synthetic) monitoring provides unique insights that RUM cannot offer, but is more complex to deploy and configure and is not intended for discrete monitoring of end-user performance. Synthetic monitoring typically has capability beyond just browser clients into thin-client technology, such as Citrix, and legacy fat clients such as SAP GUI.

• Clean-room (ISP backbone) testing.

  • Test node location and class, connectivity/assurance.

  • Number of nodes in relevant regions (minimum three at same round-trip latency).

  • Number and type of test agents—native browser agents (and type) versus custom agents.

• End-user testing.

  • Consumer-grade end user versus fixed-bandwidth data center testing.

  • Number, quality, and control of test peers. It can be important to test from particular tertiary ISPs or at specific bandwidths:

    • Mobile device testing—emulation versus real devices.

    • Wireless carrier/known bandwidth versus hardwired ISP connectivity.

• The importance of controlled test conditions. Figure 7-7 illustrates the effect of wireless connectivity conditions. Paired 3G wireless carrier and WiFi-based tests for a panel of major UK ecommerce sites are shown. Note the high (and variable) discrepancy between the readings.

Figure 7-7. Test-panel mobile landing-page response, 3G versus WiFi

Passive Monitoring Tooling

At the time of writing there are over 50 passive (RUM) products available. Their large growth has been driven by the W3C’s publication, and the subsequent adoption by many browser vendors, of a number of performance-related metrics. The metrics most relevant to this discussion are for navigation and resources.(see Figure 7-8).

Figure 7-8. W3C navigation metrics

Although some convergence in functionality is becoming apparent (at least in terms of stated vendor roadmaps, even if not always currently available), the following aspects are worth considering when you are selecting a passive (RUM) product.

• Standalone or integrated (e.g., with APM tooling)

• Real-time reporting (can vary from less than 5 minutes to 24 hours).

• All traffic or traffic sampling.

• API availability and granularity.

• Alert capability; that is, the ability to customize and export to existing alert management tools.

• Browser and device coverage—versions, types (e.g., Safari).

• Manually instrumented or dynamically injected (this has a bearing on cost of ownership).

• Page or object level.

• User event capture.

• Complete use case or single page.

• Extensibility, that is, the ability to associate and report by specific session metrics (e.g., user login).

• Business metrics, including abandonment, conversion rate, bounce rate, and time on site.

• Reporting, including standard and customizable, email-based (e.g., .pdf), dashboard outputs.

• License model and economics (e.g., per domain, by traffic).

Building Blocks of an Effective Testing Framework

Best practices for external monitoring in many ways mirrors that discussed elsewhere in relation to performance testing; that is, it begins with consideration of the business rationale for, and characteristics of, the application to be monitored:

• Why does it exist?

• Who uses it?

• What are the relevant trends and strategic vision, and when is it anticipated that these will be realized?

Begin with good information on existing usage. The following questions illustrate this approach:

• Where are application users located—now and anticipated?

• How does behavior vary in different user subsets? (This may suggest the types of test transaction to monitor.)

• What is the pattern of traffic throughout the day, week, or month? This can vary in different markets. Modeling test frequencies and monitoring windows can make big differences to the license costs of some tools.

• What differences exist in browser/version and mobile devices? Popular browsers vary widely by region (e.g., Opera in Russia). Particularly in emerging markets, versions that are long obsolete in the UK and US may still have high market share. For example, until recently, Internet Explorer 6 was the most popular browser in China.

• What is the strategic direction of the business with regard to new markets?

• What is the strategic direction with regard to mobile device usage and/or development of native applications for iOS and Android?

• Who are the key digital competitors (international or local) serving the target demographic in all key markets?

It is not necessary (although desirable, if somewhat impractical) to proactively monitor all regions and devices all the time, but the monitoring regime should seek to avoid major blind spots and to validate and extend data obtained from passive sources (i.e., passive tooling).

As you answer these questions, it is essential to work from objective sources of information (such as web analytics), rather than assumptions. The absence of promotion to a particular region does not necessarily imply that it is irrelevant from a revenue perspective. Prior to the aggressive internationalization of more recent years, many high street retailers experienced an English-speaking diaspora effect in terms of site visits, both from areas with a high population of English immigrants (such as southern Spain) as well as more permanent ex-colonial outposts. The world is not homogeneous; consider cultural/behavioral characteristics affecting devices, buying behavior, and use cases in general. As an example, consider the window shopping behavior of Japanese mobile users scanning Q codes. Other factors include the quality of the local infrastructure (developing world), legal/regulatory restrictions (Cuba, China), and the nature of user devices such as PC capacity and mobile devices (Africa).

Having determined the objective data required, and the key pages and use you need to understand, you should next consider the type(s) of monitoring to be employed. Key aspects are as follows:

• Multiple perspectives (triangulation)

• APM integration (root-cause isolation)

• Integration. (external, APM, and performance testing)

• Pre/per/post-load test results (memory leakage, cache effects, and similar)

Specific Design Aspects of Active Monitoring

When considering the design of a synthetic monitoring solution, you should take into account the following points:

Triangulation

It is important to test from multiple locations, and ideally from a range of core service providers. Backbone testing is designed to provide consistent, best-case external results, and test design should enable this. Ideally, a minimum of three ISP test nodes should be used per key location (e.g., country or region). If three nodes are not available within a specific country, either due to lack of availability from your chosen vendor, or to the requirement to test smaller economic areas, then locations with approximately similar round-trip latencies should be used. Political geography does not matter, but core Internet connectivity does, and this should inform your choice. Use key Internet hubs (in Northern Europe: London, Amsterdam, Frankfurt, Paris) rather than secondary backwater locations where possible.

Avoid tier-two locations (i.e., local data center) where possible—this is clean-room testing requiring consistent test conditions without test node system or communication bandwidth constraints. Failure to adhere to this suggestion will lead to problems with interpreting long-term trend data, and will affect your ability to set and monitor effective goals.

Frequency

Key considerations around test frequency include coverage, the nature of the output required, and cost management. A common approach is to set up active monitoring at a relatively low frequency, used in concert with continuous monitoring using passive (RUM) and/or APM monitoring. The detailed object level (and availability) data provided by the active testing supports pattern analysis and rapid issue isolation/confirmation of potential issues suggested by passive (RUM) results. Detailed test configuration will depend upon the tooling used, but I tend to work with groups of four nodes, each testing at hourly intervals (i.e., providing a nominal test interval of 15 minutes). Frequencies can, of course, be increased during investigation of issues, and often the tooling itself will provide additional confirmatory testing automatically when alert thresholds are breached.

If using active testing to support dashboard displays, you must ensure that test frequencies are matched to the refresh/display rate of the dashboard. Failure to do this will lead to long latencies in flagging changes in performance status and/or misleading results.

If you are using tooling where frequency of testing is a component of cost, then it is worthwhile to devise a test plan that is modeled to the patterns of usage of the application. Taking the time to introduce maintenance windows, or reduce test frequency outside core business hours across different global time zones, can lead to major savings in license costs.

Agent

In the early days of external testing, many vendors developed their own proprietary test agents. These were designed to provide good average visibility of site response across the (very limited) numbers of major browsers then in use, while increasing the detailed information available compared to the black-box limitation of native browser testing. Modern tooling almost exclusively prefers the data relevance provided by native browser-based test agents. Ideally, you should undertake testing using a variety of major browsers. This does not necessarily need to form part of ongoing monitoring, but periodic cross-browser studies enable you to capture any gross issues of incompatibility, and take into account indicative performance offsets.

Browsers differ in their detailed mode of operation, and it is important to review site design to account for these differences across browser types and versions. Such differences include the number of parallel connections, JavaScript handling, and the sophistication (or otherwise) of script prefetching. Site modifications to work around earlier browser limitations, if not reviewed, may lead to suboptimal performance with later modified browsers.

As a general rule, where possible, active monitoring should seek to match the browser agent used to the predominant end-user browser and version in the market concerned. In cases where this is not available, some vendors will enable spoofing by setting header and document object model (DOM) identifier information on a different underlying browser engine. This approach has variable benefits depending upon the nature of any underlying site issues, but it can be useful in identifying intractable problems. Such a hybrid approach is also commonly used in mobile emulation testing—that is, setting specific device headers on an underlying PC browser engine. This is flexible, and it provides consistency of performance. However, it will not uncover issues due to client-side system limitations. For these, real-device testing should be used.

Specific Design Aspects of Passive Monitoring

The key points with passive monitoring are to understand any blind spots and to compensate for them. Many passive (RUM) tools (those based on the W3C navigation metrics) cannot collect detailed metrics on Safari or older browsers. Supplementing with appropriate active testing (e.g., during periods of low base traffic) will ensure rapid response should issues occur outside core trading periods.

Integration with APM tooling will also serve to extend this understanding across the full range of visitor usage. Advanced passive (RUM) tools supporting individual session-level information are particularly useful. Such products support more advanced business—providing relevant metrics such as the relationship between performance and bounce rate or transaction abandonment.

When interpreting passive (RUM) outputs, bear in mind that the results relate to successful visitors to the site. Visit requests that were not successful will not appear. Potential visitors (or visitor subsets, such as mobile users of a particular carrier or device) who persistently experience problems may not seek to access the site in the future. For these reasons, it is advisable to approach passive (RUM) data interpretation from the perspective of expected results. Significant deviation from expectation (e.g., the absence or very low conversion rates for a popular device or certain region) should prompt you to perform validation testing using active monitoring and visual fidelity testing to offer confirmatory assurance of performance.

Key Performance Indicators for Web-Based Ecommerce Applications

KPI determination should reference many factors. In summary, these include the following:

• Best-practice values derived from top performers in a relevant business sector, “bellwether” sites (i.e., places where your customers are visiting if not on your site), and expert advice.

• Performance of key competitors.

• Historic performance of your site.

• Key use cases—that is, those with maximum business relevance. These fall into three categories:

  • Direct revenue sources (e.g., purchase transactions).

  • Activities that reduce business overhead (e.g., web-based customer service chat versus telephone-based support).

  • Functions that are important to brand perception and/or deferred revenue. Examples include color simulator for paint/car manufacturer; booking sales representative appointments.

• Key pages. Use case should always be deconstructed to examine the performance of each page or logical step, as an exclusive focus on end-to-end response times may mask poor performance at a key step. Poor performance at individual key steps has been shown to be highly correlated with abandonment.

• Subpage performance—that is, response and consistency of individual objects/links, either core or third party.

The purpose of KPIs and any SLAs based on them is to ensure and optimize business outcomes. These outcomes can take many forms, but those most relevant to ecommerce are revenue, customer satisfaction, and brand perception. Site/application behavior has both a direct and indirect impact on performance. Direct factors include lack of availability and high levels of delayed page response and inconsistency.

From an end-user-experience perspective, key performance-related metrics are response time (seconds), availability (%), and consistency (standard deviation or median absolute deviation). You need to monitor and understand the influence (if any) of performance on conversion rate, visitor abandonment, bounce rate, and stickiness (time on site).

Performance has been shown to have a less direct effect on other key revenue metrics (e.g., conversion and abandonment). These factors are influenced by many variables, including site design, user demographics, and aspects of the offer (e.g., shipping costs). Setting and managing performance KPIs while controlling for other variables has been shown to have a positive effect on digital revenue. Visitor metrics should be analyzed using robust statistics to mitigate the distorting effect of outliers. Once thresholds have been established, the use of—and setting KPIs for—APDEX-derived measures of visitor satisfaction is useful. (APDEX will be described shortly.)

Consider key markets: the nature of competitors and delivery conditions (quality of core Internet infrastructure, browser version, mobile device usage, and other factors) affects the detail of the monitoring undertaken and target KPI values. There can be little point in applying common KPIs derived from performance in advanced markets indiscriminately across all regions. It’s better to understand local conditions and constraints and seek to be the best on the ground market by market. This is likely to provide a more pragmatic and lower-cost approach.

You should have in place a procedure for the regular, systematic review of all KPIs. Factors to be considered include the following:

• Historic performance relative to existing values

• Trending of all defined metrics and business processes

• Performance and trends in key competitors and bellwether sites (e.g., BBC, Facebook)

A fully evolved performance strategy gives attention to both objective (hard) and subjective (soft) aspects of performant delivery:

Hard metrics

• External: set target responses to the ISP cloud (best case) as well as to PC and mobile device end users across a range of connection bandwidths, to cached/uncached visitors, and between total (all object) page response and above-the-line (browser fill) response time. External KPIs/SLAs should exist for the following:

  • Key use cases

  • Key pages

  • Third-party objects/affiliates

• Internal: Set and manage internal (infrastructure-based) KPI metrics. These metrics are outside the scope of this chapter.

Soft metrics

• Visual fidelity across key browser version/operating system/screen resolution combinations, and to key mobile devices/form factors.

• Fidelity of page links (internal and external).

Setting KPI Values

Consider the nature of the application and the goals for the business/brand. For example, do you wish to be in (and invest to achieve) the number-one benchmark position, top 10 percent, top quartile? A common approach is to aim for a position in the top 25 percent of all companies in your business sector (in key markets), with a response at least as good as a panel of specified direct competitors.

Reference second-order values (quoted retail industry average), as outlined in Table 7-3.

Quoted average values are very dependent upon the nature and positioning of particular sites, the selection of data, and the dynamic nature of the metrics themselves. Users should discuss this with their web analytics and/or tag management system provider to derive target KPI data more specific to their precise offer. Consider the following:

• Understand the key users of the site in terms of browser versions, connectivity (fixed wire and mobile dongle), mobile device, location, and ISP.

• Distinguish between the performance of first time (uncached) users and frequent visitors. Note that unless specifically designed (by scripting duplicate consecutive tests), active (synthetic) monitoring testing is usually uncached, thus providing worst-case responses.

• Understand key trends in interface (e.g., specific mobile phone or tablet) uptake among your key target demographics.

• Test performance to major browser (e.g., Internet Explorer, Chrome, Firefox) upgrades at beta stage if possible.

• In terms of capacity, undertake performance assurance of application KPIs:

  • Across the range of demand of the normal (daily/weekly/monthly) business cycle.

  • To anticipated peak demand—Christmas trading, sales, and so on (typically 1.5x projection from previous years).

  • Prior to release of significant site/application upgrades or other related activities (e.g., hosting provider or primary ISP changes). Periodic nonfunctional performance testing should form part of continuous integration–based DevOps strategies.

External KPIs are typically as follows:

• Business relevant, that is, they affect revenue throughput directly or indirectly.

• Based on degradation over a period or with a defined pattern of occurrence rather than point or transient episodes.

• Based on the three cardinal metrics:

  • Response time (seconds)

  • Availability (% successful tests)

  • Consistency (standard deviation, seconds)

What to measure? Typically:

• Use-case times

• Individual page times

• Response consistency

• Object/delivery component (e.g., DNS) times

External KPIs should specify the source of testing:

• Site visitor monitoring (perceived render times, browser version/device response times)

• Clean-room ISP cloud testing (absolute metrics, competitor benchmarks)

• Scheduled end-user testing (geographic/specific location testing, subjective metrics)

• Performance capacity testing (internal/cloud/end user)

There are three types of metrics:

• Absolute

  • Breaches indicate a prima facie issue requiring intervention and resolution.

  • Deviations from range of standard behavior across the business demand cycle.

• Subjective

  • Based on judgment and/or empirical behavior data (e.g., abandonment rates).

• Relative

  • To competitors (public sites).

  • To other internal applications.

  • To other objective criteria.

The preceding categories apply to monitoring metrics, that is to say, the application response across the normal cycle of business usage. Performance metrics apply to the application response across conditions of simulated demand (performance testing). SLAs imply involvement of another actor (typically external, but may be a defined internal IT/operations function).

Data Preparation

The importance of pattern interpretation (that is, the variation of site behavior across the business demand cycle) has been mentioned. However, in drawing conclusions from data, it is important to consider and minimize where possible the inherent limitations in the figures presented. These derive from three principal sources:

• Limitations inherent in the nature of the test design. As an example, some tooling vendors use artificially throttled bandwidth tests from core Internet cloud locations as a proxy for end-user performance. Others use outdated HTTP 1.0–based agents, which cannot monitor modern client-based processes.

• Limitations dependent upon the test source—for example, using tertiary retail ISP test locations rather than near-tier-one dedicated ISP or LINX-based tests.

• Limitations based on the nature of the results obtained. This is covered in the following section.

Statistical Considerations

Ongoing week-to-week or month-to-month performance monitoring (as opposed to specific deep-dive issue-resolution activities) are based on three parameters:

Availability:: The proportion of external scheduled tests that are successful or fail—reported as a percentage of total (active) tests. Response time:: The time taken for a process, page, or transaction to execute. Various subtleties should be considered (e.g., total load time versus above-the-fold or browser-fill times); these are covered elsewhere. Reported in seconds or milliseconds. Consistency:: The variation of the results from multiple repetitions of a given test. Often reported as standard deviation, although there are better metrics (as discussed momentarily).

Treatment of availability is acceptable, provided any inherent underlying confounding variables are noted/known. These would include such factors as low test volumes. As an example, a single failure from a run of 5 tests is reported as an availability of 80 percent. The same single failure from 100 tests results in a reported availability of 99 percent. When you are considering availability metrics, it is important to exercise particular care in distinguishing between reported (or what I term nominal) availability and actual functional availability.

Active (synthetic) monitoring tooling will typically record a page load as successful even if a proportion of the content has failed to download (or, indeed, if it is an error trap page of the “Our site is having problems; please try later” type delivered in place of the requested page). Before you regard reported availability as accurate, therefore, it is strongly recommended that you do the following:

• Smoke test the results. A recorded availability of less than, say, 85 percent would almost always be accompanied by other symptoms, even if they are calls to the customer support center. Is this a test artifact?

  • Does the site work when you visit it?

  • What do the site traffic and order volumes look like?

• Run a payload test. This is very useful for picking up the delivery of partial or error pages.

Measurement of response times and consistency can present more difficulties if you do not take care to understand (and, if necessary, correct for) any inherent distortion in the raw data.

Consider the charts in Figure 7-15 and Figure 7-16.

Figure 7-15. End-user average response time

Figure 7-16. End-user outlier response time

Figure 7-15 illustrates the value of examining the raw individual data points before determining the best statistical treatment. Averages can hide a multitude of sins. In this case, a scattergram display was used. It represents each individual data point as a single point, color-coded by its origin (location of ISP node). The value of such an approach in understanding underlying application behavior is clear.

Point A shows a state change in response from the British Telecom test node location. As BT carries over 80 percent of UK traffic, and any performance changes (e.g., due to peerage issues between customers using BT and the data center home ISP) can have major commercial consequences in terms of user experience. After the state change at A, two separate performance bands are detected in the BT results. This is characteristic of load-balancing issues, but more relevant to our current discussion is the effect on the distribution of results producing a multimodal distribution.

At point B, you can see the effect of an issue localized to the Verizon (orange) node. In this case, the effect is transient, and would be well reflected in an increase in average response, flagging alerting, and more detailed investigation.

Figure 7-16 illustrates a different scenario. In this case, the scattergram shows a high dispersion of results that is not transient (as with the Verizon example), but systemic. The application throws off frequent, sporadic results that are much higher (or in a few cases, lower) than the majority of the results. In the chart, the core response times (average of ~26 seconds) are bounded by the red lines. Outliers (which in certain cases are three times as long) appear in the (blue) shaded portion of the chart outside this area.

This kind of dispersion in results is not uncommon, particularly in applications that are operating close to their infrastructure capacity, either in the general run of business, or at times of peak demand (e.g., pre-Christmas trading, or in intranet applications close to internal deadlines).

The issue that this presents in interpretation is that the simple and commonly used measures of central tendency—the arithmetic mean (for response) and the standard deviation (for consistency, or results variability around the mean)—make a core assumption of normality of distribution This is demonstrated in Figure 7-17.

Figure 7-17. Normal distribution curve

As you can see, this distribution is symmetrical (i.e., the proportion of results of a given value are the same to the left and right of the center line). The presence of outliers (or other distortions such as distributions that exhibit kurtosis or skewedness) render this core assumption invalid and tend to likewise invalidate results that are based on it. In simple terms, if an application has responses to (say) 1 in 10 visitors in one month that are much higher than the majority, and the following month this does not occur (or the proportion of outliers changes significantly), the mean value will be markedly affected. This sensitivity to data quality makes the mean a poor choice when you are comparing a long series of performance results. Fortunately, a number of alternatives exist. These are more or less involved depending upon the standard characteristics of the data. While professional statisticians may have resource-to-data transformations (e.g., logarithmic), in the case of most correctly specified corporate systems, one of two approaches usually suffices, namely:

Monitor the median rather than the mean. The median is the value that cuts the data in half. It is, in statistician’s parlance, a more robust indicator than the mean. In other words, it is less affected by small numbers of results that vary from a standard normal probability distribution.

In cases where preliminary examination of the data shows a large number of outliers and/or the results of such regular outliers are likely to significantly distort the mean, you should see a truncated distribution when reporting on central tendency (i.e., the peak of the probability distribution curve—which is what the mean seeks to do). As mentioned, data transformation is one solution.

Provided that the residual data volume is sufficiently high, a simpler solution is to use a trimmed mean. The trimmed mean is calculated in the same way as a simple arithmetic mean (i.e., the sum of all values divided by their number), but a fixed proportion of data points are removed (the same amount for either end of the scale). As an example, a 20 percent trimmed mean would discard the highest and lowest 20 percent of values. The amount of trimming required will depend upon the particular results distribution obtained.

If the data is reasonably normally distributed, then the standard deviation can be used. Should many outliers exist, another truncated distribution (such as Caudy) may be preferred to model the data. This is similar to Figure 7-17, but it has fatter tails. You may be more familiar with Student’s t, which has the same probability distribution (at 1 degree of freedom) as the Caudy distribution. The equivalent consistency measure for this model is the median absolute deviation (MAD). Due to differences in its calculation, the MAD is more resistant to the distorting effects of outliers than the standard deviation.

Although replacing the mean and standard deviation with the median (or trimmed mean) and MAD will assist in ongoing comparison and iterative goal setting for your site, it has the disadvantage that these figures are not quoted for other (e.g., competitive) sites. Although the assumption of normality cannot, of course, necessarily be made for these sites either, it may be helpful to note that, for normal distributions, multiplying the MAD by 1.48 will give the appropriate standard deviation.

An alternative, if justified, would be to monitor key competitors and apply the transformation to the results obtained. This is rarely an appropriate use of investment, but one other metric is worthy of mention, as it can provide a comparative figure independent of the mean. This is the percentile rank. The percentile rank represents the figure above (or below) which x% of results lie. This metric can sometimes provide a simple alternative comparator for performance over time. It is more commonly used in situations such as the comparison of year-on-year performance of students at public examinations than in performance monitoring. It is most seen in an IT context (often at the 95th percentile) in situations of utilization/capacity planning, such as burstable bandwidth contracts.

Correlation

Statistics is a fascinating field of study, but the further reaches of it are perhaps best left to researchers who have the time and justification for their efforts. The small number of robust metrics just described is sufficient for almost all day-to-day monitoring purposes.

One final metric, correlation, is worthy of mention in an introductory text. You may have discovered an issue that has several possible causes. Correlation enables statistical comparison of two data sets. It provides a measure of the extent of association between them. This does not necessarily imply causation, as other factors may be acting on the data measured, but measuring correlation can certainly serve to exclude unconnected metrics, and to some extent rank others in terms of likely importance.

Correlation between data may be simple (i.e., linear) or more complex, requiring various transformations of the raw data. Rapid comparison of data sets for linear relationships is straightforward with Student’s t test. Details of how to apply the test and the tables required for derivation of the correlation coefficient can be found in any basic statistical text.

If you are using the t test, here are a few points to bear in mind:

• It is a measure of a linear (straight line) relationship between variables, but others (e.g., exponential) may exist.

• Its accuracy will be affected by the size of the sample—biological statistics sometimes works with as few as 20 pairs, but the more the better—and this is not usually a constraint in performance work.

• The correlation coefficient varies between zero (no relationship) and one (perfect relationship, either negative [–1] or positive [+1]).

Interpretation will depend upon the circumstances, but values greater than around 0.7 may be regarded as relatively strong.

One final point: if you are using Student’s t-tables, you will find values for one- and two-tailed tests. Which to apply? This depends on what you are trying to determine. A one-tailed test indicates the strength of a single-direction (positive or negative) relationship between the pairs of figures—in other words, when testing the strength of a particular hypothesis. The two-tailed test indicates the likelihood of either a positive or negative correlation (i.e., when you have no idea whether any relationship exists).

In performance work you will almost always be using one-tailed tests. As an example, consider Table 7-4: what aspects of third-party affiliate content have the most impact on site performance?

In this case, neither of the factors examined had an overwhelmingly strong claim to be the key factor, and (of course) they are all interrelated; the more third-party objects you have, the more connections there will be, and so forth. A clear understanding would require a larger sample volume and (probably) better test design than was applied here, but, inasmuch as it illustrates the point, examination of the correlation coefficients shows that, for this single test at least, the number of third-party hosts appears to have the least single influence on page load performance, while the number of third-party connections has the greatest.

Further investigation could consider whether some combination of individual factors provided a stronger result (e.g., one metric divided or multiplied by another), or indeed whether a weighted model (applying a constant value to one or several metrics) was more satisfactory. The next step would depend on your reasons for conducting the investigation.

Visitor performance map

Heads-up display of aggregate visitor satisfaction (performance versus user-defined thresholds) for satisfactory (green), tolerating (amber), and frustrating (red).

Global view (illustrated in Figure 7-21)—site performance by country. Such displays are particularly suited to the display of visitor-based passive (RUM) data. All major vendors now supply such a view as standard within their products.

Figure 7-21. Performance trend—24hr

Figure 7-22 shows performance over a rolling 24-hour period, illustrating the relationship between site traffic and performance. The tabular display below it shows metrics based on pages, originating ISP, visitor browser/version, and geographic location. It is color-coded to show the trend versus the previous day or week.

Figure 7-22. Visitor satisfaction

The stacked bar chart shown in Figure 7-23 displays the relationship between (proportion of) customer satisfaction (satisfactory/tolerating/frustrated—the APDEX adopted by many vendors in recent years) versus overall site traffic. Such a display is useful for highlighting traffic-related performance issues.

Figure 7-23. End-user visitor satisfaction

Gotchas!

Arguably, active (synthetic) external monitoring, to a greater extent than related disciplines such as application performance or passive (RUM) monitoring, is prone to errors of interpretation. These typically stem either from poor test design (not framing the question with sufficient rigor) or from failure to control the quality of the data obtained. However, as I have reiterated several times, clean-room testing, while necessary, is insufficient within a comprehensive performance management schema designed to maximize end-user satisfaction. Unfortunately, the more you try to replicate end-user conditions, the more dangers arise in capturing and reporting erroneous findings. The following list summarizes some of the areas for which it is important to check and control:

Target destination errors

You know what you entered as a test target. However, redirects, whether intentional or accidental, can lead to your testing something other than the intended page. Typical examples are the delivery of a PC site to a mobile device, or the delivery of a partial or error page instead of the complete version. Validate tests (particularly if abnormally fast or slow) by examining the size of the page downloaded and/or the number of objects. If available, consider using screen capture functionality to confirm.

Failure to discriminate between “fatal” errors

Fatal errors (i.e., those causing a page to fail, as opposed to individual content failures) are reflected as lack of availability in active testing. Always examine the point at which a page failed (i.e., how many objects successfully loaded). In addition to often providing insight into the nature of the root cause, such investigation will often indicate whether the error is of one consistent type or a mixture (although the latter may still have a common cause).

Latency

Ensure that aberrant results are not due to round-trip latency (e.g., from poor ISP node selection or inadvertent inclusion of end-user peers using distant proxy DNS servers).

Peerage

Group and compare results by ISP. Although there will always be some variation, systemic problems will be obvious if results over an extended period are compared. Filter and compare end-user results by tertiary ISP (and/or set up confirmatory tests) if in doubt.

Banding

Average traces (while they have their uses) can hide a multitude of detail. Always examine a scattergram trace of individual results to identify patterns of outliers or the delivery of banded clusters of results. Further examination at object level may provide further insight. Banding is commonly an indication of poor load-balancer configuration.

Browser version

Passive (RUM) data can be useful in identifying performance differences between browsers or versions—provided the browser concerned is recorded by your passive (RUM) tooling. Beware of reliance on active testing from a single browser agent without cross-validation.

Device and/or PC client limitations

Differences in processor speed, operating system, signal strength, and memory state (and more) can affect recorded performance.

Native versus third-party domains

Third-party affiliates can have significant and unpredictable effects on performance. Always examine at individual object level, plot suspect objects individually, and/or run confirmatory tests with object filtering.

Connection speed

Particularly when testing from end users (especially if using wireless carriers), you absolutely must filter out any very low connectivity tests (and/or clearly specify test conditions). Very low connectivity conditions are extremely distortive. They can lead you to draw incorrect inferences despite them being, in fact, a correct representation of performance in those conditions.