The purpose of any testing is to protect users from a degraded or broken experience, and poor website performance is one of the quickest ways to give your users a degraded and broken experience. Therefore, performance testing, while not a test that points out system or visual regressions, is an important part of our testing arsenal.
Performance testing measures key metrics that affect a user’s ability to use your website, including page weight, number of requests, time to first byte (TTFB), load time, and scrolling performance.
The key to performance testing is to set a proper budget and stick to it. How you set the budget and stick to it will determine how effective the tests will be in your project.
Creating a performance budget means setting target values for each key metric and then continually testing those metrics before each code merge or deployment. If any of the tests fail, the new feature will need to be adjusted, or some other feature may need to be removed.
As with financial budgets, very few people are really excited about the prospect of performance budgets. To most, a budget means spending less, getting less, having less fun, and most importantly...less! Less isn’t much fun in a world where we are always being told that we deserve more. As designers, we feel that our creativity is being stifled if we can’t toss around hi-res images and full-screen video with reckless abandon. As developers, we think that we can’t do our job without a CSS framework, a couple JavaScript frameworks, and dozens of jQuery plug-ins. Less is no fun!
As a person that has been living within a financial budget for the past four years, I certainly understand what it means to not get everything I want. On the other hand, when I do make a large, budgeted purchase, I do so without a single bit of guilt or debt. In the same way, performance budgets allow us to “spend” our budget responsibly, and without regret.
Just like fiscal discipline and financial budgets, UX discipline and a performance budget can help us to achieve our ultimate goals, which include a performant website and an engaged user.
While a financial budget is typically based off one’s income, a performance budget has more to do with external factors than internal ones.
One method of determining your performance budget is to look at your competition. While saying “at least I’m better than so-and-so” is no excuse for a poorly performing website, it does ensure that you have a competitive advantage.
Start by looking at a few of your key competitors’ homepages and other key landing pages, and then compare load times, page weight, and other key metrics with your own website. The goal here isn’t to just match their metrics. You want to make sure you are beating them by 20% or more. So if your competitor’s product listing page loads in 3 seconds, make sure that your site loads its product listing page in 2.4 seconds or less. This 20% advantage over your competitor is the difference required for a user to recognize the difference between the two tasks.
This is not something you do just once, but something that needs to be monitored regularly. You can be assured that your competitors are looking for ways to improve and optimize their own sites. And if they had been looking at your site to determine their budgets, you’ve now pushed them to reduce their budgets as well!
Regardless of your competition, it is always important to compare your performance baselines to industry averages and general best practices. There is no reason to settle for mediocre just because your competition is throwing off the curve.
HTTPArchive is a great service that measures and records the average value of various website metrics across almost half a million websites. As of April 2015, here are a few values of note:
Therefore, if you want your website to feel faster than most websites, you might consider setting a goal of having a 1,648 KB website, that is served with 79 requests, of which 44 are cacheable. This will put you 20% ahead of the average website.
So now that we know a few methods for setting our budget, what are the budget items we need to consider when setting our tests up?
The most basic test of website performance is to look at the assets that are required to render it. How heavy are those assets, and how many of them are there?
Websites are getting fatter! Between April 2014 and April of 2015, the average website grew from 1,762 to 2,061 kilobytes, a 17% increase year over year. Reaching back to April of 2011, the average page was a skimpy 769 KB!
While page weight is not the only factor affecting the load time of your website, it certainly plays a large part. Page weight also has another side effect as we remember that more and more people are accessing our sites on mobile devices, and they are paying to download those bytes. The heavier your page, the more you are going to be costing your customers, especially in developing nations. Consider checking out What Does My Site Cost? to see what that new carousel is costing your mobile customers in Germany.
When looking to reduce the weight of your pages, there are a few obvious places to start:
<picture>
tag and srcset
attribute to download appropriately sized images.The browser is required to perform an HTTP request for every single file needed to fully render a page. Because each browser has a limited number of concurrent HTTP requests it can make, a large number of individual files means that the browser has to make numerous round trips to the server. The effect of these rounds trips is compounded on slower networks, so limiting the number of round trips needed to gather the required files will pay off greatly.
You can reduce the number of round trips in a few ways:
Regardless of the number and size of your site’s assets, there are a number of other timing metrics that impact a user’s perceptions of your site’s performance.
Hybrid metrics don’t measure a discrete value; they are scores based on numbers performance indicators.
PageSpeed is a website tool and Chrome extension made by Google that analyzes the performance and usability of a website, providing a score out of 100 and explaining ways that the user can improve that score. Tests include:
As stated on the project page, the speed index is the average time at which visible parts of the page are displayed. It is expressed in milliseconds and dependent on size of the view port.
This hybrid timing metric provides a score that takes into account many of the metrics just discussed, and combines them with a measurement of what the user is actually able to see of your site as it loads. Speed index is one of the best measurements of actual end-user experience.
Now that we know what types of metrics we can test and how to set performance budgets, let’s take a quick look at a few methods for automating the testing process. Whether you are testing a single website or dozens of them, no one wants to perform these measurements manually.
The first tool we’ll look at for automating this workflow is Grunt PageSpeed. As the name implies, this is a Grunt plug-in that allows us to run Google’s PageSpeed test on our website. So rather than plugging your URL into the test page or using a Chrome extension, you can run this Grunt task before every merge request or continuous integration build.
To set up Grunt PageSpeed, start with the standard commands to install and require our plug-in:
$
npm
install
grunt
-
pagespeed
--
save
-
dev
// Added to Gruntfile.js
grunt
.
loadNpmTasks
(
'grunt-pagespeed'
);
// Added to grunt.initConfig inside of Gruntfile.js
pagespeed
:
{
options
:
{
nokey
:
true
,
url
:
"http://redhat.com"
},
desktop
:
{
options
:
{
paths
:
[
"/en"
,
"/en/services"
],
locale
:
"en_US"
,
strategy
:
"desktop"
,
threshold
:
80
}
},
mobile
:
{
options
:
{
paths
:
[
"/en"
,
"/en/services"
],
locale
:
"en_US"
,
strategy
:
"mobile"
,
threshold
:
80
}
}
}
This code will allow us to automatically run both desktop and mobile tests on an array of pages inside of our base URL (in this case, http://www.redhat.com/). As long as our score comes back over 80, the tests will pass. If it is below 80, we’ll get a failing test, which signifies that changes need to be made to hit our threshold again.
Another great Grunt tool is Grunt Perfbudget. This Grunt plug-in taps into Marcel Duran’s WebPageTest API, allowing us to programmatically pull results from WebPageTest and compare them with our set budgets. If you haven’t used WebPageTest yet, you’ll be in for a treat. It is able to test numerous metrics for your site while simulating different types of connections and locations around the globe. I won’t get into everything the site can do, but after five minutes of viewing the results for your own site, I’m confident you’ll love the wide array of information it provides.
So let’s see what this looks like set up in Grunt:
You can currently get a limited-use API key at the WebPageTest website.
$ npm install grunt-perfbudget --save-dev
// Added to Gruntfile.js grunt.loadNpmTasks('grunt-perfbudget'),
perfbudget: { default: { options: { url: 'http://www.redhat.com/en', key: 'SEE_NOTE_ABOVE', budget: { visualComplete: '4000', SpeedIndex: '1500' } } } }
This setup allows us to automatically run the Red Hat homepage through the entire WebPageTest suite of tests, and check the returned values against the budgets I have set. In this case, I have set my Visually Complete timing metric to 4,000 milliseconds and the Speed Index to 1,500. If either of those tests comes back above our budget, we get a big error message telling us to revisit the most recent code push and see what we did to break our budget.
3.149.249.174