Obtaining the tools you need to run split tests

To run split tests, you need effective tools. This section tells you about the technology you need to run split tests so that you can optimize your campaign for maximum results.

Test duration calculator

A test duration calculator is a simple calculator that determines how long you need to run your split test to get a reliable test result. You input data such as the existing conversion rate, the number of variations in the test, the amount of traffic your site gets, and more. The calculator then determines how many days to run this test to get a reliable result. Figure 13-1 shows the free test duration calculator offered by Visual Website Optimizer.

Not every page on your website needs to be tested or requires optimization. In the next section, we discuss the ways for you to isolate the pages to test so you can maximize your return on investment (ROI).

Following the split test guidelines

When you’re looking for pages to split test, use the following guidelines to determine how worthy a page is to test. First, here’s what not to test:

• Your worst-performing pages (this sounds counter intuitive, but we explain why).
• Pages that don’t impact your longer-term business goals, for example, your 404 page.
• Pages that don’t get enough traffic to run a split test.

So why shouldn’t you test your worst-performing pages? When looking for pages to optimize, your job is to focus on opportunity pages, which are pages that will have the greatest impact on your goals. For instance, if you expect a 10 percent increase in conversions from your efforts, would you rather that lift be on a page converting at 50 percent or 5 percent? The one at 50 percent is an opportunity page.

Further, your worst-performing pages don’t need a testing campaign; rather, they need an overhaul. The ship is sinking, and you don’t have time to hypothesize over what to do next; you need to make a drastic change that likely doesn’t need to be tested. Remember, in such a case, don’t test; implement!

For the same reason that you don’t want to test the worst-performing pages, you also don’t need to test your nonconversion-oriented pages. These nonconverting pages include your About Us page or your “dead end” 404 page.

However, optimizing 404 pages has proven to be useful in marketing. Even on that page, you should include an offer, a call to action, or some additional steps to keep the user engaged. You don’t need to test adding these elements to the page, however; just add content that meets your goals and then move on to more important pages that impact conversion. Amazon’s 404 page, shown in Figure 13-2, directs people to the Amazon home page or suggests continuing to search.

The final guideline you should follow when determining whether to split test a page is the page’s traffic. Look at the number of visits and of conversions that your page gets over the potential test period. Notice where traffic falls off considerably.

You can easily identify your pages and their traffic numbers using Google Analytics. Examine the number of Unique Pageviews for pages under consideration for split testing. The best report to employ for this job in Google Analytics is in the Behavior suite, which we detail in Chapter 12. In Google Analytics, navigate to the Reporting section and then select the following: Behavior  ⇒      Site Content  ⇒      All Pages. The All Pages report loads. From there, use the filter tool in Google Analytics to search for the specific pages you’re considering for a split test. Figure 13-3 shows an example of what an All Pages Report looks like from DigitalMarketer.

After you gather the data from the All Pages report, you should contextualize the pages. You’ll always see a massive drop-off in page views (the total number of pages viewed by a user; repeated views of a single page are counted) after your home page. However, your home page is so far away from your main converting action that it doesn’t make sense to test. Now, if you see a massive drop from a product page to the checkout page, you know that something is wrong with your product page and that you need to optimize it, and that merits a split test.

By following the guidelines in this section, you can hone in on pages worthy of your time and resources for testing. When you find a page that you consider to be test worthy based on the guidelines, make sure to ask these four questions:

• Does the page get enough unique visitors?
• Does the page get enough raw conversions?
• Does this page directly impact my goals? If indirect, how far away from the primary conversion action is the page?
• What’s the potential impact on your goal, such as for sales or leads?

Answering these four questions before you commit to testing a page accomplishes the following:

• Qualifies that the page is worth using resources to test.
• Gives you an idea of whether testing will actually be useful.

By determining which pages aren’t worth split testing, you can find the pages that merit testing.

Considering qualitative data

Qualitative data is information that people can observe but not measure. In terms of digital marketing, qualitative data considers the users’ behavior. Gathering qualitative data is relatively easy and inexpensive, and it’s extremely helpful for picking the right elements to optimize on your page.

One of the most basic types of qualitative data involves click tracking, mouse movement, and scrolling. Much of this data gets reported in what is referred to as a heat map. A heat map is a visual representation of a user interacting on your site; it reveals where users focus on your site. Figure 13-4 shows a typical heat map.

Running a heat map on any page you’re split testing is a good idea. Most good testing technology tools, such as Visual Website Optimizer, include heat map technology. Heat and scroll map reports can shed light on whether a call to action (CTA) is getting clicks, or whether people are consuming your content.

Here are other types of qualitative data and how to collect it:

• User surveys: Use a tool such as TruConversion (https://www.truconversion.com/) to survey your site visitors and get qualitative data to analyze.
• Session recordings: Also use TruConversion to record visitor sessions and analyze those recording to find elements to test on the page.
• Customer service questions: Any team members who talk to customers are often a treasure trove of qualitative data.

Using qualitative tools

Qualitative data is incredibly important and severely underused. Some great tools are available, so start with one and move on to others when you run into user knowledge gaps. These tools can include:

• TruConversion (https://www.truconversion.com/): This suite of tools has heat maps, session recordings, user surveys, funnel analysis, and form-field analysis.
• Crazy Egg (https://www.crazyegg.com/): Focuses primarily on heat maps, tracking clicks, mouse movement, and scrolling.
• UsabilityHub (https://usabilityhub.com/): This site helps test design and uncover design issues with user tests such as:
  • Preference tests
  • Five-second tests
  • Click tests
  • Design surveys

UsabilityHub’s five-second test, and ones like it, are extremely powerful. The five-second test measures people’s first impressions of your site. If people can’t figure out who you are and what they’re supposed to do on your site within five seconds, you need to rethink your page.

Developing an optimization hypothesis

Your test needs a hypothesis. For your test to truly be meaningful and actionable, you need to come up with a plan, and you need to document statistics. Testing for the sake of testing or for a particular hunch only wastes your business’s time and resources. A clear hypothesis puts a stop to ad hoc testing.

Create a hypothesis based on this format:

Because we observed [A] and feedback [B], we believe that changing [C] for visitors [D] will make [E] happen. We’ll know this when we see [F] and obtain [G].

Following a basic hypothesis format like the preceding one sets your test’s scope, the segment, and the success criteria. Without a hypothesis, you’re guessing, and you don’t want to base a campaign’s success or failure on a guess.

Choosing the metrics to track

After you choose a page to split test and the variations you will be testing on the page, you need to determine the key performance indicators (KPIs) that you will use to evaluate your split test. KPIs are metrics that gauge crucial factors and help you to determine the success of a test. For instance, if you run a test that looks only at top funnel metrics, such as clicks, you don’t get a full understanding of the actual impact. For this reason, you need to select your KPIs and know how they impact your business goals.

To help define your KPIs, make sure to have page-level goals as well as campaign-level goals for all your tests. Your split test goals might look like this:

• Page goal: Leads generated
• Campaign goal: Specific product purchased

Page and campaign goals give you the short view, that is, what happened on the page; and the long view, that is, how what happened on the page impacted your overall campaign. It is possible to see an improvement in the performance at the page level while experiencing a decrease in performance at the campaign level. In our preceding example, we may run a test that generates more leads at the page level but actually decreases the number of products purchased at the campaign level.

Keep a common thread throughout your KPIs and have them measure related metrics, such as add-to-cart rate and sales.

Calculating your test timeline

Every test needs a definitive stopping point. If you test into perpetuity, you ignore the possibility that no change occurs between variants. You need to create a clearly defined test time period before you start testing, and then stick to that time table.

Use your duration calculator, mentioned in the “Obtaining the tools you need to run split tests” section, earlier in this chapter, and round up to the next week. For example, if your duration calculator says that you would have meaningful results in ten days, run the test for fourteen. People behave differently on different days, and you must account for this variance in behavior. This little trick will help you gather more complete data.

Defining goals in Google Analytics

Just having Google Analytics on your site isn’t enough; you need to establish your goals. (See Chapter 12 for how to set up goals in Google Analytics.) Setting custom events or e-commerce tracking works as well — you just need something to measure.

Having a measureable goal is important because when you have proper e-commerce or goal reporting in Google Analytics, the results of your testing are determined by objective numbers rather than subjective opinion. Having goals set up in Google Analytics is incredibly powerful and will start to show you the efficacy of your campaigns in a single platform.

Checking that your page renders correctly in all browsers

If a page isn’t performing properly, it will corrupt the data. You may think that the variation you are testing has failed because your hypothesis was incorrect, but in truth, it might be a tech issue. For instance, if one of the pages you are testing is showing a broken image, the conversion lift (or failure) for that page is not caused by the changed variable but rather by the page’s functionality, in which case your test will be for naught. Before you launch your test, double-check your page for bugs by using tools such as BrowserStack or preview options in Visual Website Optimizer.

Ensuring that you have no testing conflicts

You don’t want your tests to overlap. Therefore, you should never run multiple tests on the same page at the same time; for instance, running a second, separate test on a page while another test is already being performed on the same page results in conflicting data.

You can run tests on different pages at the same time. However, when running tests on different pages at the same time, you need to make sure that traffic included in one test isn’t included in the other.

Checking links

Just as you need to ensure that your page is functioning, you also have to make sure that your links actually work and go to the right page. A split test between a page with links and a page without properly functioning links is obviously a fatally flawed test that won’t give you true results.

Keeping variation load times similar or identical

Keep your load time in mind when you optimize. If you have a variant with a better load time, that variant will likely beat out its competition, skewing your results. Use tools such as PageSpeed (https://developers.google.com/speed) to analyze and ensure that your variant load times are as close as they can be.