How many people do you want to survey? The simple answer is, of course, as many as possible. One commonly cited reason for surveying many people is to ensure that data is statistically significant—the more respondents, the more confidence you have in your results.
Remember, however, that your survey is already biased based on the people who responded to it. Much of what you collect won’t be applicable to your site’s visitors as a whole, anyway. You still need a large sample, but not because it will accurately model your entire market. The real reasons for wanting many responses are twofold: to validate patterns, and to segment.
You need to know which responses aren’t complete outliers, and for this, you need dozens, even hundreds, of answers. Many of the clustering and visualization tools on the market require a large number of inputs to function properly; if you only have a few responses, you may as well read responses by hand.
The more important reason for a large sample size is segmentation. Once you get the results of the original survey, you’ll probably have other questions. Imagine that you’re looking at visitor satisfaction with the site after a change. You notice that it has improved significantly overall, but there’s a large range in responses.
Should you be content with that data? Of course not. You should wonder whether there’s a hidden pattern to your responses—did men prefer the change, but not women? Did it work better for younger visitors, but not older ones? If you have a large enough sample of responses, you can segment the results in ways you didn’t foresee when you created the study.
We’ll look at analysis later in this chapter. For now, know that more results are better.
On the other hand, you have a limited number of visitors, and you don’t want to distract all of them with a questionnaire. This is the dilemma of VOC: how do you strike a balance between being receptive to feedback and not annoying your audience?
You must spread out your surveys with the use of a daily quota. The quota dictates how many surveys you’ll invite visitors to each day, up to your target number of surveys. It’s based on the estimated volume of visitors and the response rate to invitations.
Imagine, for example, that you want 20 responses a day and your site has 1,000 unique visitors a day. You also know that only 10 percent of visitors that you invite actually agree to take the survey. You therefore need to invite every fifth visitor to participate in the survey, as shown in Table 7-2.
Table 7-2. Estimating invite interval for an intercepted VOC survey
Estimated visitors | Response rate | Daily quota | Invite interval |
|---|---|---|---|
1,000 | 10% | 20 | 5 |
Some VOC services have sophisticated algorithms for managing quotas based on fluctuating traffic levels and varying response rates, deciding on the fly whether to intercept a visitor. Others set a daily invite interval and stick to it, leaving you to guess at traffic and conversion as best you can. And the simplest of systems store a percentage in the JavaScript for invitations, hoping that you’ll get enough responses.
If you’re using a recruited paid panel, your respondents may install software that will augment their responses, for example, by asking them questions as they surf the site. Some hosted services have access to panels of users willing to participate in research. This allows you to preselect respondents and target a particular demographic (such as males between 25 and 30 in the U.S.), but doesn’t tell you what your actual visitors’ motivations are. As a result, private panels are more useful for usability and navigation testing than for understanding actual visitor mindsets.
While each of your visitors has voice, you may not want to hear all of them. Should you be interested in the mindset of a particular segment, you’ll need to qualify who gets a survey, based on a particular demographic (age, gender, and so on), a respondent’s personal background (past use of this or other sites), or surfographic data (ownership of a particular product, comfort using the Web).
When possible, disqualify respondents based on technical information within the page rather than through questions. For example, if you only want to survey your “Gold” customers, put a cookie into the web session flagging those visitors as Gold status and modify the JavaScript that launches the invite so that it invites only Gold customers to take the survey.