Good design is all about really listening to your users. Your users can tell you what’s wrong with your site, what’s right with your site, how you can fix things (if necessary) and how you can improve your site. There are lots of ways you can listen to your users. You can listen to them in groups (using tools like surveys), listen to them individually (with tools such as usability tests), and listen to their collective actions on your site (with tools such as site metrics and statistics). Whatever method you use, its all about “hearing” what your users are saying. If you do, your site will meet the needs of your audience and be that much better for it.
When you’re designing, redesigning, or fixing a site, you need to be able to ask your audience questions. Who are they? How often do they use the Web? What kind of computer do they use? What do they like about the site’s current design? Do they like the site’s proposed new design?
Note
Remember, as we talked about in Chapter 4, surveys are a great tool for building personas at the beginning of your design process.
This is where focus groups and surveys come into the picture. Both are great tools for giving your audience a “voice”—letting them give their opinions, answer your questions, and even give feedback in unexpected ways. It’s important to know that surveys and focus groups have different strengths and weaknesses, and those could impact which you use. Let’s take a look at some pros and cons of both:
PROS | CONS | |
|---|---|---|
Surveys | Require a relatively short time commitment from survey participants | Require a lot of advanced planning |
Can be administered to a huge amount of people | Written surveys may present problems for lower-level readers | |
Can be administered in lots of ways (paper, online, by phone, in person) | Survey questions might be misinterpreted | |
Participants can often complete the survey at their own leisure | ||
Focus Groups | Allow participants to build on each other’s ideas | Require larger time commitment from participants |
Collects information on a very specific topic from those who have a stake in the topic | Usually require compensation for the participants | |
Benefits from a trained facilitator | Public environment may intimidate some participants | |
Requires a trained facilitator |
Sharpen Your Pencil Solution
Let’s take a look at whether a survey or focus group is the best choice for each of the scenarios. How will you implement your choice?
Scenario 1
You’ve been hired to design a website for a local community access television station. The client wants to add all sorts of cool—but bandwidth intensive—Flash interactivity and video. You’re worried that this extra media might be too much for the site’s users.
You need to find out what kind of computers the TV station’s viewers use (make, model, operating system, etc.), as well as their connection speed (dial-up, DSL, Cable, etc.) so that when they come to use the site, it doesn’t fall over under the pressure of serving all that rich media content.
Scenario 2
You’re doing some volunteer design work for a local inner-city community center. You really don’t know that much about the site’s users—gender, age, computer types, etc.—and you really need to find out so that you can target the design at the right level. The director of the community center did tell you that some of your users might have a low literacy rate.
Scenario 3
You and some friends are launching a new social network for web designers and developers. You’re at the point where you’ve got a couple of different visual designs (really polished storyboards), and you need to get some specific feedback on what your potential audience thinks of the site’s design. You’re also pretty sure that your audience (who are designers themselves) will probably be up for sharing some their great design ideas with you.
When you’re deciding whether to use surveys or focus groups to find out about your users, there’s another thing to consider: cost. If you start out doing user testing with the comparative costs of surveys and focus groups in mind, that may help you decide which is the right route.
Time spent planning the focus group schedule | |
Recruiting participants | |
Rent space to hold focus group (if you don’t already have it) | |
Equipment rental (projector, screens, recording devices, etc.) | |
Food and/or compensation for participants | |
Time spent processing and evaluating the results | |
Any additional supplies | |
Cost of a trained moderator |
Sharpen Your Pencil Solution
Which method for gathering feedback from the RPM Music users is probably the best, and why? Here’s what we thought:
Surveys are cheap and effective, especially when you give participants the option of completing them online.
So you’re going with a survey, but there’s a problem. What questions are you going to ask, and how will you ask them? Oftentimes online surveys either ask the wrong questions or ask questions that return unreliable data. You really need to invest some time in crafting your survey questions so that you get the best possible data you can. Let’s take a look at what makes a good survey question.
Note
If you confuse your users, they aren’t going to provide you with accurate information. Pay close attention to how you word the questions.
Exercise Solution
Let’s take a look at the survey questions and see what we can do to make them better.
Which is your browser of choice? | |
A. Internet Explorer 6 | |
B. Internet Explorer 7 | |
C. Internet Explorer 8 |
Note
The problem with this question is that it’s biased. Biased questions encourage your participants to respond to the question in a certain way. They may contain terms or may be worded in a way that leads the participant to a specific answer. Here all of the associated options are Internet Explorer (though different versions). What if the respondent uses a different browser? Well, their only option would to choose “Other,” if it were available as option D. That would ultimately skew the results (because you won’t get any differentiation between browsers besides Internet Explorer). This question may also mislead participants into choosing Internet Explorer just because they have it in their system (as opposed to being their primary browser).
A better browser question might be...
Which is your browser of choice? | |
A. Internet Explorer | |
B. Firefox | |
C. Safari | |
D. Mozilla | |
E. Opera | |
F. Other |
Note
Although this question if focused and simple, it falls into a common trap. The terms “CMS” and “W3C-compliant” are only used by a small group of people within web technology circles. Even though some users may recognize the terms, most won’t and you risk confusing and frustrating your survey participants. Avoid using jargon in your questions and always try to write to the broadest audience possible.
Less jargon = better understanding = accurate responses
Exercise Solution
Let’s take a look at the survey questions and see what we can do to make them better.
There are many people who believe that designing with web standards is a waste of time–are you one of these people? | |
A. Yes | |
B. No |
Note
This question’s problem is that it assumes what it asks. This is a type of biased question that leads your participants to agree or respond in a certain way.
Let’s not assume...
Do you agree of disagree that designing with web standards is a waste of time? | |
A. Agree | |
B. Disagree | |
C. Undecided |
Do you believe that the current single column layout website in question is bad and should be changed to a 2 column layout? | |
A. Yes | |
B. No |
Note
The problem with this question is that it’s two questions. Participants won’t know which to answer. What if they agree with one but not both, or disagree with part or all of the question... How do they answer then? This can make them feel frustrated and could lead them to quit the survey without finishing it.
Just ask one question
Do you like or dislike the current one-column layout in the website in question? | |
A. Like | |
B. Dislike | |
C. Undecided |
Sharpen Your Pencil
It’s time to start building your user survey to help RPM Music. Write five technical questions to start with (what kind of browser users prefer, what kind of computer they have, etc.), then ask five demographic/usage questions (gender, age, how many hours/week do they use the web, etc.). Make sure you write questions that have answer choices, are short and straight to the point, and don’t lead the respondent.
Technical survey questions:
Demographic/usage questions:
Here’s the completed survey ready to upload to the site (or use a survey service to conduct it for us).
“I love the new site. It really reflects the vibe of the store. It would be really nice if I could buy digital tracks—ya know, like iTunes—instead of just getting the CD shipped to me.”
“I don’t really use the site that much. I’d rather just come into the store. I really like talking with the people who work there as well as other customers. I guess the site is ok—it looks good—I just won’t use it that much.”
“The new site looks all crazy now. Parts of the pages don’t look like they’re in the right place, and things don’t make sense. Can you fix this? I want to keep buying stuff from you guys, but it’s a little difficult with a messy site.”
“When I went to look at the site in Explorer, it looked all weird. The interface looks all out of whack... there are things that probably shouldn’t be where they are.”
Brain Power
Now that the results from the survey are in, start thinking about how all the information relates to each other. Finding patterns and connections can help shed some light on why RPM’s users are having trouble with the site.
Frank: Not necessarily. Browsers are the lens that you see the Web through–and not all browsers are created equally.
Jim: Oh, is one “brand” better than another? Is that why a lot of the RPM users seem to be using Internet Explorer?
Frank: No, actually. What I mean is that some of the browsers have peculiarities (in some cases, you might even call them bugs) that result in pages looking slightly different on one browser than they do on another.
Jim: Oh, I see. And since we want all of our users to have the same experience regardless of what browser they have, we need to take the peculiarities of each browser into account when we’re designing the site...
Frank: Exactly. This is where web standards come in (sort of).
Jim: How so?
Frank: Web standards are part of the big cross-browser compatibility picture. Because the W3C works collaboratively with browser developers (among other people and companies), web standards are (mostly) baked right into the browsers themselves. So when you design to web standards, you can be fairly confident that your site will look the same browser to browser. But some browsers have peculiarities—
Jim: Bugs.
Frank: Ok, bugs—that won’t display standards based XHTML and CSS the same way.
Jim: Oh, I see. So what browser do we design for?
Frank: Well, first off, you should always use standards compliant CSS and XHTML. That’ll solve a bunch of your problems right away. You should be intimately familiar with your users. What browsers do they use?
Jim: We know that now. Most of them use IE and Firefox.
Frank: Exactly. Design for those two browsers in particular. Finally, there are some code-based workarounds—you may have heard of them already, they’re also known as CSS “hacks”—that will help you bypass cross-browser computability problems (just like the ones the RPM Music site is experiencing). So we need to go look up some stats...
IE 6 doesn’t handle the :hover pseudo-property very well (actually, at all). We need to move our “active” class to the link tag instead of the list item and update our CSS file to apply the background to the correct element.
Exercise
Fix the RPM site by moving the “active” class from the <li> tag to the <a> tag in the XHTML markup. Make sure you also update the CSS file to reflect your changes.
Open up the index.html file and move the “active” class in the navigation to the link tag.
<li class="active"><a title="RPM Music home" href="#">Home</a></li> <li><a title="Music Store" href="#">Music</a></li> <li><a title="RPM Blog" href="#">Blog</a></li> <li><a title="More RPM Music" href="#">Etc</a></li> <li><a title="Shopping Cart" href="#">Cart</a></li>Open rpm.css and change the location of the :hover pseudo-property to the correct element.
#nav ul li.active, #nav ul li:hover { background: #333; }Launch the site in a browser and make sure everything looks as it should. If you’re running Internet Explorer 6, you should see that the hover now appears properly.
User Testing: Let your users tell you how they use your site.
If you want to know how your audience is actually using your site, you need to do some user testing. User testing, also known as usability testing, lets you evaluate a website by testing it out on actual users.
Note
You don’t have to wait until you have a finished product to do usability testing - you can do it at any time during the design process.
You’ll see where users go, which pages they spend the most time on, where they get confused, how they move from point A to point B on the site, and a lot more. User testing can identify known problems, locate unknown problems, and pinpoint usage patterns that could inform a redesign.
But you don’t just have to watch users interacting with your site, you can give them tasks and evaluate how they accomplish each task. Always remember it’s the site you’re evaluating, not the user. This is important because you don’t want to give your participants (who are taking time out of their schedule to help you) the perception that you’re testing them.
Frank: Good idea, Joe. We really should have baked usability testing right into our design process from the get go, but there’s no reason we can’t test the site after the fact. Usability testing’s a great way to figure it out a design problem because you’re looking at how actual users are using your site.
Jim: Okay, I hate to rain on your parade, but testing sounds expensive...
Joe: I have a friend who works in professional user testing. It’s big business, and her company doesn’t just do websites—they user test all kinds of products. But professional companies like that cost tens, even hundreds, of thousands of dollars.
Jim: See that’s what I was afraid of. Why so much?
Joe: They often have dedicated usability labs with computers that capture every keystroke and mouse movement, microphones that record every user comment, cameras that record the entire test, and even eye tracking devices that record where the user’s eye was focused at any given time. Plus, they’ve also got a horde of experts who can process all of the data from the tests and provide comprehensive reports and recommendations.
Frank: That’s a bit out of our budget. Any chance your friend shared a few of her secrets, Joe?!
Joe: Actually, she didn’t need to. Just because you don’t have a fancy lab doesn’t mean you can’t do useful usability testing. There are lots of budget-minded techniques we can use to test how real people use the RPM website.
Frank: So she did share some secrets after all?
Joe: Nope. All you really need to do your own usability testing is a solid plan, a computer, a few people to run the test and some willing participants. There’s even been some pretty impressive and inexpensive software—like Silverback, http://silverbackapp.com—that’s been coming onto the market recently. These applications can help you do some pretty sophisticated user testing without a crazy expensive lab.
Frank: Great, then let’s get to it. RPM’s users are waiting...
BE the user
Your job is to play the user and perform a task that would typically appear on a standard usability test. Follow the instructions below and write your impressions of the process in the space below.
Task #1: Using www.expedia.com, what would be the least expensive business class ticket price for a single adult flying from Detroit (MI) to Houston (TX), from any airport, departing on November 1st in the morning and returning November 4th in the evening? |
So what exactly do you need to do a decent usability test on a budget? Here are the basics:
Plan: You’ve got to have a plan. What are your goals (what do you want to accomplish with the usability test)? How are you going to accomplish those goals? What are your tasks going to be? Who are you going to recruit to be your participants? If you don’t have a solid plan, your usability test will be a mess from the get go. | |
Moderator Script: A moderator script is basically the script for the entire usability test. How it will run, what the tasks are, and what needs to be told to the participants. | |
Moderator: The moderator is the person who runs the usability test. They talk to the participant, tell them what they need to do, and give them clarification if they need it. | |
Note Taker: The note taker is the person who records what the user does, what they say, etc. They basically collect the data that will be analyzed after the test is finished. | |
Computer: You’ll need a computer with an internet connection and necessary software for the test (browsers, plugins, etc.). | |
Space: You need somewhere to run the test. If you don’t have a lab, this could be your office, a quite coffee shop or even the corner of a local library. | |
Participants: You need someone to actually participate in the test. The more people the better. These participants should be drawn from your audience. It won’t do you any good to select participants who would never use your site. You also need to compensate participants—give them something for their time. This may be cash ($50 is not uncommon for an approximately 3 hour session) or a gift certificate from somewhere (everyone loves Amazon). |
If you want to get the best results from your user test, you need to make sure you’re well organized and that the test runs smoothly. That means knowing the exact sequence in which everything happens, what you’re going to say to the participants, and what tasks you’re going to have them perform—and that’s what’s in a moderator script.
Study Overview
When you do the usability test, you’ll actually read this portion to your participants
Thank you for agreeing to participate in our study. We’re very interested in your feedback about booking air travel online. We’re trying to understand how people interact with multiple air travel websites during typical scenarios.
In this session, we’ll first discuss why and how you use the Internet in your daily life and how you might use air travel websites in particular. Then I’ll ask you to complete one task scenario on two air travel websites, dealing with looking up airfare prices for a specific destination.
As you’re completing the scenarios, please speak aloud to let me know what you are doing. I’ll also ask you to point out anything you encounter that’s unexpected or surprising. When you’ve completed the scenarios, I’ll ask you a series of questions about the websites and how you interacted with them. Then, I’ll ask you some questions after you have filled out a brief feedback questionnaire.
Remember that this is an evaluation of the website’s ease of use and not of your individual performance. Do you have any questions at this point?
Background Interview
A background interview provides information about the participant’s general internet usage as well as their experience with the kind of site that’s being tested. So, questions would include:
What time(s) of day do you tend to use the Internet?
What types of work or school-related activities do you perform using the Internet?
What are the main issues you have when you visit websites in general?
What is a scenario in which you might use a website like the one being tested?
Task Instructions
These are the tasks that the user will perform, which you will observe them complete and record what they do. The tasks you come up with are the “test” portion of a usability test.
Sharpen Your Pencil Solution
Let’s take a look at some possible tasks ( that a typical music site user might do), which you can use for your RPM Music usability test.
Note
These questions cover the participant’s experience during the test and their opinions about the site in question.
Post-study Questions
Post-study questions are designed to get the participants overall impressions of the site (or sites) being tested. Questions could include the following:
To what extent did the content of the website meet your expectations?
Not at all
A Little
Neutral
Somewhat met
Very much so
Overall, how easy was it to understand the organization of the website’s screens, especially the menu levels and the flow of the screens?
Very difficult
Difficult
Neutral
Easy
Very Easy
Moderator Script Construction Solution
Let’s take a look at what your RPM Music usability test moderator script might look like.
Exercise
Remember the Expedia task you did earlier in the chapter? Now it’s your turn to have a friend or family member perform the task, so you can get a taste of what it’s like to be the moderator. Make sure you have them speak out loud and record your observations below.
Task #1: Using www.expedia.com, what would be the least expensive business class ticket price for a single adult flying from Detroit (MI) to Houston (TX), from any airport, departing on November 1st in the morning and returning November 4th in the evening? |
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
When you’re doing usability testing on a small budget, you’re probably thinking that it would be easy to grab some friends or family and have them do the test. And you might also be thinking that you could persuade them to skip the compensation (because they are friends and family, y’know). The problem is that when you use friends and family for a usability test, you introduce bias into your study.
Because they’re close to you, they might not be willing (either consciously or unconsciously) to give you unbiased feedback. Also, when you’re choosing your participants, you need to choose people who would actually use the site. It’s more than likely that the people close to you aren’t part of the intended audience of the site, and, therefore, if you choose them, you may not get the results you need.
Looks like our users are a little confused by the navigation menu. They’re having trouble finding the contact information because it’s buried below a nav item that’s incredibly confusing. This should be an easy fix...
Site statistics are a handy way to find out more about what your users are doing—which pages they’re hitting (and how often), which are their favorite pages, where they’re coming from (referrers), and what searches they might be running while they’re on the site.
You can get at your site statistics by using an application (either on your server or on another server) that captures, measures, and reports all sorts of information on your site’s traffic. This process is often also called website analytics. Whatever name you use, website statistics or analytics, it’s a great way to get information about how your site’s being used by your audience. Let’s see how RPM’s stats are looking.
Mint (http://haveamint.com/) is a cool little application created by the legendary web designer/developer Shaun Inman (www.shauninman.com/) that sits on your web server and captures traffic data—which is then displayed in Mint’s customizable dashboard. The great thing about Mint is it can be extended by plugins called Peppers. These Pepper plugins and widgets (which are available at the Pepper Mill http://haveamint.com/peppermill/) are developed by all sorts of people all over the web and will work seamlessly with your Mint install.
Google Analytics (http://analytics.google.com) is a free service offered by Google that generates detailed statistics about the visitors to a website. Although Google Analytics can provide detailed information about page views, referrers, browser versions, etc., it’s geared more towards marketers than designers or developers.
Google Analytics works based on a system called “page tagging,” in which a little bit of Javascript is inserted into every page on the site. Every time that page loads, the JavaScript collects anonymous visitor data, sends it back to the Google mothership for processing, and then displays it in a handy dandy dashboard format (at the Google Analytics site).
Sharpen Your Pencil
Take a look at the analytics dashboard and see if you can identify possible problems with the RPM site. Write down your top three observations below.
__________________________________________________________
__________________________________________________________
__________________________________________________________
You’ve got Chapter 9 under your belt, and now you’ve added usability testing to your tool box. Usable sites down; next up: evolving your sites to the next level.