Card Sorting
Introduction
Card sorts show how people think content should be organized and named. Card sorts are often used to generate an information architecture. Information architecture refers to the organization of a product’s structure and content, the labeling and categorizing of information, and the design of navigation and search systems. Information architecture can refer to software or websites or physical organization (e.g., controls on the dashboard of a car, the fruit stand at your local farmers’ market). A good architecture helps users find information or items and accomplish their tasks with ease. Card sorting is one method that can help you understand how users think the information and navigation should be within your product. Information architecture is “all about three things: (1) organizing content or objects, (2) describing them clearly, and (3) providing ways for people to get to them” (Spencer, 2010, p. 4).
This method involves writing objects that are in—or proposed to be in—your product (e.g., hotel reservation, rental car agreement) on cards and asking users to sort the cards into groups that make sense to them. The objects are pieces of information or tasks that are—or will be—in your product. You want to understand how the users think those objects should be organized. There are no right or wrong groups; rather, a card sort helps you elicit from your participants the groups that already exist in their minds. You then strive to replicate these groupings in your product. By doing so, users will be able to easily find what they are looking for when using your product.
In this chapter, we discuss the uses of card sorting, how to prepare for and conduct a card sort, and how to analyze and present the information. Finally, a case study by Jenny Shirey is offered so that you may see an example of a successful card sort.
Things to Be Aware of When Conducting a Card Sort
There are a number of questions that you need to answer before starting your card sort. For example, will you use an open or a closed card sort procedure? Do you want participants to sort physical cards or use a computerized card sort program? If you are conducting a computerized card sort, will participants be remote (i.e., participating online from a location of their choice) or in person (i.e., in whatever location you choose to meet them, for example, your company’s user research facility)? If you are conducting an in-person sort, will you conduct an individual (only one participant at a time) or simultaneous (more than one participant at a time) card sort exercise?
Open vs. Closed Sort
An open card sort is one where participants are allowed to generate as many categories of information as they want and name each of those categories however they please. A closed card sort is one where participants are given a set of cards and a set of predetermined categories and asked to place the cards into those preexisting categories. The two primary benefits of an open card sort are: (1) Participants have more flexibility to express how items are grouped in their minds, and therefore the outcome of the study may more accurately reflect an intuitive grouping; and (2) participants are asked to provide names for the groups they create, and therefore you get additional data about participants’ vernacular. For these reasons, open sorts can be particularly useful early in the research process (e.g., at the concept stage, page 308). On the other hand, a closed card sort may be more appropriate when trying to improve the information architecture of an existing product. Open card sorts take more time to conduct and analyze because participants have more work to do (e.g., generate category names) and because the analysis can require more steps than a closed sort.
Physical Cards or Computerized Sort?
There are tools available that allow users to sort virtual cards on a computer rather than using physical cards. Computerized card sorting can save you time during the data analysis phase because the sorts are automatically saved in an immediately usable format. Another advantage is that, depending on the number of cards, users may be able to see all of the cards available for sorting at the same time. Unless you have a very large work surface for users to spread their physical cards on, this may not be possible for physical card sorts. Computerized sorting also has its disadvantages. First, you may need to provide a brief training session to explain how to use the software, and even with training, the user interface may be difficult for novice users to get the hang of. Second, if you run a simultaneous or group session, you will need a separate computer for each participant. This means money and potential technical issues.
Remote (Online) or In-Person Sort?
If you choose to conduct a computerized sort, some tools support remote online testing (see Remote card sort tools, page 317). Remote testing means that participants can participate in the card sort using an online tool or via a website that they access from their own computer. The advantage of this is that it allows you to gather data from users anywhere, without you having to travel. This means that you may be better able to get a geographically diverse population of participants. Furthermore, because the tests can be conducted simultaneously and do not require your active involvement, you can test a large number of participants in a short amount of time. However, the downside is that users may have a more difficult time without a facilitator in the room to answer questions, and therefore the quality of your data may suffer. You will also be less able to easily to capture think-aloud data using a remote sort.
Individual or Simultaneous Card Sort?
If you choose to conduct an in-person sort, you need to decide whether to conduct your card sort with several participants at once or one at a time. We often conduct sessions with several participants simultaneously because this allows us to collect large samples of data in a shorter time period. You can conduct an in-person card sort with as many people at a time as you can physically accommodate in your space. Even though we have a group of participants in the same room at the same time, they are not working together—they are each working individually.
The disadvantage with running several participants simultaneously is that you cannot collect think-aloud data (see Chapter 7, “Using a Think-Aloud Protocol” section, page 169), so you do not know why the users grouped the data the way they did. Although think-aloud data are helpful, participants typically provide enough information in their description of each group so the need to collect data quickly and from large samples may outweigh the benefit of having think-aloud data.
Some people dislike running a group card sort because they feel that the participants turn it into a race. In our experience, this has not been a problem. We encourage people to take their time because we will be there for as long as they need to sort the cards.
If you have the time, a hybrid approach works quite well: after collecting data from a group of participants, run one or two individual card sorts to collect think-aloud data. This additional data can help you better understand the groupings.
Preparing to Conduct a Card Sort
Now that we have presented when and why to conduct a card sort, we will discuss how to prepare for one.
Identify or Create Objects and Definitions for Sorting
There are several ways to obtain your objects (i.e., pieces of information or tasks) and definitions for sorting. The way you choose will depend on whether you have an existing product or your product is still in the conceptual stage.
Existing Product
If a version of the product already exists and you work for an organization that has an information architect or content strategist, consult him or her to see if there is an existing content inventory. If your goal is to re-architect an existing product that does not have an existing content inventory, you and the team can together identify the possible areas to re-architect. Once you have done this, you can make a list of all the objects contained within these areas. If there are objects that will be omitted in the next release, you should omit these from the card sort. Conversely, if there are new objects that the product team intends to add to the product, you should certainly include these.
The most frequent method is to work with the development team to identify the objects and then develop clear definitions. The creation of definitions can be surprisingly time-consuming since the development team may define things in terms of the way the back-end or technical components of the product works. It is your job to make sure the definitions are clear and easy for participants to understand. Without those definitions, you cannot be sure that you and the participants are on the same page, speaking the same language.
Concept
In cases when your product is still in the conceptual stage, you may not have determined a list of content or tasks for the product. While still working with the development team, you may need to supplement your knowledge with input from the marketing department or a competitive analysis (see Chapter 2, “Learn About Your Product” section, page 25). You may find it beneficial to do an interview or survey to learn about the information or tasks users would like to have in your product and the words they use to describe this information. You will need to ensure that you clearly understand what each idea means so that you can write complete definitions for your card sort.
Free-listing
Finally, you can also obtain objects for a card sort by asking participants to free-list all the items associated with a given domain (i.e., participants write down every phrase or word associated with a particular topic, domain, etc.). In free-listing, participants are asked to name every “item” they can think of that is associated with a domain—not just the ones they want for a given product or system. Using our travel example, we might want to ask participants to name every piece of information they can think of that is associated with making travel reservations. Some responses might be the following: plane ticket, car rental, hotel room, confirmation number, and frequent-flyer miles. The biggest benefit of free-listing is that you obtain information about the users’ terminology because they are offering their ideas in their own language.
How Many Cards?
We have found that it is best to limit the number of objects to be sorted at 90 or less. However, there are published studies where researchers have successfully used more than 90 cards. One study we found used 500 cards (Tullis, 1985)! We would not recommend this unless you are an expert at conducting card sorts and have a good reason for doing so. Keep in mind that the more cards there are, the longer it will take for the participants to sort, and therefore you run the risk of fatiguing and overwhelming them. In addition, sorts with large numbers of cards can take considerably longer to analyze.
If possible, run a pilot session for a card sort before you have finalized your protocol. This will help you find typos or identify confusing definitions and terms. In addition, a pilot can help you get a sense of how long it will take for participants to complete the sort and determine whether you missed any objects.
Activity Materials
You will need the following materials for an in-person card sort:
■ 3 × 5 in index cards (different-colored cards are helpful)
■ Printer labels (optional)
■ Stapler
■ Rubber bands
■ Envelopes
■ Plenty of workspace for a participant to spread out the cards
And the following materials for a computerized card sort:
■ A subscription to a remote card sort service, a computer program that offers card sort functionality, or a web-facing server hosting a card sort program you have created
■ Participants who have access to a computer and/or the Internet
Create Cards for an In-Person Sort
To create the cards, type the name of the object, a blank space, and the definition of the object either directly on card stock or on a sticky printer label (see Figure 11.1). You can also add an example of the object, if you feel it will help users understand the object. Make sure that you use at least a 12-point font. It is easy to create a file of the objects and then print out several sheets. You can then quickly stick labels on the cards. Alternatively, you could buy sheets of punch-out index cards and print directly onto the sheets; however, we have found them only in white.
The number of index cards needed (C) can be computed by multiplying the number of objects in the sort (O) by the number of participants you intend to recruit (P):
So, if you have 50 objects and ten participants, you will need 500 index cards. We recommend providing about 20 blank cards per participant for labeling their groups and in case they want to add an object.
Primary Data Collected in a Card Sort
The main type of data you will collect in a card sort (both open and closed) is how participants group items. In a closed sort, since your categories are predetermined, you will collect only information about which cards participants assign to which categories. In an open sort, since participants create the categories, in addition to which cards participants think would go together, you will also gather how many groups participants think there should be for the set of items and how they name these groups. These data give insight into how content is organized in the participants’ minds and the vocabulary they associate with the organized content.
Additional Data That May Be Collected in an Open Card Sort
There are five types of changes participants may be able to make to the cards you provide when you conduct an open card sort:
■ Add a new item
■ Rename an item
■ Change a definition
■ Place an item in multiple groups
Some of these options may not be available in a computerized card sort depending on the design of the program. Furthermore, all of these changes must be analyzed manually (see “Data That Computer Programs Cannot Handle” section, page 325). Often, the additional information that you obtain by allowing participants to make these changes justifies the additional work, but be sure to consider this in your planning.
Delete an Object
If a participant does not think an object belongs in the domain, he or she can remove it. For example, if you have the object “school bus” in a card sort for your travel app, a participant may want to remove it because in that person’s experience, school buses are never provided as an option on travel apps.
Allowing participants to remove cards reveals whether you are providing users with content or tasks that are unnecessary—which represent “noise” for the user to deal with. It can also reveal whether you (or the development team) have an incorrect perception of the domain (e.g., providing school buses on a travel app). However, you may have a product where all of your features must be included for business reasons. If this were the case, you would not want to allow participants to create a “discard” pile. Deleting an object may not be an available option for a computerized card sort.
Add a New Object
As participants read through the cards, they begin to understand the depth and breadth of information or tasks your product supports. They may realize that certain information or tasks are missing from the sort and therefore from your product. Using our travel example, a participant may notice that “airport code” is missing from the sort and add it in. Perhaps this was left out because the development team thought that the full name of the airport was more helpful and the airport code is unnecessary. Allowing participants to add cards points out information or tasks that users expect to have in your product. You should also ask users to define any objects they add and state why they are adding them. Again, adding items may not be an available option in a computerized card sort.
Rename Objects
As we mentioned at the beginning of the chapter, you can collect information about terminologies in a card sort. You might present participants with an object they are familiar with, but in their opinion, the name of the object and definition do not match up. Sometimes, differences exist between companies and different parts of the country, or there is an industry standard term that you were not aware of. Technical jargon or abbreviations that we are not aware of are sometimes used in the workplace, or users may simply have another term for the object in their workplace. By allowing participants to change the names of your objects, you collect information about terminologies that you may not have had before.
Change a Definition
Providing a definition for each term ensures that everyone is on the same page. This is important when asking participants to organize information. If everyone has a different understanding of the objects he or she is sorting, there will be no consensus in the organization of the cards. Sometimes, the definitions provided are incomplete or not quite right, so allow participants to make additions, deletions, or word changes to the definitions.
Place an Object in Multiple Groups
Sometimes, participants tell you that a single object belongs in multiple locations. In order to do this, a participant would need to create a duplicate card. This adds some complexity to the data analysis but you may want to collect this information (see “Data Analysis and Interpretation” section, page 318). You want to understand where an object best fits, so ask participants to place the card provided in the best group. Then, ask them to create as many duplicate cards as necessary and place them in the additional locations and note this to be analyzed separately.
Players in Your Activity
You will need users to take part in either an in-person or a remote card sort. For an in-person card sort, you will also require other people to help conduct the activity. In this section, we discuss the details of all the players involved in a card sort session.
The Participants
Users may not always have optimal mental models (Nielsen & Sano, 1994). Designing a system based on flawed user mental models can clearly hamper user performance. For this reason, you should avoid including users in your card sort with no or little experience in the domain of interest. Obviously, if a user does not understand a domain well and have experience in it, that person’s mental model will not be as efficient or even correct as that of others who do.
All participants should meet the same user profile (see Chapter 2, “Learn About Your Users” section, page 35). It is not advisable to mix user types. If different user types sort information differently, you may need to create a different interface for each user type. Mixing the user types in the same sort washes out those differences and could result in an interface that no one can use. If you wish to compare user types (e.g., novice versus expert), we recommend running six or eight of each type, analyzing the data, adding a couple more, seeing how the groups change, and then determining whether more participants are needed as described in the next section. Refer to Chapter 6, “Recruiting Participants” section on page 126 for more information.
How Many Participants?
Aiming for 15 participants is a safe bet. A study with 168 participants revealed that a card sort with 15-20 participants can yield a correlation of 90% with the full data set (Tullis & Wood, 2004), meaning that at about 15 participants, you get a pretty good idea of how another 150 or so participants would group items. In the same study, Tullis and Wood reported that beyond 30 participants, you get diminishing returns, meaning that you get less information for your time and effort. In academic studies, rather than industry studies, we often get to run 30 participants.
In practice in industry, we often end up running one or two group sessions with 10-12 participants of the same user type. If you are on a time and resource budget, however, run six or eight participants and analyze the data. Add an additional couple of participants and see whether the addition of each new user changes the groupings (this is a good time to collect think-aloud data). If the results are stable and the major groups do not change, there is no need to run additional participants.
How many participants are needed for the free-listing activity? The answer is, “It depends.” The best way to determine the appropriate number is to conduct the activity with five or six participants, tally the results to see the number of participants identifying each object, and then see how those results change by adding one or two new participants. If the results are stable, no further participants are needed.
The Facilitator
For an in-person card sort, a facilitator is needed for the activity. If you run participants as a group, it helps to have a colleague as an extra pair of hands, but that is optional. The job of the facilitator is to provide initial instructions, distribute the materials, answer any questions along the way, and then collect the materials. If run as a group, the majority of the session is spent sitting quietly, answering any questions, and making sure people are not comparing their sorts. If run individually, the facilitator must be familiar with the think-aloud protocol and how to instruct participants in it (see Chapter 7, “Using a Think-Aloud Protocol” section, page 169). The facilitator will also need to take notes of what a participant is thinking and record the session, in case you miss something or if you want to analyze the think-aloud results in more detail.
The Videographer
If you are conducting the card sort in a group setting, there is no discussion to video record, but if conducting the sort individually, it is beneficial to record so that you can capture the think-aloud data. You will find a detailed discussion of videotaping tips and the benefits of video recording in “Recording and Notetaking” section on page 171 in Chapter 7. If you plan to record, make sure that someone takes responsibility for this task. It is ideal if you can have someone to monitor the video equipment during the session in case something goes wrong, but if that is not possible, set up the shot, hit “Record,” and hope that nothing goes wrong. We have found that a useful video angle for card sorts, in particular, is over the shoulder of the participant. This way, you capture the way the participant moves the cards into groups and how he or she points to them as he or she thinks aloud.
Inviting Observers
If you are conducting the card sort in a group setting or remotely, there is nothing for an observer to see except for either a room full of people silently grouping cards or data flowing in from the remote sort program. If the session is conducted individually and in person, stakeholders will find it interesting to hear why people group objects the way they do (see Chapter 7, “Inviting Observers” section on page 161 for more information).
Conducting a Card Sort
You have prepared for the card sort and now you need to actually conduct the session. The timeline in Table 11.1 shows the sequence and timing of events to conduct a card sort.
Table 11.1
Timeline for conducting a card sort
| Approximate duration | Procedure |
| 3 minutes | Welcome participants (introductions, forms) |
| 5 minutes | Conduct a card sort practice |
| 3 minutes | Instructions |
| 30-100 minutes | Card sorting |
| 5 minutes | Wrap-up (thank participants, escort them out) |
Activity Timeline
The times in Table 11.1 are approximate times based on our personal experience and should be used only as a guide. The overall length of the session will obviously depend on the number of cards to be sorted and whether you are having participants think aloud. Participants can typically sort 50-70 cards in a one-hour session when not asked to think aloud. For remote sessions, we have found it is best to limit the sort to 30 minutes, which may require you to limit the number of cards.
For both remote and in-person card sorts, the basic idea is that you present people with cards (either a paper card or a virtual card on a computer) and ask them to group those cards in the way that makes the most sense to them, and then, in an open sort, to give the groups they create a name that makes sense to them. Because the basic activities in an in-person card sort using physical cards can be used as a basis for other types of sorts, we describe that process in detail here. Variations on this procedure, including using remote computerized sorts, can be ascertained by leaving out various portions of the activities described here or modifying them. For example, in a remote sort, there is no need to greet participants and offer snacks, but you could easily modify the sample script presented on page 315 to create the instructions screen for a remote computerized sort. We describe the elements unique to conducting a computerized sort separately on page 317.
In-Person Card Sort Using Physical Cards
Welcome the Participants
This is the time during which you greet your participants, allow them to eat some snacks, ask them to fill out paperwork, and get them warmed up and settled in (see Figure 11.2). The details of these stages are described in Chapter 7, During Your User Research Activity, “Welcoming Your Participants” section on page 163.
Practice
Upon their arrival, explain to the participant(s) that the purpose of the activity is to gain an understanding of how people group a set of concepts. We then begin with a practice exercise so that they understand exactly what we will be doing (see Figure 11.3). We typically write about 12-15 types of zoo animals on a flip chart or whiteboard (e.g., grizzly bear, ape, polar bear, monkey). We then ask participants to call out animals that they think belong in the same group (e.g., polar bear and grizzly bear). We circle the items and then ask them to name that group (e.g., bears).
Card Review and Sorting
Once everyone is comfortable with the concept, distribute the cards and provide some instructions. You can use the following sample script:
We are currently designing < insert product description > and we need to understand how to best organize the < information or tasks > in the product. This will help users of the product find what they are looking for more easily.
On each of the cards, we have written a < piece of information or task > in our proposed product, along with a description of it. Please read through all of the cards and make sure both the terms and definitions make sense. If the terms or definitions do not make sense, please make corrections directly on the cards. Use the blank line to rename the object to something that makes more sense to you. In addition, please let me know what changes you are making so I can be sure that I understand what you are writing.
Once you have reviewed all the cards, you may begin sorting them into groups that belong together. There are no right or wrong answers. Although there may be multiple ways you can group these concepts, please provide us with the groupings that you feel make the most sense. When you are sorting, you may place any cards that do not belong (or that you do not use, do not understand, etc.) in a discard pile, and you may use the blank cards to add any objects that are missing. If you feel that a particular card belongs in more than one location, please place the card provided in the best location you believe it fits. Use the blank cards to create as many duplicate cards as necessary and place those in the secondary groups.
When you have completed your sort, use the blank cards to name each of your piles.
You may wish to give participants a rough idea of how many groups of cards you expect. For example, you may say:
We expect you to end up with between seven and 11 groups of cards. It is OK if you end up with more or less, but feel free to use these numbers as rough guidelines.
If there are multiple participants in one room, add:
Please do not work with your neighbor on this. We want to understand how you think these cards should be grouped. We do not want a group effort—so please do not look at your neighbors’ cards.
If this is an individual sort, state:
I would like for you to think aloud as you work. Tell me what you are thinking as you are grouping the cards. If you go quiet, I will prompt you for feedback.
Whenever participants make a change to a card, we strongly encourage them to tell us about it. It helps us to understand why they are making the change. In a group session, it offers us the opportunity to discuss the change with the group. We typically ask questions like:
Spencer just made a good point. He refers to a “travel reservation” as a “travel booking.” Does anyone else call it that?
or
Keisha noticed that “couples-only resorts” is missing. Does anyone else book “couples-only resorts?”
If anyone nods in agreement, we ask him or her to discuss the issue. We then ask all the participants who agree to make the same change to their card(s). Participants may not think to make a change until it is brought to their attention; otherwise, they may believe they are the only ones who feel a certain way and do not want to be “different.” Encouraging the discussion helps us decide whether an issue is pervasive or limited to only one individual.
Participants typically make terminology and definition changes while they are reviewing the cards. They may also notice objects that do not belong and remove them during the review process. Most often, adding missing cards and deleting cards that do not belong are not done until the sorting stage—as participants begin to organize the information.
Labeling Groups
In an open sort, once the sorting is complete, the participants need to generate a name for each of the groups they have created. Give the following instructions:
Now I would like for you to name each of your groups. How would you describe the cards in each of these piles? You can use a single word, phrase, or sentence. Please write the name of each group on one of the blank cards and place it on top of the group. Once you have finished, please staple each group together, or if it is too large to staple, use a rubber band. Finally, place all of your bound groups in the envelope provided.
Computerized Card Sort
Depending on which computerized card sort program you choose and whether that sort will be conducted in person or remotely, the steps to set up the card sort will differ slightly. Generally, you will need to set up the computerized program by telling it how many participants you would expect, by telling whether your participants will be participating in an open or closed card sort, by entering the names and definitions of the text you would like to appear on each card, and by entering the instruction text that you would like each participant to see. Each computerized card sort program will include specific instructions on how to set up and run a card sort using its platform.
Computerized Card Sort Programs
There are a variety of free, freemium, and pay computerized card sort programs available for either in-person or remote studies. For example,
■ UXSORT (free)
(https://sites.google.com/a/uxsort.com/uxsort/home)
■ Optimal Workshop’s OptimalSort (freemium)
(http://www.optimalworkshop.com/optimalsort.htm)
■ NIST’s WebCAT® (free)
(http://zing.ncsl.nist.gov/WebTools/WebCAT/overview.html)
■ uzCardSort (free; open source)
http://uzilla.mozdev.org/cardsort.html
■ xSort (free)
http://www.xsortapp.com/
■ UserZoom (subscription-based; in 2015, $1000 for two months and $9000 for one year)
http://www.userzoom.com
Tom Tullis also keeps a useful and updated list of tools to analyze card sort data at http://measuringuserexperience.com/CardSorting/index.htm.
Data Analysis and Interpretation
There are several ways to analyze the data you collect during a card sort, such as simple summary, cluster analysis, factor analysis, multidimensional scaling, and path analysis. The goal of all of these methods is to understand the similarity between items and determine how to group the most similar items together based on your participant data.
Simple Summary
When testing a small number of participants (four or less) and a limited number of cards, some evaluators simply summarize or even “eyeball” the card groupings. While this is not precise and can quickly become unmanageable when the number of participants increases, it may be useful for pilot tests and/or situations where the number of participants and the number of cards are very small and your time for analysis is extremely limited.
For example, see the card sort results presented in Figure 11.4. You could ascertain the following based on “eyeballing” the groupings:
1. Two participants created five categories, one participant created four categories, and one participant created three categories.
2. All participants grouped the following items together:
ii. Sailboat
iii. Ferry
b. Air/Wings
ii. Helicopter
3. Participants disagreed on groupings for the following item:
This type of informal analysis could help you figure out areas where you needed to pay more attention and/or continue your investigation. In the example above, you would want to pay more attention to the Taxi item because there was disagreement around this item and less attention to the Sea/Water/Boats items because participants agreed those should usually go together.
Similarity Matrix
A similarity matrix, also known as a distance matrix, will allow you to understand how similar or far apart each pair of items is from the participants’ perspective. For example, based on the data presented in Figure 11.5 and our eyeball analysis, we might expect “airplane” and “helicopter” to be very similar or close together conceptually while “yacht” and “hike/walk” to be dissimilar or far apart conceptually.
To investigate this quantitatively, we need to generate a simple matrix. To generate the matrix, create a spreadsheet (e.g., in Excel, OpenOffice Calc, or Google Drive; see companion wesbite for downloadable worksheet [booksite.elsevier.com/9780128002322]) that is set up as follows:
■ Create a single sheet for each participant and one summary sheet.
■ On the x axis, list all items.
■ On the y axis, list all the items again in the same order.
■ In each cell, put a 0 if the participant did NOT group the intersecting set of items together.
For example, for participant 1, put a “0” in the cell where “airplane” and “bicycle” intersect, since he or she did not group those cards together (see Figure 11.5).
■ In each cell, put a 1 if the participant did group the intersecting set of items together.
For example, for participant 1, put a “1” in the intersection between “airplane” and “helicopter” since he or she grouped those cards together (see Figure 11.5).
■ In the summary sheet, sum the numbers for each intersecting set of items.
For example, participants 1, 2, and 4 grouped “hike/walk” and “bike” together, but participant 3 did not. So, we would add the 1 (participant 1), 1 (participant 2), 0 (participant 3), and 1 (participant 4) = 3 for the summary intersection of “hike/walk” and “bike” (see Figure 11.6).
Now that the data are quantified and organized in a matrix, you can see which items are most commonly grouped together (items with a higher number) and which items are rarely grouped together (items with a lower number). The number in each cell represents the number of participants who grouped each item together. You can also analyze these data further using cluster analysis, a specialized card sort program, a statistical package, or a spreadsheet package.
Cluster Analysis
Cluster analysis allows you to quantify and understand your card sort data by calculating the strength of the perceived relationships between pairs of cards, based on the frequency with which members of each possible pair appear together. In other words, it allows you to answer the question: Which items are often grouped together and therefore perceived to be similar, and which items are rarely grouped together and therefore perceived to be dissimilar (or “distant”)? The results are usually presented in a tree diagram or dendrogram (see Figure 11.7).
Reading a dendrogram is relatively straightforward. All items in your sort will be listed vertically. The order of the items reflects the similarity between items; that is, items placed next to each other vertically are more similar than items placed further apart. The lines extending horizontally from each item and then joining other items vertically show where items are grouped at higher levels of relationship. For example, in Figure 11.7, Taxi joins Hike/walk and Bicycle, after Hike/walk and Bicycle have already been joined.
The actual math behind cluster analysis can vary a bit, but the technique used in most computer programs is called the “amalgamation” method. Clustering begins with every item being its own single-item cluster. Then every item’s difference score with every other item is computed (i.e., considered pair-by-pair), as demonstrated in the similarity matrix section (see page 320). Those with the closest (smallest) difference scores are then joined. The more participants that paired two items together, the shorter the distance.
There are several different amalgamation (or linkage) rules available to decide how groups should next be clustered, and some programs allow you to choose the rule used. Single linkage is also called the “nearest neighbor” method because it takes only two near neighbors to join both groups. Single linkage is useful for producing long strings of loosely related clusters. It focuses on the similarities among groups. Complete linkage (or “furthest neighbor”) is effectively the opposite of single linkage and considers the most dissimilar pair of items when determining whether to join groups. This is useful for producing very tightly-related groups. For most cases, we recommend using average linkage, as this method attempts to balance the two methods above by taking the average of the difference scores for all the pairs when deciding whether groups should be joined.
Analysis with a Card-Sorting Program
Whether you have collected data in person or using a computerized card sort program, we recommend using a computer program to analyze card sort data. Data analysis using these tools has been found to be quicker and easier than using manual methods (Zavod, Rickert, & Brown, 2002). In our experience, these methods are also easier than using either a statistics package (e.g., R, SPSS) or a spreadsheet (e.g., Excel). While availability and prices of these programs change rapidly, at the time of publication, the following programs for analyzing card sort data are available for free on the Web:
■ Syntagm’s SynCaps
(http://www.syntagm.co.uk/design/cardsortdl.shtml)
■ NIST’s WebCAT®
(http://zing.ncsl.nist.gov/WebTools/WebCAT/overview.html)
Some of the programs (see “Computerized Card Sort Programs” section, pp. 317-318) also include analysis capabilities. For example, UXSORT automatically generates results including a dendrogram from card sort data collected using the UXSORT tool.
Example Analysis of Open Card Sort Data Using SynCaps
Imagine you have collected the data presented in Figure 11.4 in an in-person open card sort and you want to generate a dendrogram that shows how these items (modes of travel) cluster based on participants’ perceptions.
Prepare the Data
SynCaps only accepts data from a simple .txt file. To create this file, create a new .txt file and enter the data as follows:
Items—list all items with an “I” preceding each item.
Participant—list each participant number, preceded by a “P.”
Groups—list each group the participant creates, preceded by a “G.”
Item numbers—list each item number the participant has included in a group (e.g., I2, I4).
Figure 11.8 presents the travel data in this format.
Analyze Data
Next, open SynCaps and import the data from the .txt file you have created. Be sure to check the “step 2” import screen to make sure that SynCaps recognizes the correct number of participants and item names (see Figure 11.9).
SynCaps will now display a dendrogram with your items grouped (see Figure 11.7). Items that are similar will be displayed close to each other, and items that are dissimilar will be displayed further apart. Examining the dendrogram further, you can see that some items in the tree are grouped very far to the right (close to the items); this indicates that they were often grouped together. The further to the left the lines connecting the items merge, the less often they were grouped together. By default, SynCaps also suggests cut points or places to group items. In this case, SynCaps has suggested four categories: (1) sailboat, ferry, and yacht; (2) airplane and helicopter; (3) subway and train; and (4) bicycle, taxi, and hike/walk (see Figure 11.7). SynCaps lets you edit the number of categories to create using the “groups” drop-down function. A guideline to use in determining how many categories to create is to use the mean number of groups created by your participants. In this example, our participants created 5, 3, 5, and 4 groups, so the mean number of groups is 4.25, which rounds down to 4.
You may use the same process described here to analyze closed card sort data.
Analysis with a Statistics Package
Statistical packages like R, SAS, SPSS, and STATISTICA are not as easy to use as specialized card sort programs when analyzing card sort data. If you are familiar with these packages or you have a card set that is too large for a specialized card sort program, you may use them to conduct hierarchical cluster analysis, multidimensional scaling, or path analysis. Alan Salmoni has an excellent piece on analyzing data from an open card sort using a combination of Excel and R available at http://www.uxbooth.com/articles/open-card-sort-analysis-101/.
Analysis with a Spreadsheet Package
Again, sophisticated analysis beyond simple similarity with a spreadsheet package is not as easy as with specialized card sort software. If you do not have funds to pay to use a commercial card sort analysis tool and the free options presented in this chapter are not available or appealing to you, it is possible to analyze your data using a spreadsheet. You can find excellent, step-by-step descriptions of analyzing the data with a spreadsheet tool at http://boxesandarrows.com/analyzing-card-sort-results-with-a-spreadsheet-template/ (mirrored at: http://www.joelamantia.com/html/projects/card_sort_template_ba.xls and rosenfeldmedia.com/blogs/card-sorting/card-sort-analysis-spreadsheet/).
Data That Computer Programs Cannot Handle
Computer programs dedicated to card sort analysis can be great, but they often do not do all of the analysis for you. Below are some of the issues we have encountered when using various programs. Although the data analysis for these elements is a little awkward, we think the value that the data bring makes them worth collecting.
Adding or Renaming Objects
One of the basic requirements of cluster analysis is that all participants must have the exact same set of cards in terms of name and number. If participants renamed any of the objects or if they added any cards, you will not be able to add this information into the program. You will need to record this information for each participant and analyze it separately. The number of cards added or changed tends to be very small but it is an extra step to take. Note each addition or renaming suggestion down and then tally the number of other participants who did the same thing. At the end, you will likely have a small list of added and renamed objects, along with the number of participants who made those changes. Based on the number of participants who added it, you can assess its importance.
Group Names
The group names that participants provide are not presented in the analysis. You will need to record the pile names that participants suggested and match them to the resulting categories. One option is to record the names of each group for each participant and look for similarities. How many participants created a “Boats” group? How many created an “Air” group? When examining the dendrogram, you will notice clusters of objects. See if there is a match between those clusters and the names of the groups that participants created. Alternatively, for a more systematic approach, especially with a large number of participants, you can use a word frequency analysis tool to count how many times participants associated certain words with each category. The more times a word appears with a given category, the more likely it would serve as a good name for that category.
Duplicate Objects
As we discussed earlier, sometimes participants ask to place an item in multiple locations. Because the computer programs available do not allow you to enter the same card more than once and you must have the same number of cards for each participant, include the original card in the group the participant placed it. The duplicate cards placed in the secondary groups will have to be examined and noted manually.
Deleted Objects
Many computer programs cannot deal with deleted cards. For these programs, if you have allowed participants to create a discard or miscellaneous pile of cards that they do not believe belong in the sort, there is a work-around you need to do. You cannot enter this collection of discarded cards as a group into a computer program, since the cluster analysis would treat these cards as a group of objects that participants believe are related. In reality, these cards are not related to any of the other cards. Place each rejected card in a group by itself to demonstrate that it is not related to any other card in the cluster analysis. For example, if participants placed “Taxi” and “Bicycle” in the discard pile, you should enter “Taxi” in one group and “Bicycle” in a second group.
Interpreting the Results
You now have a collection of rich data. The dendrogram displays groups of objects that the majority of participants believe belong together. Interpreting the dendrogram is straightforward—you can see visually represented which items participants considered conceptually similar. However, changes that participants make to cards can make interpretation of the results tricky. When a deleted object is repeatedly placed in a group by itself, you may see it on a branch by itself or loosely attached to a group where it really does not belong. Additionally, if participants place an object in multiple groups, they may not have agreed on the “best” location to place it. Consequently, you may find that the object is living on a branch by itself or loosely attached to a group where it does not belong. You must use your knowledge of the domain or product to make adjustments when ambiguity exists. Use the additional data you collected, such as new objects, group names, changed terminology, and think-aloud data to help interpret the data.
Let us walk through our travel example and interpret the results of the dendrogram shown in Figure 11.7. Using our domain knowledge and the group labels participants provided in the card sort, we have named each of the clusters in the dendrogram (see Figure 11.10). We appear to have four groups: “Sea,” “Air,” “Rail,” and “Ground.”
It is important to note that the card sort methodology will not provide you with information about the type of architecture you should use (e.g., tabs, menus). This decision must be made by a design professional. Instead, the tree diagram demonstrates how participants expect to find information grouped. In the case of a mobile app with tabs, the tree may present the recommended name of the tab and the elements that should be contained within that particular tab.
Now you should examine the list of changes that participants made (e.g., renamed cards, additional cards) to discover whether there is high agreement among participants.
■ What objects did participants feel you were missing?
■ What objects did participants feel did not belong?
■ What are all of the terminology changes participants made?
■ What definitions did participants change?
■ What items did users want in multiple locations?
Use this information to determine whether your product needs to add or remove information or tasks to be more useful to participants. You may recommend to the team that they conduct a competitive analysis (if they have not already) to discover whether other products support such functionality. Similarly, use the information about deleted objects to recommend that the team examine whether specific information or tasks are unnecessary.
Terminology can be specific to a company, area of the country, or individual. With each terminology change, you will need to investigate whether it is a “standard”—and therefore needs to be incorporated—or whether there are several different possible terms. When several terms exist, you will want to use the most common term.
Finally, examine the definition changes. Were the changes minor—simply an issue of clarification? If so, there is not anything to change in your product. If, however, there were many changes, you have an issue. This may mean that the product development team does not have a good grasp of the domain or that there is disagreement within the team about what certain features of the product do.
Communicating the Findings
When we present the results of a card sort analysis to executives or teams, we present the actual dendrogram generated by the application (as in Figure 11.7) and a simple table to review (see Figure 11.10). We also present a table of changes that participants made to the cards (added objects, deleted objects, terminology changes, and definition changes) and any sketches the designers may have produced to illustrate the recommendations (see Chapter 15, “Reporting Your Findings” section, page 463).
Pulling It All Together
In this chapter, we have discussed what a card sort is, when you should conduct one, and the things to be aware of. We also discussed how to prepare for and conduct a card sort, along with several modifications. Finally, we have demonstrated various ways to analyze the data and used our travel example to show you how to interpret and present the results.
Case Study: Card Sorting with Colleagues: How We Adapted Best Practices to Fit Our Needs
Jenny Shirey Lead Product Designer, Citrix
This case study describes how a team at Citrix used a combination of quantitative and qualitative card-sorting methods to develop a new information architecture for the company intranet. Highlighting the successes of the project as well as important lessons learned, this case study will both educate and inspire teams looking to adapt card-sorting best practices to fit their unique situations and needs.
In 2012, the CEO of Citrix asked the Customer Experience group to lead a redesign of the company intranet. Our vision for the redesigned intranet was to apply user-centered and design thinking methods to create a space that would serve as the online heart of Citrix.
The core team was made up of about 15 people from several functional areas, as well as stakeholders from IT, facilities, and human resources. This not only gave the team the advantage of having a diverse set of experiences and perspectives but also posed challenges related to differences in process and collaborating effectively across time zones and locations. What the team shared, however, was a desire to create an intranet that was not merely functional but design-driven and based on the real needs and desires of Citrix employees.
During our research on intranets, we found that Citrix suffered from many of the same problems that other corporations face. Our intranet had evolved over time without a holistic design process, and the information architecture, or IA, was based on organizational departments, rather than where users would naturally think to seek out content.
After conducting a heuristic evaluation of the site, we found that it was extremely difficult to find content using the existing navigation. Part of this was due to too many choices. For example, the main navigation contained 35 menu items accessible from the top level alone. To help users access certain content and tasks, a well-intentioned team had created a list of quick links along the right-hand side of the home page. However, this list had become excessively long—in some cases, containing up to 50 items. The most useful content, such as benefits information, quarterly updates, or holiday calendars, was often buried deep within several levels of navigation or duplicated across many pages.
For this reason, we knew the IA needed to be redesigned from scratch. To ensure that the new structure would be based on users’ mental models, we decided to conduct a card sort with users to test our new IA.
With card sorting, the recommended best practice is to conduct an open sort, during which participants sort content into groups and then name the groups (Spencer, 2009). Because we were on a tight time and fiscal budget, we created the initial first- and second-level categories as a team. While coming up with the categories, we focused on making key tasks easy to find, based on tasks users had mentioned in previous interviews (see “List of Key Tasks”). In addition, we referred to research on intranet IA best practices while writing the category labels (Nielsen Norman Group, 2014).
After we created the initial IA, our UX designers and user researchers developed a closed card sort study to test our proposed categories with a broader sample of employees.
Our main goal with the card sort was to find out whether our categories would enable employees to complete key tasks quickly and easily. In addition, we wanted to validate whether the labels we had chosen in our group exercise were mutually exclusive—that is, specific enough that an employee would look for a piece of content in one category and not another. We defined the accuracy of our IA by the percentage of users who sorted a card into a specific category. For example, if nine out of ten participants placed “Retirement and 401k” into “Benefits & Pay,” we would call this 90% accuracy.
We tried to avoid creating cognitive overload whenever possible. For this reason, we decided to compare the performance of one-word versions of our category labels against multiword labels. In addition, we planned to measure improvement by comparing the accuracy of our new IA with the original categories. This resulted in a total of three card sorts: one baseline test and two versions of our new IA (see Figure 11.11).
We planned to conduct the card sort with a large number of users to ensure a high level of stakeholder confidence. At the time, Citrix was made up of nearly 9000 employees, and we aimed to reach about 1% of the population, or at least 100 users, to conduct each sort. Because we were targeting users in many different locations, we created the card sort online using the tool OptimalSort, which provided reporting capabilities and the ability to include tooltip explanations on the cards.1
To keep the number of cards manageable, we had users sort types of content, rather than individual items. For example, we combined medical insurance, dental insurance, and vision insurance into one card. This method is sometimes called “sorting by topic,” as opposed to detailed content (Spencer, 118). We also wrote a tooltip explanation for each card to give more details about the content.
Finally, we ran a pilot test with several employees to ensure that the online card sort tool was working properly and that our data collection plan was sufficient. As a result of this pilot test, we adjusted some card labels. For example, we changed “internal transfers” to “employee relocation information” because our testers were confused by the first term.
First Round of Testing and Iterations
Around 130 employees participated in the first round of testing, with an average of 100 participants completing each sort. We sent participants an e-mail with links to the three card sorts, targeting groups that represented various company functions and locations. Participants were encouraged to take all three sorts and were not told which sort was which. Each sort contained the same cards but different categories (the baseline IA, IA 1, and IA 2). We included a short demographic survey to ensure that we had a representative ratio of managers to individual contributors and US- to non-US-based employees. After completing each card sort, participants had the opportunity to provide comments.
When we compared the results from the three variations, we found that our new IA worked better than the baseline, with the more descriptive labels resulting in an overall accuracy rate of 76% (see Figures 11.12–11.14). We also noticed that “Support” was not specific enough to be mutually exclusive. For this reason, we renamed this category “IT Support” and validated the change by sending out a final card sort (IA 3) to a new group of participants.
In addition to the quantitative study, we tested the final IA variation with a few employees in person, in order to understand why participants were sorting cards the way they did. Eight employees participated individually with a facilitator. They conducted the sort on a computer with OptimalSort using think-aloud protocol.
The in-person sorts helped us to gain a deeper understanding of how participants were interpreting our categories. For example, we found that “Company & Campus” was considered a “general” category that participants would use for cards that did not seem to belong anywhere else. Half of the participants also suggested that “Company & Campus” needed subcategories, such as “Facilities” or “Policies.” This showed us that we needed to conduct further research on the best labels for that section.
We also used these sessions as a way to follow up on cards that previous participants had placed into multiple categories, for example, items such as “Tuition Assistance,” “Service Awards,” and “Employment Verification Letter” (see Figure 11.14). In some cases, we found that participants felt content could belong in two different places, making these items ideal candidates for cross-linking.
Overall, the in-person card sorts enhanced the findings from the quantitative card sorts. They also helped us gain insights that we later drew upon when we began to redesign and rewrite the content for each section.
In the end, we were very pleased with the quantitative card sort results, as we found that the final variation of our top-level IA had an average accuracy rate of 84% overall (compared with only 60% accuracy with the baseline). Even better, the most commonly accessed content, according to our list of key tasks, had an accuracy rate of 97% (see Figure 11.15).
Although the categories worked well, we found that the single-word labels were not as successful as the more descriptive labels. “Workplace” and “Career” in particular were considered vague (based on comments from participants in the online survey). When combined with another word, however, categories seemed to be much clearer. For example, “Company & Campus,” while not perfect, seemed to work better than “Workplace” because participants could assume that it related to company information and physical locations. “Support” on its own was not specific enough, but simply changing the label to “IT Support” seemed to make it clearer.
In looking back on the entire process, there are a few things that we might have done differently, as well as several recommendations that we can offer to other teams using card sorting.
One question we had was whether creating the groups as a team and then conducting a closed sort was the right approach or whether we should have pushed our timeline back and conducted an open sort with a small set of users first. While we would not necessarily recommend starting with a closed sort for all website redesigns, the approach worked well in our case for two reasons. First, we had a diverse multidisciplinary team with varying perspectives that we were able to use as a starting point. Second, as Citrix employees, we were able to refer to our own experiences as users of the site.
Based on our experience, we recommend that any large information architecture redesign include team members from a variety of backgrounds. When working on an intranet redesign in particular, it is best if the people designing the structure are not content owners. This avoids the common pitfall of making an IA based on the company organizational structure.
We also learned a few things about online card sorts in particular. During our in-person card sorts, we were surprised that the majority of our participants overlooked the tooltips in the OptimalSort UI. For this reason, we will be cautious about using tooltips during future card sorts.
In addition, the number of participants needed to obtain reliable quantitative results will vary. In our case, we watched the data as they came in and noticed that the trends did not change much beyond about 50 users. Because of this, teams that are compensating participants or on a limited budget could choose to watch the data trends and close the test when the changes appear to level out. This can be easily done using an online tool.
Finally, we highly recommend conducting at least a few card sorts in person, as this results in rich feedback on the “why” behind the “what” that cannot be captured from a survey alone. In our case, we received valuable insights from our in-person card sorts and found it extremely beneficial for us to hear participants describe which cards were difficult to sort and why.
The experience we gained from this study shows that there is no one “correct” way to conduct a card sort. The appropriate methods and resulting insights will vary depending on the content, team, and participants; thus, some level of experimentation and iteration will always be necessary. With card sorting, as with all user research, the most important things to remember are to be curious about users’ point of view, open to surprises, and flexible enough to make adjustments along the way.