Concluding Final
Introduction
In earlier chapters, we have presented a variety of user research activities to fit your needs. After conducting a user research activity, you have to effectively relay the information you have collected to the stakeholders in order for it to impact your product. If your findings are not communicated clearly and successfully, you have wasted your time. There is nothing worse than a report that sits on a shelf, never to be read. In this, the concluding chapter, we show you how to prioritize and report your findings, present your results to stakeholders, and ensure that your results get incorporated into the product.
Prioritization of Findings
Clearly, you would never want to go to your stakeholders with a flat list of 400 user requirements. It would overwhelm them and they would have no idea where to begin. You must prioritize the issues and research you have obtained and then make recommendations based on those priorities. Prioritization conveys your assessment of the impact of the research findings. In addition, this prioritization shows the order in which the research findings should be addressed by the product team.
It is important to realize that usability prioritization is not suitable for all types of user research activities. In methods such as a card sort, the findings are presented as a whole rather than as individual recommendations. For example, if you have completed a card sort to help you understand the necessary architecture for your product (see Chapter 11, page 304), typically, the recommendation to the team is a whole architecture. As a result, the entire object (the architecture) has high priority. On the other hand, most other methods typically generate data that are conducive to prioritization. For example, if you conducted a product evaluation, you might encounter a number of issues from terminology to discoverability to task success. Rather than just handing over the list of findings to the product team, you can help the team understand which findings are the most important so that they can allocate their resources effectively. In this chapter, we will discuss how to do this. We also distinguish between usability prioritization and business prioritization. There is no one “right” answer regarding prioritization. Here, we will highlight a sequence of the prioritization activities where prioritization is broken into two phases. We have found this method effective, but close-knit teams with strong established relationships or teams with a strong understanding of user research can often accomplish prioritization in one phase. The two-stage process is highlighted in Figure 15.1.
First Prioritization: Usability Perspective
There are two key criteria (from a usability perspective) to take into consideration when assigning priority to a recommendation: impact and number.
■ Impact refers to your judgment of the impact of the usability finding on users. Information about consequence and frequency is used to determine impact. For example, if a particular user requirement is not addressed in your product so that users will be unable to do their jobs, clearly this would be rated as “high” priority.
■ Number refers to the number of users who identified the issue or made a particular request. For example, if you conducted 20 interviews and two participants stated they would like (rather than need) a particular feature in the product, it will likely be given a “low” priority. Ideally, you also want to assess how many of your true users this issue will impact. Even if only a few users mentioned it, it may still affect a lot of other users.
The reality is that prioritization tends to be more of an art than a science. We do our best to use guidelines to determine these priorities. They help us decide how to determine a rating of “high,” “medium,” or “low” for a particular finding. To allow stakeholders and readers of your report to understand the reasons for a particular priority rating, the description of the requirement should contain information about the impact of the finding (e.g., “the issue is severe and recurring,” or “the requested function is cosmetic”) and the number of participants who encountered or mentioned it (e.g., 20 of 25 participants).
Guidelines for the “high,” “medium,” and “low” priorities are as follows (keep in mind that these may not apply to the results of every activity):
High:
■ The finding/requirement is extreme. It will be difficult for the majority of users to use the product and will likely result in task failure (e.g., data loss) without addressing it.
■ The requirement will enhance how users do their work in a significant way—breakthrough idea.
■ The finding/requirement has a large impact and is frequent and recurring.
■ The finding/requirement is broad in scope, has interdependencies, or is symptomatic of an underlying infrastructure issue.
Medium:
■ The finding/requirement is moderate. It will be difficult for some users to use the product as a result of the issue or without the required feature.
■ The finding/requirement reflects an innovation that will aid users in accomplishing their tasks.
■ A majority of participants will experience frustration and confusion when completing their work (but not an inability to get their work done) as a result of the finding or without the required feature.
■ The requirement is less broad in scope and its absence likely affects other tasks.
Low:
■ A few participants might experience frustration and confusion as a result of the issue or without the requirement.
■ The requirement reflects a minor impact feature enhancement.
■ The requirement is minor in scope and its absence will not affect other tasks.
■ It is a cosmetic requirement.
If a requirement meets the guidelines for more than one priority rating, then it is typically assigned the highest rating.
Of course, when using these rating guidelines, you will also need to consider your domain. If you work in a field when any error can lead to loss of life or an emergency situation (e.g., a nuclear power plant, hospital), you will likely need to modify these guidelines to make them more stringent.
Second Prioritization: Merging Usability and Product Development Priorities
The previous section described how to prioritize recommendations of a user research activity from a usability perspective. The focus for this kind of prioritization is, of course, on how each finding impacts the user. In an ideal world, we would like the product development team to make changes to the product by dealing with all of the high-priority issues first, then the medium-priority, and finally the low-priority issues. However, the reality is that factors such as budgets, contract commitments, resource availability, technological constraints, marketplace pressures, and deadlines often prevent product development teams from implementing your recommendations. They may really want to implement all of your recommendations but are simply unable to. Frequently, user research professionals do not have insight into the true cost to implement a given recommendation. By understanding the cost to implement your recommendations and the constraints the product development team is working under, you not only ensure that the essential recommendations are likely to be incorporated but also earn allies on the team.
This type of prioritization typically occurs after you have presented the findings to the development team. After the presentation meeting (described later), schedule a second meeting to prioritize the findings you have just discussed. You may be able to cover it in the same meeting if time permits, but we find it is best to schedule a separate meeting. At this second meeting, you can determine these priorities and also the status of each recommendation (i.e., accepted, rejected, or under investigation). A detailed discussion of recommendation status can be found below (see “Ensuring the Incorporation of Your Findings” section, page 469).
It is valuable to work with the product team to take your prioritizations to the next level and incorporate cost. The result is a cost-benefit chart to compare the priority of a usability recommendation to the cost of implementing it. After you have presented the list of user requirements or issues and their impact on the product development process, ask the team to identify the cost associated with implementing each change. Because developers have substantial technical experience with their product, they are usually able to make this evaluation quickly.
The questions in Figure 15.2 are designed to help the development team determine the cost. These are based on questions developed by MAYA Design (McQuaid, 2002).
Average the ratings that each stakeholder gives these questions to come up with an overall cost score for each user research finding/recommendation. The closer the average score is to 7, the greater the cost for that requirement to be implemented. Based on the “cost” assignment for each item in the list and the usability priority you have provided, you can determine the final priority each finding or requirement should be given. The method to do this is described below.
In Figure 15.3, the x-axis represents the importance of the finding from the usability perspective (high, medium, low), while the y-axis represents the difficulty or cost to the product team (rating between 1 and 7). The further to the right you go, the greater the importance is to the user. The higher up you go, the greater the difficulty to implement the recommendation. This particular figure shows a cost-benefit chart for a focus group travel example.
The Chart Quadrants
As you can see, there are four quadrants into which your recommendations can fall:
■ High value. Quadrant contains high-impact issues/recommendations that require the least cost or effort to implement. Recommendations in this quadrant provide the greatest return on investment and should be implemented first.
■ Strategic. Quadrant contains high-impact issues/recommendations that require more effort to implement. Although it will require significant resources to implement, the impact on the product and user will be high and should be tackled by the team next.
■ Targeted. Quadrant contains recommendations with lower impact and less cost to implement. This may be referred to as the “low-hanging fruit:” They are tempting for the product development team to implement because of the low cost. The impact is lower, but they tend to be easy, so it is good to tackle some of these along with the “high-value” and “strategic” recommendations.
■ Luxuries. Quadrant contains low-impact issues/recommendations that require more effort to implement. This quadrant provides the lowest return on investment and should be addressed only after recommendations in the other three quadrants have been completed, if at all.
By going through the extra effort of working with the product team to create this chart, you have provided them with a plan of attack. In addition, the development team will appreciate the fact that you have worked with them to take their perspective into account.
Presenting Your Findings
Now that you have collected your data and analyzed them, you need to showcase the results to all stakeholders. This presentation will often occur prior to the prioritization exercise with the product development team (i.e., you will have a usability prioritization, but it will not factor in the product team’s priorities yet). The reality is that writing a user research report, posting it to a group website, and e-mailing out the link to the report are just not enough. User research reports fill an important need (e.g., documenting your findings and archiving the detailed data for future reference), but you must present the data verbally as well.
You really need to have a meeting to discuss your findings. This meeting will likely take an hour. In the case of in-depth research like field studies, you may require more time. The meeting ideally should be done in person. Then you can see for yourself the reaction of your audience to the results. Do they seem to understand what you are saying? Are they reacting positively with smiles and head nods, or are they frowning and shaking their heads? If this is not possible, use a web sharing app with cameras.
Why the Verbal Presentation Is Essential
Since your time and the stakeholders’ time are valuable, it is important to understand why a meeting to present your results and recommendations is so critical. No one wants unnecessary meetings in his or her schedule. If you do not feel the meeting is essential, neither will your stakeholders. What follows are some reasons for having a meeting and for scheduling it as soon as possible. When you schedule the meeting, be sure to insert a clear agenda with goals to set expectations.
Dealing with the Issues Sooner Rather Than Later
Product teams are very busy. They are typically on a very tight timeline and they are dealing with multiple sources of information. You want to bring your findings to the attention of the team as soon as possible. This is for your benefit and theirs. From your perspective, it is best to discuss the issues while the activity and results are fresh in your mind. From the product team’s perspective, the sooner they know what they need to implement based on your findings, the more likely they are to actually be able to do this.
Ensuring Correct Interpretation of the Findings and Recommendations
You may have spent significant time and energy conducting the activity, collecting and analyzing the data, and developing recommendations for the team. The last thing you want is for the team to misinterpret your findings and conclusions. The best way to ensure that everyone is on the same page is to have a meeting and walk through the findings. Make sure that the implications of the findings are clearly understood and why they are important to the stakeholders. The reality is that many issues are too complex to describe effectively in a written format. A face-to-face presentation is key. Also, stepping the team though infographics, storyboards, and other visual assets is very helpful.
Dealing with Recommendation Modifications
It may happen that one or more of your recommendations are not appropriate. This often occurs because of technical constraints that you were unaware of. Users sometimes request things that are not technically possible to implement. You want to be aware of the constraints and document them. There may also be a case where the product is implemented in a certain manner because a key customer or contract agreement requires it. By having the stakeholders in the room, you can brainstorm a new recommendation that fulfills the users’ needs and fits within the product or technological constraints. Alternatively, stakeholders may offer better recommendations than what you considered. We have to admit that we do not always have the best solutions, so a meeting with all the stakeholders is a great place to generate them and ensure everyone is in agreement.
Presentation Attendees
Invite all the stakeholders to your presentation. These are typically the key people who are going to benefit and/or decide what to do with your findings. Do not rely on one person to convey the message to the others, as information will often get lost in translation. In addition, stakeholders who are not invited may feel slighted, and you do not want to lose allies at this point. We typically meet with the product manager, the engineer(s), and sometimes the business analysts. The product manager can address any functional issues, the schedule, and budget issues that relate to your recommendations, while the development manager can address issues relating to technical feasibility and the time and effort required to implement your proposals. You may need to hold follow-up meetings with individual engineers to work out the technical implementation of your recommendations. Keep the presentation focused on the needs of your audience.
Creating a Successful Presentation
A timely presentation of results is one of the most important factors. We have seen far too many researchers let data get stale and take weeks to deliver findings to the stakeholders. At this point, the team may have moved on and made decisions without you. You also risk creating the perception that research takes too long to be useful and lose critical buy-in. As a part of any study, it is key to block most if not all of your time poststudy so you can quickly get findings to the team. There are some cases, such as a large field study, where this might not be possible; however, in that situation, you keep your stakeholders engaged throughout the process by sending out periodic updates, posts on a blog site, or video snippets. For the most part, you should shoot for one to three days to generate findings.
The format and style of the presentation is as important as the content that you are trying to relay. You need to convey the importance of your recommendations and do it in a way that is easy to understand and empathize with. Do not expect the stakeholders to automatically understand the implications of your findings. The reality is that product teams have demands and requirements to meet from a variety of sources, such as marketing, sales, and customers. User research is just one of these many sources. You need to convince them that your user findings are significant to the product’s development. There are a variety of simple techniques that can help you do this.
One important element is to think about your presentation in its entirety from creation to delivery. Figure 15.4 below is a great illustration of many of the elements that you should consider in advance as you formulate your ideas all the way through follow-up.
Use Visuals
Regardless of the tool you use to deliver your findings, you will want to make it visual.
Visuals can really help to get your points across, and we deem this to be one of the most important aspects of your presentation. Screenshots, photographs, results (e.g., dendrogram from a card sort), storyboards, personas, and proposed designs or architectures are all examples of visuals you can use. Insert them wherever possible to help convey your message. These are the elements that will bring your story alive and make it stick. Video and highlights clips can be particularly beneficial. Stakeholders who could not be present at your activities can feel a part of the study when watching video clips or listening to audio highlights. This can take some significant time and resources on your part, so choose when to use these carefully. For example, if the product team holds an erroneous belief about their end users and you have data to demonstrate it, there is nothing better than visual proof to drive the point home (done tactfully, of course).
The Delivery Medium
The way you choose to deliver your presentation can have an effect on its impact. Know your audience and what will be most impactful for them. Often we use a variety of formats, which may include some or all of the following:
Slides
PowerPoint can be an effective format to communicate your results. In the past, we used to come with photocopies of our table of recommendations and findings. The problem was that people often had trouble focusing on the current issue being discussed. If we looked around the room, we would find people flipping ahead to read about other issues. This is not what you want. By using slides, you can place one finding per slide so the group is focused on the finding at hand. Also, you are in control so there is no flipping ahead. There are also tools like Prezi that make for a more interactive presentation rather than the serial flow of a PowerPoint.
Posters
Posters are an excellent lingering technique that can grab stakeholders’ attention and convey important concepts about your work quickly. You can create a different poster for each type of user, activity conducted, major finding, storyboard, design principles, etc. It obviously depends on the goals of your study and the information you want the stakeholder to walk away with. The poster can contain a photo collage, user quotes, artifacts collected, results from the activity (e.g., dendrogram from a card sort, charts from a survey), and recommendations based on what was learned (see Figure 15.5). Display posters in the hallways where the product team works so people will stop and read them. It is a great way to make sure that everyone is aware of your findings.
Immersive Experiences
In some cases, it may merit creating a rich immersive experience. A large health insurance provider was faced with the challenge of getting thousands of employees to internalize their customers. The insurer took their user research and built an innovative and engaging mobile persona room and other creative learning experiences to help employees “walk in the shoes” of their customers. It was a huge success and went on a road show to many different sites.
Delivering a Successful Presentation
You have created your masterpiece and you are ready to go. Here are some things to consider during your presentation.
Keep the Presentation Focused
You will typically have an hour at most to meet with the team, so you need to keep your presentation focused. If you need more than an hour, you are probably going into too much detail. If necessary, schedule a second meeting rather than conducting a multihour meeting (people become tired, irritable and lose their concentration after an hour).
You may not have time to discuss all of the details, but that is fine because a detailed user research report can serve this function (discussed later). What you should cover will depend on who you are presenting to and what it is you want to achieve from that meeting. Hopefully, the product team has been involved from the very beginning (refer to Chapter 1, page 4), so you will be able to hit the ground running and dive into the findings and recommendations (the meat of your presentation). The team should be aware of the goal of the activity, who participated, and the method used. Review this information at a high level and leave time for questions. If you were not able to get the team involved early on, you will need to provide a bit of background. Provide an “executive summary” of what was done, who participated, and the goal of the study. This information is important to provide context for your results.
Start with the Good Stuff
Start the meeting on a positive note; begin your presentation with positive aspects of your findings. You do not want to be perceived as someone who is there to deliver bad news (e.g., this product stinks or your initial functional specification is incorrect). Your user research findings will always uncover things that are perceived “good news” to the product team.
For example, let us say you conducted a card sort to determine whether an existing travel website needed to be restructured. If you uncovered that some of the site’s architecture matched users’ expectations and did not need to be changed, while other aspects needed to be restructured, you would start out your discussion talking about the section of the product that can remain unchanged. The team will be thrilled to hear that they do not need to build from scratch! Also, they work hard and they deserve a pat on the back. Obviously, putting the team in a good mood can help soften the potential blow that is to come.
Everyone loves to hear the positive things that participants have to say, so when you have quotes that give praise to the product, be sure to highlight them at the beginning of the session.
Prioritize and Start at the Top
It is best to prioritize your issues from a usability perspective prior to the meeting (refer “First Prioritization: Usability Perspective” section, page 451). It may sound obvious, but you should begin your presentation with the high-priority issues. It is best to address the important issues first because this is the most critical information that you need to convey to the product team. It also tends to be the most interesting. In case you run out of time in the meeting, you want to make sure that the most important information has been discussed thoroughly.
Avoid Discussion of Implementation or Acceptance
The goal of this meeting is to present your findings. At this point, you do not want to debate what can and cannot be done. This is an important issue that must be debated, but typically, there is simply not enough time in the presentation meeting. It will come in a later meeting as you discuss the status of each recommendation (discussed later). If the team states, “No, we can’t do it,” let them know that you would like to find out why, but that discussion will be in the next step. Remind them that you do not expect the research findings to replace the data collected from other sources (e.g., marketing and sales), but rather that the data should complement and support those other sources and you will want to have a further discussion of how all the data fit together. A discussion of when and how to determine a status for each recommendation can be found below (see “Ensuring the Incorporation of Your Findings” section, page 469).
Avoid Jargon
This sounds like a pretty simple rule, but it can be easy to unknowingly break it. It is easy to forget that terms and acronyms that we use on a daily basis are not common vocabulary for everyone else (e.g., “UCD,” “think-aloud protocol,” “transfer or training”). There is nothing worse than starting a presentation with jargon that no one in the room understands. If you make this mistake, you are running a serious risk of being perceived as arrogant and condescending. As you finalize your slides, take one last pass over them to make sure that your terminology is appropriate for the audience. If you must use terminology that you think will be new to your audience, define it immediately.
Next Steps
Be sure to convey the next steps. You do not want to leave the team with a pile of insights and walk away. The next step could be a variety of things depending on circumstances. For example, it might be generating a prototype to bring the findings to life, it might be a recommendation for a follow-up activity, such as a design sprint, or it could be a follow-up meeting to define the product requirement for the next release based on your findings. Whatever the next step, be explicit. You want to keep the momentum going and ensure your findings get used appropriately.
Reporting Your Findings
By this point, you have conducted your activity, analyzed the results, and presented the recommendations to the team, and now, it is time to archive your work. It is important to compile your results in a written format for communication and archival purposes. This could be in the form of a linear report or, if you have a lot of materials for stakeholders to consume (e.g., videos, photos, literature review, posters), a website may be more usable. After your deliverable is created and finalized, you should post it on the web for easy access. You do not want to force people to contact you to get the report or other deliverables or to find out where they can get it. The website should be well known to all stakeholders. The more accessible it is, the more it will be viewed. In addition to making it accessible online, we recommend sending an e-mail to all stakeholders with the executive summary (discussed below) and a link to your deliverables as soon as they are available.
Report Format
The format of the report should be based on the needs of your audience. Your manager, the product development team, and executives are interested in different data. It is critical to give each reader what he or she needs. In addition, there may be different methods in which to convey this information (e.g., the web, e-mail, paper). In order for your information to be absorbed, you must take content and delivery into consideration. There are three major types of report:
■ The recommendations report
■ The executive summary
The complete report is the most detailed, containing information about the users, the method, the results, the recommendations, and an executive summary. The other “reports” are different or more abbreviated methods of presenting the same information. You can also include items such as educational materials and posters to help supplement the report. These are discussed in “Creating a Successful Presentation” section on page 457.
The Complete Report
Ideally, each user research activity that you conduct should have a complete report as one of your deliverables. This is the most detailed of the reports. It should be comprehensive, describing all aspects of the activity (e.g., recruiting, method, data analysis, recommendations, conclusion).
Value of the Complete Report
The truth of the matter is that not everyone creates a full report. The understanding of user research, the culture of your company, and the pace of work may not make this a reasonable or useful task. For many companies, the presentation deliverable may serve as the final report.
A full report can serve some important functions. Also, regulated industries (e.g., drug manufacturers) may be required for legal reasons to document everything they learned and justify the design recommendations that were made. We will present some additional reasons why it holds value, and you decide. See Appendix B for a sample report template.
The good news is that it is really not much extra work to pull a complete report together once you have a template (discussed later). Plus, the proposal for your activity will supply you with much of the information needed for the report. (Refer to Chapter 6, “Creating a Proposal” section, page 116.)
Archival Value
The complete report is important for archival purposes. If you are about to begin work on a product, it is extremely helpful to review detailed reports that pertain to the product and understand exactly what was done for that product in the past. What was the activity? Who participated? What were the findings? Did the product team implement the results? You may not be the only one asking these questions. The product manager, your manager, or other members of the user research group may need these answers as well. Having all of the information documented in one report is the key to finding the answers quickly.
Another benefit is the prevention of repeat mistakes. Over time, stakeholders change. Sometimes when new people come in with a fresh perspective, they suggest designs or functionality that have already been investigated and demonstrated as unnecessary or harmful to users. Having reports to review before changes are incorporated can prevent making unnecessary mistakes.
The detail of a formal, archived report is also beneficial to people that have never conducted a particular user research activity before. By reading a report for a particular type of activity, they can gain an understanding of how to conduct a similar activity. This is particularly important if they want to replicate your method.
Certain Teams Want the Details
Complete reports are important if the product team that you are working with wants to understand the details of the method and what was done. This is particularly important if they were unable to attend any of the session or view videotapes. They may also want those details if they disagree with your findings and/or recommendations.
Consulting
Experienced user researchers working in a consulting capacity know that a detailed, complete report is expected. The client wants to ensure that they are getting what they paid for. It would be unprofessional to provide anything less than a detailed report.
Key Sections of the Complete Report
The complete report should contain at least the following sections.
Executive summary
In this summary, the reader should have a sense of what you did and the most important findings. Try not to exceed one page or two pages maximum. Try to answer a manager’s simple question: “Tell me what I need to know about what you did and what you found.” This is one of the most important sections of the report, and it should be able to stand alone. Key elements include:
■ Statement of the method that was conducted
■ Purpose of the activity
■ The product and version number your research is intended to impact (if applicable)
■ High-level summary of the participants (e.g., number of participants, job roles)
■ The key, high-level findings in one or two pages at most
Background
This section should provide background information about the product or domain of interest.
■ What product was the activity conducted for? Or what domain/user type/question were you investigating?
■ What is the purpose of the product?
■ Were there any past activities for this product? Who conducted them and when?
■ What was the goal of this activity?
Method
Describe in precise detail how the activity was conducted and who participated.
■ Participants. Who participated in the activity? How many participants? How were they recruited? What were their job titles? What skills or requirements did an individual have to meet in order to qualify for participation? Were participants paid? Who contributed to the activity from within your company?
■ Materials. What materials were used to conduct the session (e.g., survey, cards for a card sort)?
■ Procedure. Describe in detail the steps that were taken to conduct the activity. Where was the session conducted? How long was it? How were data collected? It is important to disclose any shortcomings of the activity. Did only eight of your 12 participants show up? Were you unable to complete the session(s) due to unforeseen time restraints? Being up front and honest will help increase the level of trust between you and the team.
Results
This is where you should delve into the details of your findings and recommendations. It is ideal to begin the results section with an overview—a couple of paragraphs summarizing the major findings. It acts as a “mini” executive summary.
■ What tools, if any, were used to analyze the data? Show visual representations of the data results, if applicable (e.g., a dendrogram).
■ Include quotes from the participants. Quotes can have a powerful impact on product teams. If the product manager reads quotes like “I have gone to TravelMyWay.com before and will never go back” or “It’s my least favorite travel site,” you will be amazed at how he or she sits up and takes notice.
■ Present a detailed list or table of all findings with recommendations. The level and type of detail will depend on the kind of activity you have conducted. For example, a usability test will have a list of issues with priorities and detailed recommendations, while a field study may be designed with personas as the final deliverable. If appropriate, include a status column to track what the team has agreed to and the priority. Document what recommendations they have agreed to and what they have rejected. Where possible, talk explicitly about the expected effectiveness of your recommendations once they are adopted (e.g., improved click-through rates, increased time on-site, decreased time on task, and reduced usability errors on subsequent usability tests). It adds credibility when you can speak to the expected impact.
Conclusion
Provide the reader with a summary of the activity and indicate the next steps.
■ How will this information aid the product?
■ What do you expect the team to do with the data?
■ Are there any follow-up activities planned or recommended?
■ Are there any limitations to how the data can be used?
Appendixes
Here, include any documents that were a part of the process (e.g., the participant screener, the protocol that was used for the activity, any surveys that were given, etc.).
The Recommendations Report
This is a version of the report that focuses on the findings and recommendations from the activity and is ideal for activities that result in tactical recommendations like usability tests or card sorts. This report format is ideal for the audience that is going to be implementing the findings—typically the product manager or developer manager. In our experience, developers are not particularly interested in the details of the method we used. They want an action list that tells them what was found and what they need to do about it (i.e., issues and recommendations). To meet this need, we simply take the results section of the complete report (discussed above) and save it as a separate document.
We find that visuals such as screenshots or proposed architecture flows, where appropriate, are important for communicating recommendations. A picture is truly worth a thousand words. We find that developers appreciate that we put the information that is important to them in a separate document, saving them the time of having to leaf through a 50-page report. Again, the more accessible you make your information, the more likely it is to be viewed and used. You should also provide your developers with access to the complete report in case they are interested in the details.
The Executive Summary Report
The audience for this report is typically executive vice presidents and senior management (your direct manager will likely want to see your full report). The reality is that executives do not have time to read a formal user research report; however, we do want to make them aware of the activities that have been conducted for their product(s) and the key findings. The best way to do this is via the executive summary. We typically insert the executive summary into an e-mail and send it to the relevant executives, together with a link to the full report. (The reality is that your executives still may not take the time to read the full report, but it is important to make it available to them.) This is especially important when the changes you are recommending are significant or political. Copy yourself on the e-mail and you now have a record that this information was conveyed. The last thing you want is a product VP coming back to you and asking, “Why was I not informed about this?” By sending the information you can say, “Actually, I e-mailed you the information on June 10.”
Report Supplements
You can create additional materials to enhance your report. Different people digest information best in different formats, so think of what will work best for your audiences. Educational materials and posters (as discussed above) are two ways to help relay your findings in an additional format. You normally would not create this additional content for every study, but it can be helpful for new teams or business-critical projects.
Ensuring the Incorporation of Your Findings
You have delivered your results to the stakeholders, and now you want to do everything you can to make sure the findings are acted upon. You want to make sure that the data are incorporated into user profiles, personas, functional documentation, and ultimately the product. As mentioned in the previous section, presenting the results rather than simply e-mailing a report is one of the key steps in making sure your results are understood and are implemented correctly. There are some key things you can do to help ensure your findings are put to use. These include involving stakeholders from beginning to end, becoming a team player, making friends at the top, and tracking outcomes.
Stakeholder Involvement
A theme throughout this book has been to get your stakeholders involved from the inception of your activity and to continue their involvement throughout the process (refer to Chapter 1, page 4). Their involvement will have the biggest payoff at the recommendations stage. Because of their involvement, they will understand what the need was for the activity, the method used, and the users who were recruited. In addition, they should have viewed the session(s) and should not be surprised by the results of your study. By involving the product team from the beginning, they feel as though they made the user research discoveries with you. They have seen and heard the users’ requirements firsthand. Teams that are not involved in the planning and collection processes may feel as though you are trying to shove suspect data down their throats. It can sometimes be a tough sell to get them to believe in your data.
If the team has not been involved in the user research process at all, start to include them now—better late than never! Work with them to determine the prioritization of your findings (see “Prioritization of Findings” section, page 450) and continue to do so as the findings are implemented. Also, be sure to involve them in planning for future activities.
Be a Virtual Member of the Team
As was mentioned earlier in this book, if you are not an organizational part of the team, you will want to become a virtual member. Be an active, recognized member of the product team from the moment you are assigned to the project (refer to Chapter 1, page 4). Take the time to become familiar with the product and with the product team priorities, schedule, budget, and concerns. You need to do your best to understand the big picture. You should be aware that usability data are only one of the many factors the team must consider when determining the goals and direction of their product (refer to Chapter 1, “A Variety of Requirements” section, page 12).
Recognizing this time investment, the product team will trust you and acknowledge you as someone who is familiar with their product and the development issues. This knowledge will not only earn you respect but also enable you to make informed recommendations that are executable because you are familiar with all the factors that impact the product’s development processes. The team will perceive you as someone who is capable of making realistic recommendations to improve the product. In contrast, if you are viewed as an outsider with the attitude “You must implement all of the users’ requirements and anything less is unacceptable,” you will not get very far.
Obtain a Status for Each Recommendation
Document the team’s response to your recommendations (e.g., accept, reject, needs further investigation). See Appendix B, the report template. This shows that you are serious about the finding and that you are not simply presenting them as suggestions. We like to include a “status” column in all of our recommendations tables. After the results presentation meeting (discussed above), we like to hold a second meeting where we can determine the priority of the results in terms of development priorities (see “Second Prioritization: Merging Usability and Product Development Priorities” section, page 453) and the status of each recommendation. If the team agrees to implement the recommendation, we document the recommendation as “Accepted.” If the recommendation is pending because the product team needs to do further investigation (e.g., resource availability, technical constraints), we note this and state why. No matter how hard you try, there will be a few recommendations that the product development team will reject or not implement. You should expect this—not all requirements make sense for the business and/or are feasible. In these cases, we note their rejection and indicate the team’s reasoning. Perhaps they disagree with what the findings have indicated, or they do not have the time to build what you are requesting. Whatever their reason is, document it objectively and move on to the next recommendation. We also like to follow up with the team after the product has been released to do a reality check on the status column. We let them know ahead of time that we will follow up to see how the findings were implemented. Did they implement the recommendations they agreed to? If not, why? Be sure to document what you uncover in the follow-up.
Ensure the Product Team Documents Your Findings
As mentioned in Chapter 1, Section “A Variety of Requirements,” page 12 there are a variety of different kinds of product requirements (e.g., marketing, business, user research). Typically, someone on the product team is responsible for creating a document to track all of these requirements. Make sure your user research findings get included in this document. Otherwise, they may be forgotten.
As a reader of this book, you may not be a product manager, but encouraging good habits within the team you are working with can make your job easier. The product team should indicate each requirement’s justification, the source(s), and the date it was collected. This information is key to determining the direction of the product, so it is important to have the documentation to justify the decisions made. If certain user research findings are rejected or postponed to a later release or sprint, this should also be indicated within the document. By referring to this document, you will be able to ensure that the user research is acknowledged—and if it is not incorporated, you will understand why. This is similar to the document that you own that tracks the status of each recommendation, but a product team member will own this document and it will include all of the product requirements, not just user requirements. If the team you work with uses a formal enhancement request system, make sure that your findings are entered into the system.
Track Outcomes
As was just discussed above, you should track the recommendations that the product development team has agreed to implement. If you are working as an external consultant, you may not be able to do this because your contract ends, but if you are an internal employee, it is important. We formally track this information in what we refer to as “the user research scorecard.” Figure 15.6 illustrates the information that we maintain in our scorecard.
User Research Scorecard
We use this information for a number of purposes. If a product is getting internal or customer complaints regarding usability and the team has worked with us before, we like to refer to our tracking document to determine whether they implemented our recommendations. In some cases they have not, so this information proves invaluable. Executives may ask why the research group (or you) has not provided support or why your support failed. The tracking document can save you or your group from becoming involved in the blame game. If the product development team has implemented the recommendations, however, we need to determine what went wrong. The tracking document helps to hold the product team and us accountable.
We also find that the tracking document is a great way to give executives an at-a-glance view of the state of usability for their products. It helps them answer questions such as “Who has received user research support?” “When and how often?” and “What has the team done with the results?” We assign a “risk” factor to each of the activities, based on how many of the recommendations have been implemented. When a VP sees the red coding (indicating high risk), he or she quickly picks up the phone and calls the product manager to find out why they are at that risk category. It is a political tactic to get traction, but it works.
It is important to note that this kind of tracking can be useful within many companies. However, you need to know your company culture. It may not work for you. For example, if you are a part of a small company or a startup, such level of detail and tracking may not be important, as teams are smaller and speed is of the essence.
Pulling It All Together
In this chapter, we have described the steps to take after you have conducted your user research activity and analyzed the data. You may have conducted several activities or only one. In either case, the results and recommendations need to be prioritized based on the user impact and cost to the product development to incorporate them. In addition, we have described various formats for showcasing your data, presenting the results, and documenting your data. It is not enough to collect and analyze user research; you must communicate and archive them, so they can live on long after the activity is done. Good luck in your user research ventures!