Chapter 6: Running Your Sessions

The big day is here. You are about to run your research sessions. As we discussed in Chapter 5, you’ll have done a lot of your preparation in the days beforehand, which takes most (but not all) of the stress out of it.

Research cycle: fieldwork

Roles

We spoke about giving everyone in the research team roles and responsibilities in Chapter 5.

The four key roles on the day of research are:

• Interviewer. This person is responsible for talking to the users and running the test sessions.
• Host (of the observation room). This person is responsible for looking after the people watching the research in the observation room. They will facilitate discussions and make sure observers know how to get the most out of watching the research.
• Note-taker This person is in charge of capturing the information from the test session, such as positive moments, pain points, quotes or suggestions.
• Observer This person has come along to watch the testing but is encouraged to get involved and take notes.

Research sessions are most productive when all of these roles are working in harmony with each other, so it’s worth having a discussion beforehand about who’s going to do what. Even so, on the day it’s a good idea to re-iterate what each person’s supposed to be doing, and to lay out the ground rules. We’ll talk about these roles in the following sections.

The Interviewer

The primary role of the interviewer is simply to interview. You want your entire focus to be on your participant, whether that is in person or remotely. We’ll say a lot more about the role of the interview in the next chapter.

The Host

The host’s role is to manage the observation room, to make sure there’s productive engagement with the session, and the outputs are on track. Without someone to keep them on-task, observers can become boisterous and unfocused, which can make it hard for others to concentrate and listen. So the host needs to juggle people-management and task-management responsibilities.

The type of responsibilities the host can expect are:

• Lead on-the-spot analysis activities – for example, sticking Post-It notes onto screengrabs.
• Ensuring the observers are being catered for with food and drink.
• Keeping an eye out for any support the interviewer might need, and giving them encouragement or feedback.
• Liaising with the venue staff.
• Manage the observers’ behaviour.
• Fielding any last-minute recruitment problems.
• Facilitating conversations about findings and implications.
• Social hosting and small talk.
• Collating and passing on questions from the observers to the interviewer.

The host’s role is really hands-on, and it demands stakeholder management skills, the ability to juggle tasks, and a bit of authority, as you may need to be assertive to keep control of the room.

Testing can be a long day, and it can difficult for people to stay focused and to keep listening and paying attention. You may also need to discourage observer/note-takers from getting on their phones or checking their emails. To do this, you need to keep their hands and minds busy, so encourage them to help out by taking notes. We’ll explain more about how to do this later in the chapter.

Explain the Rules to Observers

At the beginning of the research session, the host needs to explain the rules of the day to observers. This is important to make sure that their involvement in the day is productive and enjoyable for them.

Rules of the day:

• Encourage everyone to introduce themselves and explain who they are, what their role is and their involvement with the research project.
• Reflect on the objectives of the research and what you are there to learn about. This can help observer/note-takers to focus on what they need to be paying particular attention to and listening out for.
• Share copies of the discussion guide and information about the participants (without personal details).
• Tell people that any calls must be taken outside of the observation room to minimize disruption.
• Discussion during the sessions must be kept to a minimum so that everyone can concentrate on watching and listening to the session. The real discussions can be done during the breaks.
• If you are watching sessions using a two-way mirror, the lights must be down and there must be no loud talking or laughing.
• Questions should be saved up during the sessions and given to the host towards the end. The host can then pass this on to the interviewer.

But the main instruction – over and above taking part in any activities or note-taking – is for them to watch and to listen.

Explain the Research Process to Newcomers

If you’re working with stakeholders or project members who are new to user research, it’s worth giving them some background information on why we run the sessions the way we do:

• Over recruiting. You need to prepare observers for the possibility that participants won’t show up as planned. Hopefully you or your recruiter will have confirmed the participants on the morning of the research. However, there is still a chance of no-shows . Overall, you can expect around 10% of your participants to not show up, but this does vary depending on the demographic and topic. Reassure them by telling them how you’ve over-recruited to minimize the disruption and risk of a no-show, and that a spare participant is available to jump in should someone not arrive.
• Reviewing the approach. Let observers know that there will be a gap between sessions. This gives everyone in the observation room time to chat about how it went, and reflect on anything that could be tweaked or improved.
• The first session will probably overrun. Give observers a warning that the first session nearly always overruns, and will likely be the longest of the day. Explain that the sessions are likely to get shorter as the day goes on, because the interviewer becomes more confident with the discussion guide and interviews get more concise and focused.
• Collect questions during the interview. Let observers know that they’ll have the opportunity to put questions to the participant the end of the session. The host should collect them throughout the session rather than firing them over to the interviewer in real time, as this can be incredibly distracting and disruptive. Explain to observers that there will be 5-10 minutes at the end of the session for these questions to be fed in by the interviewer.
• Take stock after three or four sessions. Explain that after half of the sessions, you’ll take time out to reflect on how the day has gone so far. This can give you a chance to think about improvements or tweaks to the session, any new activities you want to take into the session, or anything you want to remove.

Note-taker

As note-taker, you’re responsible for capturing the salient and important comments, thoughts, behaviours and suggestions you hear from participants. Because there’s so much that you could record, it can sometimes be difficult to know what to note down during a research session, which is why it’s important to pilot your note-taking beforehand and discuss it with the rest of the team.

The note-taker also often acts as assistant to the host, so their full list of jobs includes:

• Capturing notes
• Starting and stopping the recording
• Troubleshooting any problems with the prototype or technology
• Anything else that the host may delegate to them

To help with this, we’ll often print out the objectives and keep them on the wall in the observation room. You also need to consider how you are going to analyse your results.

Finally, we should emphasize that, although the note-taker is specifically tasked with taking detailed notes, it’s a good idea for the observers to join in too. They’re likely to do this in different ways:

• The note-taker will be concentrating on creating notes throughout. They’ll be capturing more detail, possibly in a different format from the observers.
• The observers will be creating notes whenever they feel the urge, but they shouldn’t be expected to be doing so consistently throughout.

Taking Notes

There are various different ways to take notes during a research session. Below, we’ll walk you through the different formats and approaches.

The way that you take notes will depend on the type of analysis you will be doing and the deliverables of your research. This can save you heaps of time later in the project.

Post-It Notes

Post-It notes are a really fantastic tool for note-taking. They’re great for capturing issues and mapping out processes. What’s more, they lend themselves to color coding and sorting activities, and their tactile nature makes them more engaging to work with as a group than notepads or laptops.

In the observation room, it’s a good idea to use three colors to allow you to take different types of notes.

• We use pink Post-It notes to make notes about pain points that come up during the session. These are problems that participants encounter, things they find confusing or difficult or goals they can’t achieve because something is missing or poorly labelled.
• Green Post-It notes are for positive comments or interactions. A participant may successfully complete a journey or express delight or excitement at a particular feature.
• We use yellow Post-It notes for suggestions or more general observations. Suggestions are things that the participants express – maybe they would like a way to compare several products. Or maybe they would like web chat as an additional way to get in touch.

Using Post-It notes

There’s no hard and fast rule: depending on the nature of the research you may want to use colors to signify other types of note, such as quotes or steps in a journey.

Our rules of recording on Post-It notes are:

• One comment or observation per Post-It.
• Make sure the participant number is on the Post-It note.
• Include the time on any key moments that you may want to go back and re-watch or use in presentations or compilation videos of your testing.

Post-It notes are a great format because they force you to be selective with what you capture. You can’t write everything you hear verbatim, as you might be tempted to do if you had a blank document open on your laptop. It forces you to focus on the key points.

Post-It notes are also helpful in analysis, as they can be moved around and grouped. If you have lots of observers, they can be a great indication of the salient points or issues that matter because you’ll have lots of duplicates. We talk more about using Post-It notes for analysis in Chapter 8.

When it comes to arranging your notes, you have a number of options:

• You can stick them directly onto printed out screengrabs of your product.
• You can arrange them as a journey map, grouping them under each stage of the process.
• You can agree headings mid-way through the session, and group your Post-Its under those headings.
• You can create a group for each participant.
• You can conduct an affinity sort (see Chapter 8), to uncover themes.

Taking Notes in a Spreadsheet or Trello

Earlier, we mentioned that the outputs of some research you’re doing may be a list of issues. It is likely that you will be asked to prioritize that list so the team knows what to tackle first.

Using a spreadsheet to capture usability issues is useful if:

• You’re working on sprints and/or iterating on your designs.
• You’ve been asked to produce a prioritized list of issues.
• You need to provide a set of recommendations.

Taking notes straight into a spreadsheet can speed up your analysis and output. Otherwise, you may find that you spend time writing up your Post-It notes into a spreadsheet, which is a waste of your time!

Here’s a format that works well for note-taking:

• In the first column of the spreadsheet, we place screen grabs of each of the screens we’re testing.
• If it’s a live website, the second column would have the URL of the page. This is especially useful for people who weren’t able to see all of the research, or if you’re going to be referring back to the document for some time.
• The third column may have the name of the page – this is useful for cross-referencing with the discussion guide. It makes it clearer and easier to follow during the session, as well as in the analysis, if each page has a name.
• The fourth column is where we list the issues. Each issue has its own line. The fourth column can be used for tallying up how many users encounter the same issue – this can be useful in your analysis when you’re prioritising the issues you find.
• The final column is used to propose solutions – this will be filled in after your analysis.

Alternatively, you can use Trello in a similar way:

• Create a list for each page.
• In each list, create a card and add a screengrab of the page and the URL.
• Then add a new card for each issue or observation, within the relevant list.
• Use colors to indicate which participant the cards apply to.

Taking Typed or Written Notes

If you’re running depth interviews, you may not necessarily be showing users a prototype or a thing. You may just be exploring a journey, or trying to understand the participants’ needs and experiences. Because this kind of discovery research is more open-ended, you’re likely to need to use a mixture of Post-It notes and written notes to be able to capture what you’re hearing and seeing.

If you’re using your laptop to take notes, you can use whatever word processing software you like the most. Something like Pearnote or Evernote works well.

Here are some tips for taking notes on a computer or paper during discovery research:

• At the start of the session, write down the participant number.
• As with writing Post-It notes, jot down the time when you hear or see something that is salient or particularly interesting that you may want to go back to.
• Create a code so you can label every comment you capture. This is the equivalent of using color-coded Post-It notes. For example, you may use ‘O:’ at the start of an observation, while ’P:’ can be used to signify a pain point, and ‘Q:’ indicates a quote. If you’re using a computer, then these codes enable you to find quotes and so forth in the document later using control + F.
• You can also use a code system that relates to particular points in the discussion guide, for example ‘111’ might be used to indicate content about the delivery process.
• Use square braces if you want to comment or say that you missed a bit.
• Try using markdown to simplify your note-taking.
• Consider how you want to phrase the things you write down – ie, do you just write it down verbatim or do you want to paraphrase? Do you inject your own thoughts or just record what happened?

Example of note taking code

Having a code will make your analysis easier and more efficient. It also helps you to focus and narrow your note-taking. It will also stop you taking notes verbatim, which could lead to pages and pages of unstructured content.

Taking Notes in Guerrilla/Field Research

If you’re guerrilla-testing as a pair, one of you can do the interviewing and running of the session while the other takes notes.

If you’re guerrilla-testing on your own, you’ll need to take notes as well. This is no different to taking notes in a lab testing session, which we talked about earlier in the chapter. However, it’s difficult to take notes and interview at the same time, so avoid it if you can.

In guerrilla research, the medium for note-taking will depend on the setting:

• If you are able to run your sessions from somewhere like a cafe, you may want to take typed notes.
• If you are testing people on the go, for example on a bus, taking notes on a laptop may be more difficult, and you may not have access to anywhere to charge your laptop. In this case, it’s worth having a semi structured form that you can fill out. Have a box for each main question area, with a space for you to jot down your notes and observations.

Taking Notes after the Interview

Whatever type of interview you’re doing, and whichever note-taking method you’re using, you should always try to jot down your thoughts after the interview. Ideally, try to do this immediately after the session, but if not you can do it at the end of the day or on the journey home. You don’t need to write lots: half a page is plenty. Try to cover off the participant’s main priorities, the answers to the main research questions, changes to rolling hypotheses, your feelings and impressions during the interview, and any ideas that occurred to you.

Tools For Recording

Where possible, we would always recommend that you record your test sessions. There are a few reasons for this.

• There will be occasions where you may not have time to take proper notes about something you heard. On these occasions, you will make a note that there is something interesting to listen to at a certain point in the recording. This allows you to go back to it when you have more time. It is important that the quotes you use are accurate.
• Recordings are also great for providing evidence. There is nothing as powerful for stakeholders as hearing real users struggle with parts of what you are testing. That can be one of the most helpful things to drive improvements in a website or service.
• Recordings are also helpful at refreshing your memory about what happened in the session, or who the participant was.
• They enable you to make a transcript and review the whole session if you need to.
• Finally, they are a great training tool. Take some time to watch some of your sessions to see how you can improve your interviewing technique.

The way you record your sessions will depend on what you are testing and where you are testing it.

	Lab	Guerrilla	Remote
Audio	Stand alone microphone Built in laptop/desktop microphones Recording software, such as Pearnote.	Dictaphone Or use the voice recording software built into your smartphone. Recording software, such as Pearnote.	Built in laptop/remote microphones.Stand alone microphones may also be able to pick up the sound. Recording software, such as Pearnote.
Video	Most labs will have cameras built into the set up. If not, you can use a portable video recorder. Record the screen of mobile devices using Quicktime through your labs desktop.	Camera phone, portable video recorder such as a GoPro. Record key moments to cut down on editing time later.	Screen share using something like GoToMeeting. Screen recording software, such as Quicktime.
Photos	Stills from video recording software.	Camera phone is perfect. You may want to opt for something handsfree like the Narrative Clip. Take photos of people using the thing you are testing – action shots are great	Stills from video recording software if you are camera sharing.

Top Tips For Recording Your Sessions

Getting back to base to discover that you’ve got a poor recording, or no recording at all, is extremely disheartening, so make sure you’re prepared:

• Take a test recording. When you arrive in a new location, take test recordings of your video and audio.
• Get permission. Also ask your participants before you start recording. Some people may not feel comfortable, and you have to respect that. It can also be a good idea to check at the end, especially if the participant has shared some personal information.
• Be aware that recording in public places can be noisy. This can make it difficult when you try and listen back to your recordings. Place your recording device as close as you can to your participant, use a directional mic or a lapel mic, if you have access to one and the participant is OK with it.
• Always take photos of your sessions and outputs. They are great for deliverables and can help to jog your memory in the analysis.
• Small and discreet recording equipment is best. We’ve found that this makes people feel more comfortable, and they quickly forget there is a camera around.
• Make sure you have enough space and charge on your equipment. You don’t want to run out halfway through a session or when you’re out in the field away from power.
• Take spares. Spare batteries, chargers and spare recording equipment. Be prepared for your equipment to fail, and have alternatives ready.
• Some recording software will record the sound of your key taps. If you’re using a laptop to record sound as well as taking notes, check whether or not you can hear your key taps. It can really affect the quality and usefulness of your recordings.
• If you’re working in the lab, using multiple recorders. Use several cameras to capture different angles (for example, recording the screen as well as a shot of the participant’s hands interacting with the device). Use at least two audio recorders. Multiple recordings means you’ve got backup if anything goes wrong.
• Record the screen when testing on a mobile device. These can be used for deliverables and can be powerful evidence to support your findings. You can use software like Quicktime through the desktop in your lab to connect to your mobile device.
• Be aware that using software such as Quicktime in the lab means you can’t record picture in picture. This means that your recording won’t have input from any other cameras that you may have set up in the lab.
• Find software that records audio as well as your screen. We use Silverback but there are lots of other similar tools that you can try, like Camtasia or Screenflow. Something like this is ideal as it saves you time cutting together the audio and the video after testing.

Some guerrilla research

The Practicalities of Using Different Devices

Because there are an ever-growing range of screen sizes and devices available, you may find yourself working on prototypes for one or more of these devices, so you need to be prepared to test on all of them.

The practicalities of testing on any of these devices is pretty much the same. You just may find that the facilities you’re using are more geared up to testing on one – usually desktop. That means that testing on another sort of device will take a bit more planning and preparation.

If you are using a lab that is geared towards testing on desktop, you need to first make sure you have access to the devices that you want to test on. Check that the lab supports the devices. If you’re not sure, arrange to visit in the week beforehand to check the setup. You then need software that allows you to hook the testing device to the desktop screen so you and observers can see what the participant can see. We tend to use something like Quicktime so we can view the device screen on the desktop. This makes it nice and big in the observation room, and we can record it all as if we were testing on desktop.

Remote Testing When Using Different Devices

Remote testing can take a little more consideration, as we mentioned in Chapter 4. Screen-sharing software is a good option – you can then check everything is working on your machine, share your screen with the participant, and give them control of your screen. You can also record your screen using screen capture tools. Products like GoToMeeting, Google Hangouts or Skype will allow you to do this.

It is a little trickier with mobile and tablet remote testing as it takes more setting up. However, there is software available that allows your participant to join the session with the device that you need to run the research on, such as GoToMeeting. Keep in mind that they may need to be sent links to websites or prototypes depending on what you are testing, and this will need to be prepared and sent well in advance.

Guerrilla Testing With Different Devices

Before you go out testing, you need to make sure you have access to the devices you need, and that they are fully charged. If you’re testing with a specific smartphone, you’ll want to check that your potential participants are users of that smartphone. For example, if your users haven’t handled an Android phone before, the learning curve and potential frustration could take a substantial chunk of your session, which is already pressed for time.

Summary

• The key to smooth research sessions is in the preparation leading up to them.
• There are four main roles your project team will need to take on: Interviewer, Moderator, Observer/Note-taker and Note-taker
• The host is in charge of the observation room and needs to set the rules, guide the discussion and encourage observers to take part in activities or help with the note-taking.
• The note-taker will be capturing the insights from the sessions. The type of notes needed will depend on the deliverable. Key insights are: pain points, positive observations, quotes and suggestions.