Sharpening Steel
How to Improve Your Skills and Help Others Improve Theirs
Abstract
This chapter focuses on how to improve your moderating skills for one-on-one user research studies. By practicing with a role-playing session, watching videos of yourself moderating, and asking for specific feedback from others, you can improve and polish your moderating technique. We also discuss how to help other user researchers improve their skills and what kind of things to look for when providing a critique.
Keywords
user research practice; usability improvement; user research skills
16.1 Working on your moderating skills
16.2 Integrating tips and feedback
Since you’re reading this book, you’re already aware that moderating is a skill that can always be improved. You may have noticed that user researchers rarely talk to each other about our moderating skills. Instead, we commiserate with each other about challenging participants or unique situations. This commiseration is valuable, but it can be difficult to translate those stories into actionable ways to improve yourself. Think of your moderating skill as a blade that needs to be sharpened on a regular basis if you want it to work efficiently. If you’ve ever gone a few months between moderating sessions, you’ve probably felt a bit rusty and like you were off of your game. This chapter focuses on how to get feedback on your moderating, and how to give feedback to others. These techniques will help you improve and refine your moderating skills whether you’re a novice or an expert.
16.1 Working on your moderating skills
No matter how experienced you are, at various times in your career you’ll inevitably say or do things while moderating that you’ll wish you hadn’t. You’ll say something outrageously leading or give away something about how a task can be accomplished, and watch in horror as those words float out of your mouth. Even the most experienced moderators among us can get sloppy and fall into bad habits. We all make mistakes. But mistakes don’t make you a bad moderator as long as you recognize when you could have done something better and take steps to avoid those mistakes in the future.
The following are some ways in which you can elicit feedback and improve your own moderating skills.
Watch yourself
First, watch a recording of a session that you’ve moderated. Yes, this may be painful, especially if you’re fairly new to moderating. You may hate the sound of your voice and cringe at the verbal tics you never noticed before. But, take a deep breath and remember that you’re doing this to be better and more successful at your job! Watching yourself is the easiest way to build a realistic awareness of what your moderating is really like. For specific ideas on what to look for while watching the recording, see the sidebar “What to Look for in a Moderating Critique.”
If you can plan ahead of time to record a session specifically for this purpose, and your facility has the appropriate setup, adjust the cameras so you can capture both you and the participant. This view will let you watch your body language and give a wider lens to your interactions with the participant.
Ask a colleague to watch
One of the best ways to get feedback on how you’re doing is to ask a trusted colleague with user research experience to watch while you moderate. If possible, and your confidentiality policies allow it, have him watch you with a real participant, but if that’s not possible, ask another colleague to be your practice participant. Offer a friendly incentive, such as taking him out for a coffee, especially if you’re asking him to help outside of his normal work hours.
If you can’t work out a time when your colleague can watch you actually run a session, see if you can have him watch a recording instead. You can either sit with him as he watches to receive his feedback in context of the recording, or talk with him afterwards.
If you’re not sure how to ask your colleague to watch you and provide feedback, consider something like:
“I’d really like to get some feedback on my moderating. Do you have some time to watch one of my sessions? I’d really appreciate your perspective and recommendations about my tone, my questions, body language, and anything else you think might help me improve.”
If you feel comfortable taking it a step further, consider asking him to also:
Take notes while watching. This will help him document and remember concrete examples of things you did well and could do better.
Be as candid as possible, because you value constructive criticism.
Spend some time debriefing with you, rather than just handing off his notes. Ideally, you’d like to have a conversation.
Pay attention to specific areas that you would like advice for, especially if you’re showing a recording. Ask him to think about what he would do in certain situations in which you struggled. Also, consider giving him a copy of the sidebar “What to Look for in a Moderating Critique.”
You could also ask multiple colleagues (who ideally have experience moderating research sessions) to watch a session. We recommend that you know and trust your audience if you do this, as it is humbling to have a group of people scrutinizing your work! It may be more comfortable to have colleagues who you’ve worked with repeatedly and whose judgment you trust, rather than a random group of people who don’t know you or have limited experience with user research. We recommend asking an individual colleague to take notes while watching and debrief with you afterwards.
Depending on your colleagues, they might feel more comfortable if you broaden the scope to have them comment on anything related to the study setup and protocol in addition to your moderating interactions.
Even better than asking for feedback as a one-time thing, consider finding one or more colleagues to be recurring “moderating mentors.” Ask your mentors to observe your user research studies, either via real time or recordings, and pay attention to the items previously listed. Having a mentor at every study will help you understand how you’re improving over time. The benefit of having multiple mentors is receiving feedback from moderators with multiple perspectives and styles so you can begin to calibrate your own approach.
However, we know that not everyone has the luxury of working with a team of user researchers. If you are a “team of one” and don’t have any colleagues who can provide feedback, reach out to local (or national) user experience groups to see if they can pair you with a more experienced mentor. Even if your mentor is in a different physical location, the Internet has made it easier than ever to share recordings and watch sessions remotely.
Practice with someone
If you’re just starting your research career, you may be nervous about moderating with a real participant in front of your stakeholders. The best way to reduce your nervousness is by practicing as much as you can! Grab a colleague or family member as the participant. If you’re not sure what to ask, try something like:
“Do you have time later this afternoon? I need a participant for a practice session. I’d go through the script I’ll use with actual participants, but I wouldn’t be recording you or using any of your feedback.”
You can use a study plan that you’ve used for a recent study as long as you don’t violate any confidentiality agreements you have in place with your stakeholders. Another option is to adapt a study plan for a product that is publicly available (for a usability study or contextual inquiry), or a different topic (for an interview).
Go through the session end to end, starting with welcoming the participant and going all the way through thanking him for his time. Do your best to pretend that this is an actual session with a participant who you don’t know.
Record the session to watch later, although if you do so, be sure to provide the participant with a consent form explaining how his recording may be used (e.g., you’ll not share it publicly, it will only be viewed by yourself and possibly a few respected colleagues with the purpose of improving your skills).
Practicing isn’t just for new or inexperienced moderators. Even experienced moderators can benefit from additional practice, especially if you give the participant free reign to act unexpectedly at various points throughout the session. When we’ve held practice sessions, some of the types of things we’ve asked the participant to do at random times include:
You can even let the participant flip through this book’s situation chapters to get ideas on what he might want to do during the session. We also recommend watching the companion videos provided on our website for additional ideas: http://www.modsurvivalguide.org/videos. Watch one of the videos, and then act the same scene out with someone playing the participant. Take the suggested approach and apply it to your own style of moderating. How does it feel? What might you need to adjust to better suit your personality and comfort levels?
Think of all of these practice sessions like workouts before your big game. This kind of role-playing lets you experience situations in a relatively safe zone and can better prepare you for what might happen during a real session.
Observe others moderating and ask questions
If you’re lucky enough to work with other user researchers, sit in on as many of their sessions as possible. Even experienced moderators can pick up tips, tricks, and reminders about behaviors they want to try to emulate (or avoid) by watching others. Take notes while you watch so you can review later.
If you’re unsure about why a moderator did something during a session, ask him about it afterwards. Remember to keep your tone curious and courteous, not accusatory.
Seek out resources
While there aren’t a ton of resources focused on moderating, the topic is discussed in almost any book about user research. Find books on the research methods that you’re most interested in, and read their tips on moderating for those methods. There are also many articles that pop up on this topic in professional magazines and journals.
Many user experience conferences that focus on methodology and research techniques will include presentations or tutorials about moderating. Dumas and Loring (2008) also recommend attending a short course on conducting usability tests, although these are usually focused on new or inexperienced practitioners and may be less useful for someone who wants to refine their skills.
Listen to stories from other moderators
The great thing about listening to stories from other moderators is that you can learn something without needing to go through the experience yourself. It’s not as powerful as experiencing it yourself, even in a practice setting, but learning vicariously gives you a framework for thinking about how you might deal with similar situations. While we hope this book is helpful in this respect, we encourage you to reach out to others in your research community so you can learn from each other.
16.2 Integrating tips and feedback
Of course, becoming a better moderator isn’t as simple as taking feedback and tips and just doing it. These techniques won’t immediately make you a superstar moderator—like most skills, moderating takes lots of practice, over a long period of time.
Once you’ve sought out all available resources, practiced, and received feedback on your moderating, take an incremental approach rather than trying to change everything all at once. You have a lot to think about during research sessions, and you don’t want 80% of your brain power going to overanalyzing your performance. Revisiting the metaphor of improvisational comedy, the best improvisation scenes come when the players are not “in their heads.” In other words, if you’re thinking too hard, you’re blocking the passageways for the instincts formed by all of the training, practice, and feedback you’ve received up until that point. The same is true for moderating—don’t get stuck on acting and reacting perfectly, or you’ll just freeze up.
Instead, plan out the incremental changes that you can make. When you run your next session (even if it’s just for practice), focus on one or two small things that you want to improve about your moderating. Make a note for yourself about it in your study plan so you have an additional reminder. Aside from those small things, free yourself up to think on your feet and use your instincts to respond genuinely to the participant and acquire solid research findings. Focus on the same thing across a few sessions or even an entire study. Then, for the next session, focus on another area you want to improve.
As an example of how to plan out these changes, let’s say that you’ve received feedback from a colleague saying you tend to cut off the participant, answer questions too soon, and unconsciously tap your pen on your clipboard. In your next session, focus on addressing the problem of cutting off the participant. Try deliberately to let him finish his thoughts before saying anything. You may have to do this for a few sessions until you feel like you’re not thinking about it so much. Then introduce the next change—addressing the fact that you answer every question a participant asks right away. So in preparation for the next session, write a note that says “WAIT FIVE SECONDS!” at the top of every page of your study plan. Every time the participant asks you a question, count to five in your head before doing or saying anything. Continue working on this for a few sessions until it too feels natural. Then, focus on the pen-tapping for the next go-round.
16.3 Giving feedback to other moderators
Giving solicited feedback
If someone asks you to give feedback on his moderating, take it as a compliment—that person thinks enough of your skills to want your perspective. Congratulations! Be sure to take his request seriously; don’t respond with “Oh, your moderating is fine,” even if you think it is. As we’ve mentioned before, even good moderators can improve and need to keep their skills fresh.
For tips on how to give feedback, see the sidebar “What to Look for in a Moderating Critique.”
WHAT TO LOOK FOR IN A MODERATING CRITIQUE
Here are some pointers on what to look for when watching your own moderating or giving feedback to other moderators (in which case, replace “you” with “the moderator”). Pay attention to:
How you start the session. The first encounter and introductions set the stage for the rest of the session. If you’re nervous, that might be reflected in the participant’s behavior.
Where you’re positioned relative to the participant. Are you too close or too far? How might your positioning be affecting the research? For example, is sitting directly beside the participant encouraging him to fraternize too much?
What you’re saying to the participant. Are you asking leading questions? Are you using terms that are visible in the interface you’re testing? Are you being kind, yet in control? Are you talking too quickly, or too slowly?
The tone of what you’re saying to the participant. Do you sound condescending? Conversely, do you sound unsure or reluctant?
Anything about your behavior that you find annoying or distracting. Are you playing with your watch, your hair, or your clipboard?
Anything unnecessary that you’re doing or saying. Is there something you could stop doing that either gets in the way or doesn’t add value to the session?
The participant’s reactions in particular situations, both his body language and what he is saying. If necessary, go back a bit in the recording and replay it. Did you do (or not do) anything that made the participant react? What did you miss about the participant’s behavior and attitude while you were sitting with him? Which of his reactions are you surprised by?
If, and how, you adapt your behavior throughout the session. How might you adapt your moderating style based on the participant’s reactions? Should you?
Examples of what you did well. What should you continue to do in the future?
If you encountered a situation that’s listed in this book, go back and look at that section. How did you do compared with what is listed there? If you deviated from our suggestions, did it make sense to do so based on your circumstances? Every scenario is unique, and real-life scenarios can include a confusing combination of situations. You may encounter times when the goals and strategies for handling one situation conflict with the goals and strategies for the other, combined situation. If that is the case, use your best judgment and the suggestions in Chapters 2 and 3 to help decide what should take priority. If you’re unsure about whether you did the right thing, start a discussion with colleagues about the best course of action and the trade-offs you had to make.
Giving unsolicited feedback
You may find yourself needing to give feedback to someone who hasn’t asked for it. Maybe you’re working with a consultant who is moderating some research for you, and you’ve noticed some behaviors that are influencing the participant. Or, maybe you have junior team members helping with a study who need a reminder to avoid asking leading questions.
You may feel nervous if the person you need to give feedback to is more senior than you, or is someone who is well-known in the field. One approach you can take to avoid sounding confrontational is to ask the person about why he handled the situation the way he did, taking a curious and conversational tone:
“I noticed that when the participant asked how he was doing, you told him that other participants had the same problems he had. I’m curious about why you answered that way—is that a standard response?”
“During the session, the participant seemed really upset about how poorly he thought he was doing. I’ve seen other sessions where the moderator offers to let the participant take a break when the participant gets that upset. What do you think about that—is that ever appropriate?”
If you feel that you have a strong working relationship with the person you’d like to give feedback to and aren’t worried about his reaction, you can try a more direct approach. But don’t just start giving him feedback; start by asking if he’d like to hear it:
“I had a chance to watch one of your sessions, and thought you did a good job. There were a couple of things that I noticed while you were moderating. I know you have a lot going on, but would you be interested in hearing my feedback?”
“Hey <moderator>, great session! I know that if it were me, I’d want to hear any feedback that other moderators have for me (and please do let me know any time you have feedback for me!). So I do have a couple of ideas to bounce off you about what I noticed while watching you moderate, if you’re up for hearing them.”
Of course, a less direct way of giving the feedback is to present it in a funny way. For example, “Heh, that was funny when you asked, ‘Are you having a problem?’ and the participant got upset and was like, ‘Um-NO!’” We’ve received feedback veiled in humor before and we’ve gotten the message and were glad to hear it! But of course using humor can be touchy, so if the person you’re giving feedback to doesn’t respond, or responds negatively, avoid using that approach with him in the future.
Another way to exchange feedback is to set up a process where you and your colleagues can submit your feedback. For example, if you work on a large user research team, you could create an anonymous survey to fill out after each study. Or the survey could capture “suggestion box” ideas, collecting generalized feedback that is not specifically targeted to any one researcher or study. The entire team can then get together and discuss the comments. It’s a great way to continually discuss moderating techniques and improve everyone’s skills.
If you’re hiring a consultant to work with or for you on a project, his agency might solicit feedback on his performance after the project has ended. If this mechanism is in place, make sure to include feedback on his moderating. If something like that is not in place, consider working with the agency to instate a feedback loop for this purpose.
16.4 Spread your wings
We hope this book has provided you with the confidence to handle anything that gets thrown your way while moderating user research. Even if many of the situations listed are uncommon, we recommend that you flip through them anyway. You should be able to make connections between the patterns, steps, and best practices we’ve presented to you, and our recommended courses of action in the many normal, abnormal, boring, interesting, common, intimidating, zany, funny, and scary situations you’ve read about. Internalizing those connections and recommendations, and continuously working to improve your skills as discussed in this chapter, will help you handle any new situations that arise—with confidence.
When your colleagues sing your praises about your improved moderating techniques, first pat yourself on the back because you’re the one who made that happen. Then, tell them that this book helped and point them to our website: http://www.modsurvivalguide.org. There are lots of additional resources available there, including those that you find in the appendices of this book, to get anyone started and keep you sharp. And by all means, we want to hear your stories too! Visit http://www.modsurvivalguide.org/stories and let’s keep the conversation going about how to learn and evolve as moderators.