“Excuse Me…”

Get comfortable interrupting strangers. The most effective way to quickly test out an early stage concept is to just show it to someone who has no idea you’re about to ask them anything about it. No bias. No expectations. No canned response. This form of guerilla testing is best done one-on-one, politely, respectfully, and in public spaces. Try to match the venue with your potential audience if possible. You also need to come up a test vehicle that suits that place. For example, you can’t just roll up into your local neighborhood Starbuck’s with a full-on VR rig setup without their approval. Conversely, you could easily show someone mockups on a tablet, laptop, or printed images without causing too much disruption. Using a self-contained headset like Microsoft HoloLens or similar is easy, but will cause a bit of a spectacle. Whatever, right?

When first approaching someone, quickly explain what you are doing, then ask if they’d be ok spending a few minutes giving you their reaction and advice. You don’t want to make anyone uncomfortable by approaching them unexpectedly, so be aware of any physical and verbal cues that this may not be the right person or time. If you notice any of them, apologize for interrupting and just walk away. Also realize that mixed reality is a very new concept to many people, so you might have to give them a bit of background without coloring their thinking or biasing their response. We’re looking for honest first reaction, not coerced feedback. Listen, don’t argue or dispute what people are saying. Just record their responses in some way and find your next willing participant.

A rewarding aspect of this kind of quick hit testing is the unexpected resonance or excitement when someone is blown away by your idea.

Recorded Session

Another effective way to get feedback on early stage ideas you’ve envisioned is to recruit people to come into a controlled space and record them (with permission) as they experience things for the first time. Just setup a relatively quiet space where you can talk with them as they are engaging. Shoot a video of them talking through what they see, hear, and feel – it will be invaluable when trying to objectively evaluate the result of the test. We often remember things a bit differently than it happens from the camera’s perspective. Good to have that other view to refer back to.

Use a video-capable DSLR camera on a tripod to participate yourself

Some prototyping methods produce output that can be used as physical props in the test process, such as printouts of images you created or video sequences. Best to let the test participant hold them or manipulate those themselves. It makes it a bit more direct and personal.

Whenever possible, digital prototypes should be used on-device to test out ideas. That might be as simple as a click through slide deck, or as immersive as a full on working technology prototype. Don’t be afraid to put things that are rough in front of people. Any level of fidelity works for these digital demos. These moments are where we get a more visceral reaction and truer indication of connection or not.

Either way, you’ll want to video record both the external view of the person experiencing it within a physical space, and also the internal or in-device first person view of whatever tech you are using. One exciting new development here is the ability to video record everything about the test session using advanced video compositing techniques that give us a complete view of the experience.

Using the Spectator View technology with the Microsoft HoloLens and an HD video camera, we can see and hear the person interacting with our holographic prototype from a third person perspective. It creates a movie like effect where we’re seeing the scene from an observer’s vantage point. Very helpful to take this viewpoint as it tends to be much easier to watch due to the stability of the camera versus the shakiness of a first-person camera viewpoint.

Using “Spectator View” with HoloLens captures people and holograms (source: Microsoft)

The same type of composited scene is possible with VR headsets and green screens using consumer grade equipment. The visuals generated from these are striking because of the immersive nature of the environments. Seeing the person, the digital content, and the space all together in one view is magical.

Green screen techniques allow other people to see your experience

Even though using VR provides a completely immersive experience versus seeing part of the physical room around you, it gives us a great workbench to experiment within and measure responses to.

Wide Release

The in-person and recorded session approaches don’t scale up very well, so if you need to validate the early stage idea with a much wider array of people, consider trying to get your prototypes out to diverse audience through social media or via recruiting within online discussion groups. This approach is a bit more unusual in the case of envisioning early stage concepts as it’s hard to have any sort of back and forth with the participant. Also, keep in mind it’s a completely uncontrolled environment and you’ll get the expected noise in with actual feedback.

The obvious challenges here is produce a prototype that can setup and experienced unsupervised. We also need to make it easy to share comments with us either during or after trying it out. There are lots of online services for setting up feedback channels, post surveys, and keeping track of suggestions for improvements. If you’re working at this large scale, invest some time in leveraging these mechanisms. This effort is obviously important to determining where you stand with this envisioned idea.

The upside of this approach is getting to a much wider audience for testing, much faster than any other way. The downside doing this testing in the open with little control over the sharing and discussion of these activities. Perhaps fine in some cases, detrimental to furthering the idea in others.

Results

Internalizing feedback objectively is extremely difficult. We’re emotional beings. Actions don’t lie. Criticism cuts deep. Try not to interpret or map what you saw and heard to a favorable outcome. It’s human nature to want our ideas to be liked. There’s an inherent bias to be big fans of our own ideas. That said, let it be what it is. Good or bad, people are going to say and do some things you never expected. That’s not a bad thing. Any feedback is valuable if you try to understand the comments as a quick read on a new idea. Everyone is different. People are unpredictable. What one person thinks or does never defines it for all. Don’t be discouraged by negative feedback or overly encouraged by positive reactions. Don’t dig in, be open to changing things up.

Quick Turn Changes

It’s optimal to be able to take in feedback, make a quick change, and then retest to see if there’s an improvement. That kind of quick turn refinement takes some prior planning though. It’s easily accomplished if you are using particular kinds of prototypes that allow for that. For example, if someone says they were confused by a certain response from a hologram, it would be great to jump in and make a change to try out on the next participant. If it’s a code prototype, there are ways to structure your code and assets to allow for fast changes. In a similar way, redrawing images, changing out physical props, or acting the part of an element in the experience to try something new can get you better data.

Alternate Methods

There will be times that it becomes very apparent the method you are using to test out your idea isn’t working. It may be that the medium doesn’t give you enough fidelity to get everything across that you’re trying for. Or conversely, the method you’re using is too literal when you needed more left to the imagination at this point on the envisioning process. Whatever triggers it, there are times when quick thinking is needed to save the day and keep gathering valuable feedback from people. Don’t be afraid to jump in and change things up, even if it requires some degree of improvisation or handwaving to hold the story together.

Whatever works is the best way to proceed. Just because a highly crafted video piece isn’t getting the point across well enough, doesn’t mean that you can’t just start drawing a storyboard out on a whiteboard to keep the person engaged and offering up ideas. Try anything you think might work. Don’t be self-conscious about doing this in front of the person. They’ll appreciate your creativity and drive to get more feedback and data.

Press On

If you and the team are encouraged by what you observed and discussed, there’s no question you’ll want to keep pushing on. Try to challenge yourself to not be satisfied with what you already have. Look for small refinements or big shifts to get an even better experience for people next time. And don’t forget to savor the win. You successfully envisioned something that connected with people. That deserves some celebration.

Reboot

Having a test go poorly because people don’t get the key aspects of the holographic experience is sometimes a blessing in disguise. It can result in you refocusing your effort on getting to the essence more quickly or clearly. There may be certain aspects you didn’t work out quite well enough that will now get attention. Or perhaps you overestimated how easy or clear something would come through to people. Whatever it is, you missed the mark. This is your chance to fix it – or it’s your chance to start over again. Embrace it. Love it, in fact. There’s not a Designer I have ever met that didn’t relish the chance to try something again once they’ve executed it once. You invariably learned something and will have the opportunity to do better next time. Rebooting the envisioning process with all of this new input will lead to some amazing new thoughts and executions.

Abandon

There’s a point in some prototyping projects when you have to make that tough decision to stop work and move on. No shame in that. In fact, I’d say you learn more from this “failure to connect” than any successful testing session. Having to move away from things you worked hard at or are emotionally invested in open up all kinds of new thoughts about how to do things differently next time. Those are the fuel we need to envision better next time. Walking away from a particular execution of an idea is not giving up on the idea, it’s pointing out that you didn’t choose the right way to depict or make the connection. Walk away, but explore why that may have happened.