Paint 3D

One of the new breed of apps that makes 3D approachable for anyone, Microsoft Paint 3D is deceivingly powerful. Available for free as part of the Microsoft Windows 10 Creators Update release, Paint 3D builds on the simple paint program metaphor by effortlessly extending it into three dimensions – while adding mind-blowing new capabilities. Beside the basics of extruding simple drawings and shapes into 3D objects, Paint 3D also helps you place beautiful pre-built 3D models within a virtual space that can be shared with others online. Hundreds of high quality models are available from the online service Microsoft Remix3D.com, making easy to put scenes together quickly if modeling isn’t your strong suit. Think of anything you create in Paint 3D as a stand-in hologram.

Microsoft Paint 3D is a great intro to 3D and terrifically useful in prototyping

In a pleasant throwback UI surprise, Paint 3D offers a pro-active training overlay to get novices into 3D very quickly by previewing what the basic widgets do. The focus is clearly on the creator new to 3D, but the app delivers the goods for experts, too. The ability to export to a 3D model in geometry format for use in Unity and other composition programs is a huge plus. Fast in, fast out. The rendering and shading of objects is way better than you’d think for an app like this.

SUPERPOWER – Paint 3D is an incredibly useful tool for quick creation of placeholder holograms and composition of the scene to be translated in the real world. No device required. And the ability to paint directly on a 3D object is just crazy for a seemingly simple tool like this.

HoloSketch

As a mixed reality designer, there are times when you just need to “block out” an idea in the physical space it would occupy as a sanity check. Nothing can substitute for seeing how holograms will actually appear within the physical space you are working in. HoloSketch from Microsoft Design Labs is great at that task without requiring anything other than a HoloLens. Available in the Windows Store for HoloLens, just run the app and start pulling in models from OneDrive or create your own on the spot with built-in primitives and text.

HoloSketch excels at the composition part of prototyping (source: Microsoft)

The ability to compose a scene is just one step in the prototyping process, but it’s a critical one. Ask any filmmaker about what makes a scene memorable, and beside the emotional aspect of the dialogue and interaction, the placement and relative positioning of elements in the scene makes a huge difference in the impact for the viewer. HoloSketch gives you a straightforward way to try out many iterations of your ideas.

SUPERPOWER – HoloSketch gives you the ability to easily import and place holograms in-device. Nothing beats the ability to quickly holographic placeholders within the physical world.

Unity

Without question, the most useful all-around tool for working in mixed reality is Unity . This constantly improving tool allows designers and developers to create, compose, and test out their mixed reality ideas with real models and code within an integrated development environment (IDE) . Formerly considered a more technical tool for multi-platform game development, Unity is now very approachable tool for anyone to do simple modeling, 3D scene composition, wiring up interactivity, and applying incredible special effects.

Unity supports the preview of your project within its own window or a software device emulator, and serves as the first stop for many in the development path to getting actual bits on device.

Unity gives anyone a straightforward way to get from idea to working code

Designers have often loathed the prospect of learning true 3D modelers and animation systems, not to mention development environments. Although Unity is all of those things, the benefits of diving in and leveraging it get real with your ideas far outweighs any effort overcoming the slight learning curve. It’s also built to support a very interactive and fluid workflow. Trust, me, Unity is your best friend for prototyping and turning your envisioning work into real projects.

SUPERPOWER – Unity is a one stop shop for everything mixed reality. This must have tool enables you to go all the way from idea to modeling, composition, adding object interactivity, previewing behavior, and finally preparing a project for on-device deployment.

Adobe Audition

Sound is the most overlooked aspect of mixed reality experiences (initially). Once you realize how impactful the right sounds and ambient soundtrack have on the overall experience you will automatically start figuring out where to incorporate them. That’s where Adobe Audition comes in.

The mixed reality platforms allow you to place ordinary sounds you’d create or edit in Audition anywhere in the 3D space. By setting those sounds to particular sound levels and applying special effects, you can help the listener hear them coming from discreet positions – perhaps behind or even above them. That kind of spatial sound magic needs you to add code or assign sonic behaviors to objects in programs like Unity. Spatial sound is a very convincing effect and incredibly useful interface tool for designers.

Adobe Audition delivers what you’ll need for audio design

Audio software and 3D modeling software are among the most complex and scary looking software humans have ever produced. Don’t let that dissuade you from jumping in and learning just enough about the basics to get some great use out of this pro quality tool. Coming up with just the right sound for your ambient environment, individual interactions with holograms, and interface cues can all make the difference between a good app and a breakthrough experience.

Adobe Audition is just one of the many sound creation and editing apps you could use to record and manipulate sound – its key benefit being its high quality and comes included with an Adobe Creative Cloud suite subscription.

SUPERPOWER – every sound-related function available to you in one place with Audition. Sound is so important to the overall holographic experience that learning how to leverage the power here is a real advantage to the envisioneer.

Mental Canvas

The first time you see a 3D sketching app at work you realize it’s completely natural, obvious, and one of those things that you could even imagine existing for real. Mental Canvas is a giant leap forward for sketching and an absolute wonder to use. Using a standard digital sketchbook metaphor, Mental Canvas brings an innovative set of functionalities to the task of sketching out scenes – the ability to zoom through the 3D space that your sketch exists in.

Formerly, digital sketches were done in layers just like in Photoshop. Mental Canvas changes the game by adding depth to the mix, made even more graceful and fluid through the use of the Microsoft Dial and a Pen on the Microsoft Surface touchscreen computer.

Mental Canvas brings a third dimension to sketching using the Surface Dial and Pen

The combination of the Dial, Pen, and Surface touchscreen creates the perfect storm of productivity for the drawing crowd. There’s never been anything quite like this available to artists and designers to add depth to scenes in such a natural way. Huge advance.

SUPERPOWER – being able to zoom through space to examine different areas of your holographic idea sketch is ground-breaking and incredibly useful during rapid ideation. This setup allows you to move through the scene in early sketch form to convey key elements.

Photoshop and Illustrator

Unlike many other areas of production design, you shouldn’t reach for these workhorse tools first. Using Adobe Photoshop or Adobe Illustrator would mean you’re probably thinking of locking in to high-fidelity comps of your ideas in 2D, which is fine, but not the optimal path to fully immerse yourself for mixed reality. Once you get going with other techniques, these tools are great for compositing static elements together into comps to share with people, creating storyboard frames, or doing some conceptual illustrations.

It’s hard to overstate the utility of these two applications in the workflow for mixed reality (or any creative endeavor for that matter). The interesting twist is that we’ll be using them more like a game designer would – for conceptual sketches, high-quality texture creation, image labels, stand-in assets, and more.

Adobe Photoshop and Adobe Illustrator are your best choice for 2D asset creation

These incredibly deep tools also come in handy to create just any kind of visual asset imaginable for our prototypes ahead of time so that part of our process flows more smoothly. There are other apps that do this same kind of creation and editing, but none are used more widely, and have more online tutorials walking you through amazing techniques step-by-step.

SUPERPOWER – same as always, Adobe Photoshop and Adobe Illustrator are the best at what they do. They are highly reliable tools to help produce the critical parts of our digital assets.

Visual Studio

The various flavors of Microsoft Visual Studio serve as industrial-strength development environments for mixed reality development. The Windows Mixed Reality platform requires this and the Windows SDK to produce the packages that are deployed to the device itself. Familiar to any developer who has leveraged Microsoft technologies, this dependable workhorse has some useful new features to help with deployment and debugging of mixed reality apps.

Microsoft Visual Studio is an industrial-strength development tool for mixed reality

The incredible flexibility, extensibility, and accessibility of Visual Studio lends itself well to being a foundational part of this evolving platform. As new libraries and SDKs come online for mixed reality, you can be they’ll be available within Visual Studio first.

SUPERPOWER – Visual Studio has always been known for its world-class debugging tools, whether using an emulator or remotely on device. IntelliSense rules.

Apple Xcode

There are some holographic prototyping techniques that Apple Xcode and its associate augmented reality development SDK called ARKit would be perfect for. Being able to simulate putting a hologram into a physical space using your iPhone or iPad as the viewing device is more than handy. Even though I wouldn’t consider the augmented reality objects ARKit displays as true holograms, they are great stand-ins. The fact that they’re available on such ubiquitous hardware makes this a very attractive choice for trying out quick thoughts about placement and feel if you don’t have access to the HoloLens or other mixed reality headsets.

Xcode and the ARKit enable you to create Augmented Reality apps for iOS devices

Another advantage to using Apple Xcode to develop some of your prototypes is taking the opportunity to see how iOS and MacOS apps fit into the mixed reality ecosystem. There are plenty of places where using the most popular services from inside your mixed reality experience is exactly the right thing to do. No better way to investigate those possible outcomes than trying it out in-context on a device.

SUPERPOWER – Xcode is a fantastic development environment for coding in, and let’s face it – iOS is still a fun platform to develop and test on.

GitHub

You might not think right off that a popular source code repository and revision management system like GitHub is a key tool for mixed reality development but given the number of changes and updates needed to move your projects quickly forward, this service is almost a must have.

Besides the obvious benefits of keeping our projects organized and shareable, there are so many occasions where having a previous version of code or assets would have saved us an enormous amount of time in the form of do-overs and recoding.

GitHub makes it easy to version control and share your project with other developers

Another benefit of using a service like GitHub is the ability to create your own public profile that details your interests, skills, and recent projects. What a great way to find people who share similar tech interests. Collaboration and sharing are highly encouraged in this type of tool, but you can also protect your projects as private if you need space to explore on your own.

SUPERPOWER – GitHub is the industry’s best practice for keeping code accessible, backed up, and shareable among collaborators. You want to be able to easily rollback or fork branches for new explorations with no fear.

Voice Recorder

There’s no faster way to get your ideas across to other people than just speaking them out loud. Straight from your thoughts to their ears, no tools to get in the way of your flow. Storytellers love telling stories obviously, so let’s leverage that for envisioning ideas.

Using a smartphone to record your narrative is fast and (mostly) free

There are a gazillion free voice recording apps for just about every kind of smartphone there is, so no reason to not have one ready to record your thoughts whenever the ideas hit you. These apps all allow you to get the recording onto your computer or share with other people.

SUPERPOWER – We love hearing people’s voices telling us engaging stories.

Smartphone Video

Just about every smartphone has a decent quality video camera built-in. People use them without even thinking to record events and share snippets of video on social media. We often have friends or family act silly and do things they’d normally not do just because the video is being recorded. We can use that to our advantage by having people help us act out the scenes behind our holographic ideas. Some people will be the holograms and others the participants. Have some fun with it.

Getting people (or things) to help act out your holographic scenarios isn’t that hard

We are all so familiar with how to share these recorded videos with teammates, collaborators, or prospective audiences, this overly obvious tool should become a very popular way of getting early ideas in front of people.

SUPERPOWER – Video is a super effective way to get the gist of an experience across because you are using real people in physical spaces who can react to direct and ad lib where necessary.

Autodesk SketchBook

Autodesk SketchBook is still one of my favorite tools. Here’s why… even the most rudimentary and quickly drawn storyboard sequence has more power to convey and move people than most professionally written commercials. There’s a natural connection formed when we see hand drawn artwork of any kind – even better when it’s trying to tell us a story. Perhaps it’s all those childhood memories of a happier time drawing away for the pure pleasure of it. Not sure. But, I do know when storyboards are drawn in this way the command people’s attention, at least for a short period. And as storytellers, that’s all we can ask for.

Autodesk SketchBook is a great digital storyboarding tool for envisioning

The other important thing about SketchBook is it was designed for the pen. So much better than trying to draw with a mouse! It’s like night and day if you’ve never tried drawing that way. Another key to getting the most out of SketchBook is using the layers just like you would in an Adobe product. It gives you a simple way to draw the background frames, then start layering elements and characters on top, successively. Don’t like what you did? Erase it or just hide that layer. SketchBook Pro is an upgrade that gives you unlimited layers among other amazing features. I’d buy it just for the unlimited layers, they are so fundamental to using the tool.

SUPERPOWER – SketchBook lets you make that deep emotional connection with your story by the nature of it being hand drawn. And it’s a world-class digital sketching tool to boot.

Filmmaker Live

What if you could see a mixed reality sequence unfold just like you were directing a motion picture? What if nothing had to be prebuilt before you jumped in and starting building sets and adding actors? Filmmaker Live by Accelerated Pictures is a new breed of cinematic tool that lets you do just that. It brings new-to-world envisioning capabilities to the iPad, Surface tablet, or even smartphones. The app puts you in the director’s chair and gives you the power of an entire film crew to block out scenes, figure out timing, and actually run the actors through their lines and cues. It’s a completely different way of thinking about envisioning. Brilliant and fun.

Filmmaker Live is a cinematic tool for storytelling in MR and VR

By taking the perspective of a cinematic production, we not only get a glimpse into that world of filmmaking, but our world of mixed reality design and development takes on a different feel – more along the lines of what a cinematographer may experience as they search for the right camera angles and aesthetic look. It’s a fascinating learning experience and very helpful in thinking about how our participants and onlookers will experience the action.

SUPERPOWER – Filmmaker Live gives us a glimpse into the future of envisioning. By taking the viewpoint of a director on set, we can focus on the cinematic elements of our story – something that’s been more difficult to understand and practice before now.

PowerPoint

Last but certainly not least, the venerable presentation app Microsoft PowerPoint is one of your best methods for getting the story across in its early form. I’m not saying you should put together a densely-packed slide with bullet points telling your story in stilted sentence fragments, no. I am suggesting that you may not know how incredibly powerful the much-maligned program is at conveying stories using imagery and animation. In fact, I’d bet that you have never really witnessed the full storytelling capabilities of PowerPoint or Keynote for that matter.

Your story ideas can be animated and annotated quickly in PowerPoint

These programs have the ability to embed images, video, sound, text, ink, links, and true interactivity to create stunningly persuasive stories. The animation capabilities of individual elements or groups is way more sophisticated than most people realize. Dozens of special effects and timings can be applied to slide transitions, clicks or touches, and auto-playing sequences.

And the best part of using this type of presentation tool is the ability for anyone else to edit it easily. The collaborative nature of our work makes PowerPoint an incredibly useful tool to work with.

SUPERPOWER – The ability for anyone to create and share these story ideas makes PowerPoint or other presentation software a great tool choice. The super bonus here is that anyone else can edit the story so easily and send it back, making collaboration simple.