Introduction

If you’ve ever watched any of the Iron Man movies starring Robert Downey Jr., you’re probably familiar with Jarvis. Jarvis, short for “Just a Rather Very Intelligent System,” is a highly advanced computerized AI system. Recently, Jarvis inspired Mark Zuckerberg to build his own AI-powered home-automation system (https://youtu.be/ZGLPxEv_EWo).

Incredibly, you can build your own wearable Jarvis-like system by combining the thrilling power of novel holographic devices like HoloLens with Microsoft Cognitive Services and Microsoft Mixed Reality. Such systems can quickly process visual content and describe it via spoken words or text displayed on screen. With just a few voice commands or air gestures, you can ask your Mixed Reality “Jarvis” system to look up what you see on the web and tell you more about it. You could use this system with AI to, say, develop a device to support visually impaired persons (https://youtu.be/R2mC-NUAmMk).

In this book you will learn how to develop AI-powered Mixed Reality apps using various strategies and technologies, including Universal Windows Platform (UWP), Microsoft Cognitive Services, Unity, Vuforia, and Xamarin UrhoSharp. First, you will learn how to set up the development environment, install the necessary tools, and use the HoloLens emulator and Mixed Reality simulators. Then you will start writing UWP two-dimensional apps that run across every Windows 10 device (including HoloLens and immersive headsets) and adjust their views and functionalities to specific devices. Subsequently, you will learn how to transfer images from the world-facing camera of the headset to the machine learning modules of the computer vision service to obtain image descriptions that are spoken by the device or displayed in text form. Afterward, you will use sensor readings from the mobile device to control content displayed on the Windows Mixed Reality headset.

After learning about 2D app development, you will move on to building 3D apps. You’ll learn how to do this from the ground up. First, you’ll explore the fundamental concepts of 3D graphics. You will then learn how to use Unity Editor to build 3D Mixed Reality apps, including setting up scenes, adding built-in and custom 3D objects, and formatting these objects with materials to create holograms. Afterward, you will learn how to make your holograms behave like real objects with various physics simulations and to interact with these holograms through air gestures and voice commands. Next, you will learn how to attach augmented reality to real objects with Vuforia. Finally, you will build a Mixed Reality app with UrhoSharp—a library for writing cross-platform 3D apps. This will enable you to transfer your skills to other platforms, such as iOS or Android.

Audience and Expected Skills

This book is devoted to developers, students, engineers, enthusiasts, designers, scientists, and researchers who would like to use their existing programming skills to develop software for Windows Mixed Reality (HoloLens and immersive headsets) with Unity, Vuforia, and UrhoSharp. I assume the reader knows fundamental aspects of C# programming and is experienced in Windows programming. I do not, however, assume any previous knowledge of Unity, Vuforia, or UrhoSharp.

Tools and Required Hardware

To implement all examples presented in this book, you will need a system that runs Windows 10 (Creators Update or later) and uses Visual Studio 2017 Community as the development environment.

Organization of This Book

The book is divided into three parts:

• Part I: Fundamentals Chapters 1 through 3 introduce Mixed Reality in Windows 10, define all related terms, show you how to prepare your development environment, and explain possible approaches to developing apps for Windows Mixed Reality.
• Part II: Developing 2D UWP Apps Chapters 4 through 7 cover 2D app development with a special focus on Universal Windows Platform (UWP) programming interfaces. Programmers familiar with UWP can easily port their existing apps to new holographic or Mixed Reality platforms and thereby extend the spectrum of their users.
• Part III: Developing 3D Apps Chapters 8 through 15 teach you how to create 3D Windows Mixed Reality apps with Unity, Vuforia, and UrhoSharp. These chapters assume no prior knowledge of those tools. The aim of this part is to guide you through all the steps and leave you with all the skills necessary to create exciting AI-powered 3D apps for Windows Mixed Reality.

Conventions

The following conventions are used in this book:

• Boldface type is used to indicate text that you type.
• Italic type is used to indicate new terms and file names.
• Code elements appear in a monospaced font.

About the Companion Content

This book includes companion code to enrich your learning experience. The companion code for this book can be downloaded from the following page:

https://github.com/dawidborycki/MixedReality-Samples.

As shown in Figure I-1, the code is partitioned into subfolders, which correspond to particular chapters. To improve book readability, in many places I refer to the companion code rather than showing the full listings, so it is good to have the solution open while reading this book.

You can also find the companion files at https://aka.ms/ProgMixedReality/downloads.

Acknowledgments

The publication of this book would not have been possible without Loretta Yates and Laura Norman, who accepted my book proposal and provided initial feedback. I am grateful to John Ray for thoroughly checking every project presented in this book and providing useful, positive comments on the content. Many thanks, too, to Kate Shoup for copyediting the book. Finally, I appreciate ongoing support from my family: my wife, Agnieszka; my daughter, Zuzanna; and my son, Ksawery. I would achieve nothing without them.

Errata and Book Support

We have made every effort to ensure the accuracy of this book and its companion content. Any errors that have been reported since this book was published are listed on our Microsoft Press site at:

https://aka.ms/ProgMixedReality/errata

If you find an error that is not already listed, you can report it to us through the same page.

If you need additional support, email Microsoft Press Book Support at [email protected].

Please note that product support for Microsoft software is not offered through the addresses above.

Stay in Touch

Let’s keep the conversation going! We’re on Twitter: @MicrosoftPress.