Cover image for Programming for Mixed Reality with Windows 10, Unity, Vuforia and UrhoSharp, First Edition

Programming for Mixed Reality with Windows 10, Unity, Vuforia and UrhoSharp, First Edition

Author Dawid Borycki
Release Date: 2018/08/01
ISBN: 9781509306916
Topic:
0%
39
Chapters
0-1
Hours read
0k
Total Words
    Start Reading Now    
       Add to Wishlist       
View table of contents

Book Description

Develop AI-powered apps and games for HoloLens and mixed reality headsets

 

This is a complete guide to programming AI-powered mixed reality apps with the Windows Mixed Reality platform. Step by step, Dawid Borycki guides you through every type of mixed reality development, from simple 2D to cutting-edge apps for Microsoft HoloLens and immersive headsets. First, you'll learn to write UWP 2D apps that adapt for any Windows 10 device, transfer camera images to machine learning services, and use mobile sensor readings to control headset content. Then, building on these skills, you'll master every facet of building cross-platform 3D apps that link augmented reality content to real objects.

 

Advanced Microsoft technology expert Dawid Borycki shows how to:

 

  • Set up a mixed reality development environment with all the tools you'll need, including HoloLens and Mixed Reality simulators
  • Write Universal Windows 2D apps for all Windows 10 devices, and adjust their views and capabilities to specific hardware
  • Render media streams, acquire video from world-facing cameras, and detect human faces in those images
  • Generate descriptions of what the user sees, and present them via text or speech
  • Build AI-powered voice-controlled apps with Microsoft Cognitive Services (MCS)
  • Master core concepts and techniques for building 3D mixed reality apps with Unity
  • Format 3D objects with materials to create holograms
  • Add interactions to holograms based on user input
  • Integrate HoloLens capabilities, including air gestures, spatial sound, and mapping

Table of Contents

  1. Cover Page
  2. Title Page
  3. Copyright Page
  4. Dedication Page
  5. Contents at a Glance
  6. Contents
  7. About the Author
  8. Introduction
    1. Audience and Expected Skills
    2. Tools and Required Hardware
    3. Organization of This Book
    4. Conventions
    5. About the Companion Content
    6. Acknowledgments
    7. Errata and Book Support
    8. Stay in Touch
  9. Part I Fundamentals
    1. Chapter 1 Introduction to Windows Mixed Reality
      1. Virtual, Augmented, and Mixed Reality
        1. Virtual Reality
        2. Augmented Reality
        3. Mixed Reality
      2. Windows Mixed Reality Hardware
        1. Microsoft HoloLens
        2. Immersive Headsets
      3. Mixed Reality in Windows 10
      4. Summary
    2. Chapter 2 Development Tools
      1. Universal Windows Platform
      2. Adaptive Coding and Target Device Families
      3. .NET Core and .NET Native
      4. Microsoft DirectX
      5. Unity
      6. The Mixed Reality Toolkit
      7. Vuforia
      8. UrhoSharp
      9. Summary
    3. Chapter 3 Configuring the Development Environment, Emulators, and Hardware
      1. Setting Up the Development Environment
        1. Windows 10
        2. Visual Studio 2017
        3. Unity Editor
        4. Vuforia
        5. UrhoSharp
        6. HoloLens Emulator
        7. Mixed Reality Simulator
      2. HelloWorld Project
      3. Deploying Apps to the HoloLens Emulator
        1. Using the Emulator
        2. HoloLens Device Portal
      4. Running Apps in the Mixed Reality Simulator
      5. Summary
  10. Part II Developing 2D UWP Apps
    1. Chapter 4 UWP Basics
      1. XAML and UI
      2. Data Binding and the Model–View–ViewModel Software Architecture Pattern
      3. Implementing Data Binding
      4. Using Converters
      5. Notifying the Binding about Changes to Source Properties
      6. Adapting a View to the Platform
      7. Summary
    2. Chapter 5 Media Controls
      1. Displaying Images
      2. Capturing Video
        1. Camera Capture
        2. Face Detection
      3. Audio
        1. Playing Sounds
        2. Speech Synthesis
        3. Speech Recognition
      4. Summary
    3. Chapter 6 Microsoft Cognitive Services
      1. Exploring Microsoft Cognitive Services
      2. Implementation of the Vision Assistant
        1. Computer Vision API Client
        2. Bing Web Search API Client
        3. UI Declaration
        4. Initializing the ViewModel
      3. Supporting Voice Commands
        1. Speech Recognizer and Recognition Constraints
        2. Responding to Voice Commands
      4. Summary
    4. Chapter 7 Communicating with Devices
      1. Socket Communication
      2. Implementation
        1. Accelerometer Reading
        2. Common ViewModel
        3. Provider
        4. Consumer
      3. Testing the Solution
      4. Summary
  11. Part III Developing 3D Apps
    1. Chapter 8 3D Graphics Fundamentals
      1. Pixels, Voxels, and Coordinate Systems
      2. Vertices, Primitives, and Polygons
      3. Viewing
      4. Transformations
      5. Quaternions and Rotations
      6. Shaders and Lighting
      7. Textures and Materials
      8. Particle Systems
      9. Glossary of Terms
      10. Summary
    2. Chapter 9 Unity Basics
      1. The Unity Editor
        1. Understanding Game Objects and Components
        2. Configuring Your Unity ID and Signing In
        3. Creating a New Project
        4. The Toolbar
        5. The Hierarchy Window
        6. Scene View
        7. The Inspector Window
        8. The Project Window
      2. Creating Scenes
        1. Working with 3D Objects
        2. Using Materials
        3. Applying Prefabs
        4. Adding a Wind Zone
      3. Building and Deploying the Project
        1. Building the Project
        2. Deploying to Immersive Headsets
        3. Deploying to a HoloLens
        4. Using Holographic Emulation
      4. Mixed Reality Toolkit for Unity
      5. 3D Models
      6. Summary
    3. Chapter 10 Physics and Scripting
      1. Rigidbody
      2. Colliders
        1. 3D Primitive Colliders
        2. Mesh Colliders
        3. Using Colliders
      3. Physic Material
      4. Scripting
        1. Creating the Can Pyramid
        2. Creating a Script
        3. Script Structure
      5. Ray Casting
        1. Understanding Ray Casting
        2. Using Ray Casting to Detect Cans in the Scene
      6. Projectiles
      7. Collision Detection, Triggers, and Explosions
        1. Adding Spike Balls to the Scene
        2. Creating a Script to Blow Up the Projectile
      8. Object Tagging
      9. Handling Mixed Reality Input
        1. Motion Controllers
        2. InteractionSourceState
        3. InteractionSource
        4. InteractionSourcePose
        5. InteractionManager Events
        6. Putting It All Together
        7. Testing the App
      10. Dynamic Scene Adjustment
      11. Debugging
        1. Drawing Rays
        2. Attaching the Visual Studio Debugger to Unity
      12. Unity C# Projects
      13. Deploying the Project to the Hololens Emulator
      14. Summary
    4. Chapter 11 Animations and Navigation
      1. Animations
        1. Adding Assets and Creating the Scene
        2. Creating and Configuring the Animator Controller
        3. Animator Properties
        4. Animation Clips
        5. Script
        6. Audio Effects
      2. Navigation and Pathfinding
        1. The Nav Mesh Agent Component
        2. The Navigation Mesh Component
        3. Pathfinding
      3. Custom Animations
        1. Creating a Custom Animation Clip
        2. Animation Toolbar
        3. FPS and Curves
        4. Adding Custom Animations to Other Objects
      4. Summary
    5. Chapter 12 Understanding HoloLens-Specific Features
      1. Configuring Gaze Input
        1. Creating the Cursor
        2. Configuring the Cursor
        3. Implementing the Gaze Script
        4. Handling Messages
      2. Supporting Voice Commands
        1. Creating a Singleton Class
        2. Building the Recognizer
      3. Gestures
        1. Configuring the Scene and Project Settings with the Mixed Reality Toolkit
        2. Adding the GazeManager Script
        3. Adding the GestureHandler Script and GestureRecognizer Class
        4. Testing the App
      4. Spatial Mapping
        1. Creating a Spatial-Mapping Renderer
        2. Creating a Spatial-Mapping Collider
        3. Spatial-Mapping Prefab
        4. Hologram Placement
      5. Mixed Reality Toolkit Interfaces
      6. Summary
    6. Chapter 13 An AI-Powered 3D App
      1. Configuring Project and Scene Settings
        1. Exploring the Mixed Reality Camera Prefab
        2. Creating Quad and 3DTextPrefab Objects
      2. Setting Up Photo Capture
        1. Initializing PhotoCapture
        2. Taking Photos
        3. Releasing the PhotoCapture Object
        4. App Capabilities
      3. Using the Computer Vision API from MCS
        1. Installing HttpClient
        2. Building the VisionApiClient Script
      4. Billboarding and Tag-Along
      5. Summary
    7. Chapter 14 Attaching Holograms to Real Objects with Vuforia
      1. Creating the Project and Adding the AR Camera Prefab
      2. Adding the Image Prefab for the Image Target
        1. Adding a Hologram to the Target
        2. Testing the App
        3. Adding a Text Description
        4. Implementing Extended Tracking
      3. Implementing Augmented Navigation
        1. Using the DefaultTrackableEventHandler Script
        2. Preparing the Scene
        3. Implementing the Patrolling Script
        4. Testing the App
      4. Adding a Custom Image Database
        1. Obtaining a Vuforia Developer License Key
        2. Creating a Database
        3. Adding a Target
        4. Downloading the Database and Importing It into Unity
        5. Selecting the Custom Image Target
        6. Configuring for HoloLens
      5. Summary
    8. Chapter 15 UrhoSharp
      1. Setting Up the Project
      2. Implementing the Entry Point
      3. Adding Primitive Components and Materials
      4. Handling the Gaze
        1. Indicating Gaze Direction
        2. Raycasting
      5. Adding Gestures and Actions
        1. Tap
        2. Double-tap
        3. Manipulation
      6. Applying Physics
        1. RigidBody
        2. CollisionShape
        3. Detecting and Handling Collisions
        4. Projectiles
      7. Spatial Mapping
      8. Summary
  12. Index
3.21.248.47