Cover image for Kinect for Windows SDK Programming Guide

Kinect for Windows SDK Programming Guide

Author Abhijit Jana
Release Date: 2012/12/01
ISBN: 9781849692380
Topic:
0%
122
Chapters
0-1
Hours read
0k
Total Words
    Start Reading Now    
       Add to Wishlist       
View table of contents

Book Description

Build motion-sensing applications with Microsoft’s Kinect for Windows SDK quickly and easily

  • Building application using Kinect for Windows SDK.
  • Covers the Kinect for Windows SDK v1.6
  • A practical step-by-step tutorial to make learning easy for a beginner.
  • A detailed discussion of all the APIs involved and the explanations of their usage in detail
  • Procedures for developing motion-sensing applications and also methods used to enable speech recognition

In Detail

Kinect has been a game-changer in the world of motion games and applications since its first release. It has been touted as a controller for Microsoft Xbox but is much more than that. The developer version of Kinect, Kinect for Windows SDK, provides developers with the tools to develop applications that run on Windows. You can use this to develop applications that make interaction with your computer hands-free.

This book focuses on developing applications using the Kinect for Windows SDK. It is a complete end to end solution using different features of Kinect for Windows SDK with step by step guidance. The book will also help you develop motion sensitive and speech recognition enabled applications. You will also learn about building application using multiple Kinects.

The book begins with explaining the different components of Kinect and then moves into to the setting up the device and getting thedevelopment environment ready. You will be surprised at how quickly the book takes you through the details of Kinect APIs. You will use NUI to use the Kinect for Natural Inputs like skeleton tracking, sensing, speech recognizing.

You will capture different types of stream, and images, handle stream event, and capture frame. Kinect device contains a motorized tilt to control sensor angles, you will learn how to adjust it automatically. The last part of the book teaches you how to build application using multiple Kinects and discuss how Kinect can be used to integrate with other devices such as Windows Phone and microcontroller.

Table of Contents

  1. Kinect for Windows SDK Programming Guide
    1. Table of Contents
    2. Kinect for Windows SDK Programming Guide
    3. Credits
    4. About the Author
    5. Acknowledgement
    6. About the Reviewers
    7. www.PacktPub.com
      1. Support files, eBooks, discount offers and more
      2. Why Subscribe?
      3. Free Access for Packt account holders
      4. Instant Updates on New Packt Books
    8. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    9. 1. Understanding the Kinect Device
      1. Components of Kinect for Windows
        1. Inside the Kinect sensor
          1. The color camera
          2. IR emitter and IR depth sensor
            1. How depth data processing works
          3. Tilt motor
          4. Microphone array
          5. LED
      2. Kinect for Windows versus Kinect for Xbox
      3. Where can you use Kinect
      4. Summary
    10. 2. Getting Started
      1. System requirements for the Kinect for Windows SDK
        1. Supported operating systems
        2. System configuration
        3. The Kinect sensor
          1. The Kinect for Windows sensor
          2. The Kinect for Xbox sensor
        4. Development tools and software
      2. Evaluation of the Kinect for Windows SDK
      3. Downloading the SDK and the Developer Toolkit
      4. Installing Kinect for Windows SDK
        1. Installing the Developer Toolkit
        2. Components installed by the SDK and the Developer Toolkit
          1. Kinect management service
        3. Connecting the sensor with the system
          1. Verifying the installed drivers
            1. Not able to view all the device components
            2. Detecting the loaded drivers in Device Manager
      5. Testing your device
        1. Testing Kinect sensors
        2. Testing the Kinect microphone array
      6. Looking inside the Kinect SDK
      7. Features of the Kinect for Windows SDK
        1. Capturing the color image data stream
        2. Processing the depth image data stream
          1. Near Mode
        3. Capturing the infrared stream
        4. Tracking human skeleton and joint movements
        5. Capturing the audio stream
        6. Speech recognition
        7. Human gesture recognition
        8. Tilting the Kinect sensor
        9. Getting data from the accelerometer of the sensor
        10. Controlling the infrared emitter
      8. The Kinect for Windows Developer Toolkit
        1. The Face Tracking SDK
        2. Kinect Studio
      9. Making your development setup ready
      10. The Coding4Fun Kinect Toolkit
      11. Summary
    11. 3. Starting to Build Kinect Applications
      1. How applications interact with the Kinect sensor
        1. Understanding the classification of SDK APIs
      2. Kinect Info Box – your first Kinect application
        1. Creating a new Visual Studio project
          1. Adding the Kinect libraries
        2. Getting the Kinect sensor
          1. The Kinect sensor
          2. Defining the Kinect sensor
          3. The collection of sensors
        3. Starting up Kinect
          1. Inside the sensor.Start() method
          2. Enabling the data streams
        4. Identifying the Kinect sensor
          1. Initializing the sensor using device connection ID
        5. Stopping the Kinect sensor
          1. The Stop() method does the clean-up operation
        6. Displaying information in the Kinect Info Box
          1. Designing the Info Box UI
          2. Binding the data
            1. A quick look at INotifyPropertyChanged
            2. Using INotifyPropertyChanged for data binding
            3. Setting the DataContext
            4. Setting up the information
        7. That's all!
      3. Dealing with the Kinect status
        1. Monitoring the change in sensor status
          1. Properties of the StatusChangedEventArgs class
        2. Resuming your application automatically
        3. Building KinectStatusNotifier
          1. Setting up an application
          2. How it works
        4. Using KinectStatusNotifier
          1. Test it out
      4. Summary
    12. 4. Getting the Most out of Kinect Camera
      1. Understanding the Kinect image stream
        1. Types of color images
      2. Different ways of retrieving the color stream from Kinect
        1. Event model
        2. Polling model
      3. KinectCam – a Kinect camera application
        1. Setting up the project
        2. Designing the application – XAML and data binding
        3. Capturing color image from the Kinect camera
      4. Enabling the color stream channel
        1. Enabling a channel with the image format
        2. Choosing the image format
        3. Disabling the color stream channel
        4. Attaching the event handler
        5. Processing the incoming image frames
        6. Rendering image frames on the UI
        7. Running the KinectCam
      5. Looking inside color image stream helpers
        1. The ColorImageStream class
        2. The ColorImageFrame class
      6. Capturing frames on demand
      7. Extending the KinectCam
        1. Getting the frame number
        2. Changing image format on the fly
          1. Bind available image formats
          2. Changing the color image format
        3. Calculating frame rate
          1. How to calculate frame rate
        4. Capturing and saving images
          1. Saving images periodically
          2. Trying to save image frames directly
        5. Changing the sensor elevation angle
          1. Maximum and minimum elevation angles
          2. Adjusting the Kinect sensor angle
        6. Playing around the color pixels
          1. Applying RGB effects
          2. Making grayscale effects
          3. Inverting the color
      8. Applying more effects to the camera
        1. Applying the backlight compensation mode
        2. Applying slow motion effects
        3. Kinect Camera Effects – application
      9. Seeing in low light
      10. Making your application perform better
      11. Using the Coding4Fun toolkit
        1. Installing the Coding4Fun Kinect toolkit
          1. Using assembly
          2. Using the NuGet package
        2. Using Coding4Fun Kinect libraries in your application
      12. Summary
    13. 5. The Depth Data – Making Things Happen
      1. Understanding the depth data stream
        1. Depth data – behind the scenes
        2. Stereo triangulation
      2. Capturing and processing depth data
        1. Enabling the depth stream channel
        2. Attaching the event handler
        3. Processing the depth frames
        4. Depth data at first look
      3. Looking inside depth image stream helpers
      4. Depth data and distance
        1. How the distance is calculated
        2. Getting the distance from a particular pixel
        3. Accessing the range of distance
        4. Colorize depth data processing
      5. Working with depth range
        1. Special depth range values
      6. Depth data distribution
      7. Player index with depth data
        1. How player index works
        2. Identifying players
      8. Getting the depth and player index automatically
      9. A 3D view of depth data
        1. The basics of the coordinate system
        2. Basic elements of 3D graphics
        3. Setting up the project
        4. Give it a 3D effect
          1. Creating the ViewPort
          2. Using the camera
            1. Controlling the camera position
          3. Creating the 3D Model
            1. Building the mesh object
          4. Setting up the initial data points
        5. Getting the depth data from Kinect
        6. Have a look at 3D depth
      10. Summary
    14. 6. Human Skeleton Tracking
      1. How skeleton tracking works
        1. Steps to remember
      2. Skeleton tracking with the Kinect SDK
      3. Start tracking skeleton joints
        1. Tracking the right hand
          1. Setting up the project
          2. Creating a joint placeholder
          3. Get Kinect running and instantiate skeleton tracking
            1. Enabling and disabling the skeleton stream
          4. Processing the skeleton frames
          5. Mapping the skeleton joints with UI elements
          6. Running the application
          7. Adding more fun
      4. Flow – capturing skeleton data
      5. An intrusion detector camera application
        1. Adding night vision
      6. Looking inside skeleton stream helpers
        1. The skeleton frame
        2. The skeleton stream
      7. Skeleton-tracking mode
        1. Default skeleton tracking
        2. Seated skeleton tracking
          1. Using seated-skeleton tracking
          2. Points to be considered with seated-skeleton tracking
      8. Skeleton tracking in near mode
      9. The Skeleton
        1. Skeleton-tracking state
          1. Counting the number of tracked skeletons
      10. Choosing which skeleton to track
        1. Skeleton-tracking ID
        2. Monitoring changes in the skeleton
        3. Limiting tracking for the intrusion-detector camera
      11. The building blocks – Joints and JointCollection
        1. Joint-tracking state
      12. Steps to be followed for joint tracking
      13. Create your own joints data point
      14. Bones – connecting joints
        1. Bone sequence
          1. Bone sequence for a default skeleton
          2. Bone sequence for a seated skeleton
        2. Drawing bones between joints
      15. Adjusting the Kinect sensor automatically and giving live feedback to users
      16. Skeleton smoothing – soften the skeleton's movement
        1. What causes skeleton jitters
        2. Making skeleton movement softer
          1. Smoothing parameters
        3. How to check if skeleton smoothing is enabled
        4. Exponential smoothing
      17. Skeleton space transformation
      18. The Advanced Skeleton Viewer application
      19. Debugging the applications
        1. Using conditional breakpoints
        2. Using Kinect Studio
      20. Getting data frames together
      21. Summary
    15. 7. Using Kinect's Microphone Array
      1. Verifying the Kinect audio configuration
        1. Troubleshooting: Kinect USB Audio not recognizing
      2. Using the Kinect microphone array with your computer
      3. The Kinect SDK architecture for Audio
      4. Kinect microphone array
        1. The major focus area of Kinect audio
        2. Why microphone array
      5. Audio signal processing in Kinect
      6. Taking control over the microphone array
        1. Kinect audio stream
        2. Starting and stopping the Kinect audio stream
          1. Starting audio streaming after a time interval
      7. Kinect sound recorder – capturing Kinect audio data
        1. Setting up the project
        2. Designing the application – XAML and data binding
        3. Recording the Kinect audio
          1. Starting the recording
          2. Playing the recorded audio
        4. Running the Kinect Sound Recorder
      8. Processing the audio data
        1. Echo cancellation
        2. Noise suppression
        3. Automatic gain control
        4. Audio data processing with the Kinect sound recorder
      9. Sound source localization
        1. Sound source angle
          1. Confidence level
        2. Beamforming
          1. Beam angle mode
        3. Extending the Kinect Sound Recorder with sound source localization
      10. Summary
    16. 8. Speech Recognition
      1. How speech recognition works
      2. Using Kinect with your Windows PC speech recognition
      3. Beginning with Microsoft Speech API (SAPI)
        1. Steps for building speech-enabled applications
        2. Basic speech-recognition approach
        3. Building grammar
          1. Using Choice and GrammarBuilder
            1. Appending new grammars
          2. Building grammar using XML
          3. Creating grammar from GrammarBuilder
          4. Loading grammar into a recognizer
          5. Unloading grammars
      4. Draw What I Want – a speech-enabled application
        1. Setting up the project
        2. Designing the application – XAML and data binding
          1. Data binding
        3. Instantiating speech recognizer
        4. Working with the speech recognition engine
          1. Configuring Kinect audio
          2. Creating grammar
          3. Start the speech recognizer
        5. Drawing an object when speech is recognized
        6. Testing your application
      5. Summary
    17. 9. Building Gesture-controlled Applications
      1. What is a gesture
      2. Approaches for gesture recognition
      3. Basic gesture recognition
        1. Gesture-detection technique
          1. Representing skeleton joints
          2. Calculating the distance between two joints
        2. Building a clapping-hands application
          1. Setting up the project
          2. Implementing the gesture recognizer
            1. Defining the types of gestures
            2. Defining the types of recognition results
            3. Creating the event argument for the gesture
            4. Wrapping up everything with the gesture recognition engine
          3. Plugging gestures into the application
          4. Testing your application
        3. A virtual rope workout application
        4. Hands-raised-above-head gesture recognition
        5. Steps to recognize basic gestures
      4. Algorithmic gesture recognition
        1. Which gestures can be considered as algorithmic
        2. Understanding the algorithmic gesture detection approach
        3. Implementing an algorithmic gesture
          1. Adding gesture types
          2. Extending the Event argument
          3. Adding a GestureHelper class
          4. Defining the GestureBase class
          5. Implementing the SwipeToLeftGesture class
          6. Adding the ZoomIn, ZoomOut, and SwipeToRight gesture classes
          7. Implementing the GestureRecognitionEngine class
          8. Using the GestureRecognitionEngine class
          9. A demo application
        4. Making it more flexible
      5. Weighted network gesture recognition
        1. What is a neural network
        2. Gesture recognition with neural networks
        3. Jump tracking with a neural network – an example
      6. Template-based gesture recognition
      7. Building gesture-enabled controls
        1. Making a hand cursor
          1. Getting the hand-cursor point
        2. Identifying the objects
        3. Enabling action for the objects
      8. The Basic Interaction – a WPF application
      9. Key things to remember
      10. Summary
    18. 10. Developing Applications Using Multiple Kinects
      1. Setting up the environment for multiple Kinects
        1. Plugging the first Kinect sensor
        2. Plugging the second Kinect sensor
        3. Kinect sensors require an individual USB Controller
      2. Multiple Kinects – how to reduce interference
      3. Detecting multiple Kinects
        1. Getting access to the individual sensor
        2. Different ways to get a Kinect sensor's reference
      4. Developing an application with multiple Kinects
        1. Setting up the project
        2. Designing the UI
        3. Creating the KinectInfoCollection
        4. Getting information from Kinects
        5. Running the application
      5. Controlling multiple sensor status changes
        1. Extending Multiple Kinect Viewer with status change
          1. Registering and handling the status change
          2. Running the application
        2. Identifying the devices automatically
        3. Integrating with KinectStatusNotifier
        4. Capturing data using multiple Kinects
      6. Handling a failover scenario using Kinects
      7. Challenges faced in developing applications using multiple Kinects
      8. Applications where multiple Kinects can be used
      9. Summary
    19. 11. Putting Things Together
      1. Taking Kinect to the Cloud
        1. Required components
          1. Windows Azure
          2. The Windows Azure SDK
          3. The Kinect for Windows SDK
        2. Designing the solution
        3. Real-time implementations
      2. Remotely using the Kinect with Windows Phone
        1. Required components
          1. The Windows Azure Service Bus
          2. The Windows Phone SDK
        2. Designing the solution
        3. Real-time implementations
      3. Using Kinect with a Netduino microcontroller
        1. Required components
          1. Microsoft .Net Micro Framework
          2. Netduino
          3. The Netduino SDK
        2. Blinking of the on-board LED
          1. Changing the Deployment Transport
          2. Running the application
        3. Connecting Kinect to a Netduino
          1. Using an Internet connection
          2. Listening to the request
          3. Sending a request from a Kinect application
        4. Taking it further
      4. Augmented reality applications
      5. Working with face tracking
      6. Working with XNA and a 3D avatar
      7. Summary
    20. Index
3.129.39.55