Kinect was developed by Microsoft as a motion sensing device for video games, but it works well as a mapping tool for TurtleBot. Kinect is equipped with an rgb camera, a depth camera, an array of microphones, and a tilt motor.

The rgb camera acquires 2D color images in the same way in which our smart phones or webcams acquire color video images. The Kinect microphones can be used to capture sound data and a three-axis accelerometer can be used to find the orientation of the Kinect. These features hold the promise of exciting applications for TurtleBot, but this book will not delve into the use of these Kinect sensor capabilities.

Kinect is connected to the TurtleBot netbook through a USB 2.0 port (USB 3.0 for Kinect v2). Software development on Kinect can be done using the Kinect Software Development Kit (SDK), freenect, and Open Source Computer Vision (OpenCV). The Kinect SDK was created by Microsoft to develop Kinect apps, but unfortunately, it only runs on Windows. OpenCV is an open source library of hundreds of computer vision algorithms that provides support for 2D image processing. 3D depth sensors, such as the Kinect, ASUS, PrimeSense, and RealSense are supported in the VideoCapture class of OpenCV. Freenect packages and libraries are open source ROS software that provides support for Microsoft Kinect. More details on freenect will be provided in an upcoming section titled Configuring TurtleBot and installing 3D sensor software.

Microsoft has developed three versions of Kinect to date: Kinect for Xbox 360, Kinect for Xbox One, and Kinect for Windows v2. The following figure presents images of the variations and the subsequent table shows their specifications:

Microsoft Kinect versions

Microsoft Kinect version specifications:

fps: frames per second

ASUS Xtion, Xtion PRO, PRO LIVE, and the recently released Xtion 2 are also 3D vision sensors designed for motion sensing applications. The technology is similar to the Kinect, using an IR projector and a monochrome CMOS receptor to capture the depth information.

The ASUS sensor is connected to the TurtleBot netbook through a USB 2.0 port (or USB 3.0 for the Xtion 2) and no other external power is required. Applications for the ASUS Xtion sensors can be developed using the ASUS development solution software, OpenNI2, and OpenCV. OpenNI2 packages and libraries are open source software that provides support for ASUS and PrimeSense 3D sensors. More details on OpenNI2 will be provided in the following section, Configuring TurtleBot and installing 3D sensor software.

The following figure presents images of the ASUS sensor variations and the subsequent table shows their specifications:

ASUS Xtion and PRO versions

ASUS Xtion and PRO version specifications:

PrimeSense was the original developer of the 3D vision sensing technology using near-infrared light. They also developed the NiTE software that allows developers to analyze people, track their motions, and develop user interfaces based on gesture control. PrimeSense offered its own sensors, Carmine 1.08 and 1.09, to the market before the company was bought by Apple in November 2013. The Carmine sensor is shown in the following image. ROS OpenNI2 packages and libraries also support the PrimeSense Carmine sensors. More details on OpenNI2 will be provided in the upcoming section titled Configuring TurtleBot and installing 3D sensor software:

PrimeSense Carmine

PrimeSense has two versions of the Carmine sensor: 1.08 and the short range 1.09. The preceding image shows how the sensors look and the subsequent table shows their specifications:

Intel RealSense technology was developed to integrate gesture tracking, facial analysis, speech recognition, background segmentation, 3D scanning, augmented reality, and many more applications into an individual's personal computer experience. For TurtleBot, the RealSense series of 3D cameras can be combined with powerful and adaptable machine perception software to give TurtleBot the capability to navigate on its own. Given that these cameras are powered by a USB 3.0 interface, connecting to the TurtleBot's netbook or SBC is straightforward. Any of these cameras will provide color, depth, and IR video streams for navigation.

In the following paragraphs, we will introduce and describe the latest Intel RealSense series cameras: R200, SR300, and ZR300. Intel's Euclid Development Kit integrates the RealSense camera technology with a power computer to provide a compact all-in-one imaging system. The Euclid will be introduced and described also in a subsequent paragraph.

Intel RealSense Camera R200: It is a long range, stereo vision 3D imaging system. The R200 has two active IR cameras positioned on the left and right of an IR laser projector. These two cameras provide the ability to implement stereo vision algorithms to calculate depth. Images detected by these cameras are sent to the R200 application-specific integrated circuit (ASIC). The ASIC is custom designed to calculate the depth value for each pixel in the image. The IR laser projector is a class-1 laser device that emits additional illumination to texture a scene for better stereo vision performance. The R200 also has a full HD color imaging sensor for color vision data. The R200 has the added advantage of outdoor use because of the IR projection system. Adjustments for a fully sunlit environment might be necessary. The product datasheet for the R200 can be found at the website: https://software.intel.com/sites/default/files/managed/d7/a9/realsense-camera-r200-product-datasheet.pdf.

Intel RealSense Camera SR300: It's a short range, coded light 3D imaging system that is optimized for background segmentation and facial tracking applications. The SR300 is the second-generation improvement over the Intel RealSense F200 camera in depth range and higher quality depth data. The F200 and SR300 imaging systems use embedded coded light IR for depth measurement and full HD color for imaging. The SR300 camera contains an IR laser projector system and a fast video graphics array (VGA) IR sensor that work together using a coded light pattern projected on a 2D array of monochromatic pixel values. This coded light method provides depth imaging with reduced exposure time allowing for faster dynamic motion capture. This technology has two significant drawbacks:

The camera can operate to produce independent color, depth, and IR video streams or synchronizes these video data streams to a client computer system. The SR300 product datasheet can be found at: https://software.intel.com/sites/default/files/managed/0c/ec/realsense-sr300-product-datasheet-rev-1-0.pdf.

Intel RealSense Camera ZR300: This variant is a mid-range, stereo vision 3D imaging system similar to the R200. The ZR300 stereo vision is implemented with a left IR camera, a right IR camera, and an IR laser projector. The image data is received from the cameras on the ZR300 ASIC, which calculates the depth values for each pixel in the image. The ZR300 also incorporates a six-degree of freedom IMU and a fisheye optical sensor. The ZR300 image streams have timestamp synchronization to a 50µs reference clock. The ZR300 product datasheet can be found at: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/ZR300-Product-Datasheet-Public.pdf.

Intel Euclid Developer Kit: The Euclid combines a ZR300 camera with a full featured computer into the compact size of a candy bar. The computer is an Intel Atom x7-8700 Quad-Core processor with 4 GB of memory and 32 GB of storage. It comes pre-installed with Ubuntu 16.04 and ROS Kinetic Kame. The Euclid has both Wi-Fi and Bluetooth communication and USB 3.0, Micro HDMI, and USBOTG/Charging ports. Sensors within the Euclid include an IMU, barometric pressure sensor, GPS, and proximity sensor. It can be powered by its own lithium polymer battery pack or with external power. The Euclid comes ready to use out of the box and the developer uses a web interface from a phone or computer to run, monitor, and manage the application. Many more details on the Euclid Developer Kit are available at the following websites:

Software development tools for the RealSense cameras are available with any of the following software packages:

More details on ROS RealSense software is provided in the upcoming section titled Configuring TurtleBot and installing 3D sensor software.

The Intel RealSense cameras currently considered for integrating with TurtleBot are the R200, SR300, ZR300, and the Euclid Development Kit as shown in the following figure. Specifications for the R200, SR300, and ZR300 cameras are shown in the subsequent table:

Intel RealSense cameras and the development kit

Intel RealSense camera's specifications:

The TurtleBot 3 LDS rotates in a continuous 360 degrees to collect 2D distance data that is transmitted to the SBC. The hardware configurations are described in the Introducing TurtleBot 3 section in Chapter 3, Driving Around with TurtleBot. The distance data is used for obstacle detection, SLAM mapping, and navigation. The LDS uses a Class 1 laser with a semiconductor laser diode light source. Basic specifications for the device can be found at http://wiki.ros.org/hls_lfcd_lds_driver?action=AttachFile&do=view&target=LDS_Basic_Specification.pdf.

The software driver is contained in the ROS package hls-lfcd-lds-driver. The LDS starts operating as part of the basic TurtleBot 3 operation described in the Using keyboard teleoperation to move TurtleBot 3 section in Chapter 3, Driving Around with TurtleBot. The following figure shows a side view and top view of the LDS and the subsequent table shows its specifications:

Hitachi-LG LDS

LDS specifications:

TurtleBot uses 2D and 3D sensing for autonomous navigation and obstacle avoidance, as described later in this chapter. Other applications that these 3D sensors are used in include 3D motion capture, skeleton tracking, face recognition, and voice recognition.

There are a few drawbacks that you need to know about when using the infrared 3D sensor technology for obstacle avoidance. These sensors have a narrow imaging area of about 58 degrees horizontal and 43 degrees vertical (typically), although those for Kinect for Windows v2 and Xtion 2 are slightly larger. These sensors can also not detect anything within the first 0.5 meters (~20 inches). Highly reflective surfaces, such as metals, glass, or mirrors cannot be detected by the 3D vision sensors.