ROS provides basic support for Computer Vision. First, drivers are available for different cameras and protocols, especially for FireWire (IEEE1394a or IEEE1394b) cameras. An image pipeline helps with the camera calibration process, distortion rectification, color decoding, and other low-level operations. For more complex tasks, you can use OpenCV and the cv_bridge and image_transport libraries to interface with it and subscribe and publish images on topics. Finally, there are several packages that implement algorithms for object recognition, augmented reality, visual odometry, and so on.
Although FireWire cameras are best integrated in ROS, it is not difficult to support other protocols, such as USB and Gigabit Ethernet. Since USB cameras are usually less expensive and easier to find, in this chapter we discuss several available options, and we will also provide a driver that integrates seamlessly in the image pipeline, using the OpenCV video capture API.
The camera calibration and the result integration in the image pipeline will be explained in detail. ROS provides GUIs to help with the camera calibration process using a calibration pattern. Furthermore, we will cover stereo cameras and explain how we can manage rigs of two or more cameras, with more complex setups than a binocular camera. Stereo vision will also let us obtain depth information from the world, up to a certain extent and depending on certain conditions. Hence, we will also see how to inspect that information as point clouds and how to improve its quality to the best possible extent for our camera's quality and its setup.
We will also explain the ROS image pipeline, which simplifies the process of converting the RAW images acquired by the camera into monochrome (grayscale) and color images; this sometimes requires you to debayer the RAW images if they are codified as a Bayer pattern. If the camera has been calibrated, the calibration information is used to rectify the images, that is, to correct the distortion.
For stereo images, since we have the baseline the left and right cameras, we can compute the disparity image, which allows us to obtain depth information and a 3D point cloud once it has been fine-tuned; here, we will also give you tuning advice, as this can be quite difficult for low-quality cameras that sometimes require good calibration results beforehand. Finally, by using OpenCV inside ROS, even though it's only version 2.x (version 3.x is not yet supported), we have the ability to implement a wide range of Computer Vision and machine learning algorithms, or we can even run some algorithms or examples already present in this library. However, we will not cover the OpenCV API, which is outside the scope of this section. We advise the reader to check the online documentation (http://docs.opencv.org) or any book about OpenCV and Computer Vision. We will also simply show you how you can use OpenCV in your nodes, with examples of feature detection, descriptor extraction, and matching to compute the homography between two images. Additionally, this chapter will finish with a tutorial that will show you how to set up and run a visual odometry implementation integrated into ROS, the viso2_ros wrapper of the libviso2 visual odometry library, using a stereo pair built with two cheap webcams attached to a supporting bar. Other visual odometry libraries will be mentioned, for example, fovis, along with some advice on how to start working with them and how to improve the results with RGBD sensors (such as Kinect), sensor fusion, and additional information on monocular vision.