Video stabilization is a classic problem in computer vision. The idea is simple – you have a video stream that's shaky, and you're trying to identify the best way to negate the motion of the camera to produce a smooth motion across images. The resulting video is easier to view and has a cinematic look.
Over the years, there have been a number of approaches being tried to solve this. Videos have traditionally been stabilized by using data available only from images, or using specialized hardware to negate physical motion in the camera. Gyroscopes in mobile devices are the middle ground between these two approaches.
In this chapter, we'll cover the following:
- An Android camera application to record media and gyroscope traces
- Using the video and gyroscope trace to find mathematical unknowns
- Using the physical camera unknowns to compensate for camera motion
- Identifying rolling shutter in the camera
Note
Rolling shutter on a camera sensor produces unwanted effects. We'll cover this in detail in a later section. Also, refer to Chapter 1, Getting the Most out of Your Camera System, for a detailed discussion.
Before we get started, let's take a look at some techniques that were used in the past to solve this problem.
Video stabilization with images alone seems like the first logical step. Indeed, initial research on stabilizing video captured by cameras was based on using information readily available from the camera sensors to understand how they move image by image.
The idea is to find keypoints in an image sequence to understand how they move image by image. Keypoints are pixel locations on an image that match these criteria:
- Keypoints should be easily recognizable and distinguishable from each other. Corners of objects are good keypoints while a point on a blank wall is not.
- It should be possible to track keypoints across multiple images to calculate motion. You should be able to tell exactly where the keypoint has moved from one frame to another.
- For performance, identifying these keypoints should be fast and memory-efficient. This is usually a bottleneck on low memory and low power devices.
Research with these criteria led to several unique approaches like including some famous algorithms such as SIFT, SURF, ORB, FREAK, and so on. These techniques often work well.
OpenCV comes with several common keypoint detectors. These include ORB, FAST, BRISK, SURF, and so on. Check the 2D Features Framework documentation page for more information on how to use these at: http://docs.opencv.org/3.0-beta/modules/features2d/doc/feature_detection_and_description.html.
The keypoint detectors, however, have their own set of drawbacks. Firstly, the results of stabilization are highly dependent on the quality of the images. For example, a low resolution image might not produce the best set of features. Out of focus and blurry images are another concern. This puts a constraint on the types of sequences that can be stabilized. A scene with a clear blue sky and yellow sand might not contain enough features.
A surface with a repetitive pattern will confuse the algorithm because the same features keep showing up in different positions.
Secondly, if there is a large amount of motion in the image (such as people walking in the background or a truck moving across the road), the stabilization will be skewed because of it. The keypoint is tracking the moving object and not the motion of the camera itself. This limits the types of videos that can be successfully stabilized with such an approach. There are ways of getting around such constraints – however, it makes the algorithm more complex.