What we have right now is a very barebones implementation of video stabilization. There are a few more things you can add to it to make it more robust, more automated and the output more pleasing to the eye. Here are a few things to get you started.
In this chapter, we've hard-coded the axes of the gyroscope. This might not be the case for all mobile phone manufacturers. Using a similar calibration technique, you should be able to find an axes configuration that minimizes errors across the video.
We've hard-coded the direction of the rolling shutter. Using specific techniques (like blinking an LED really fast at the camera), it is possible to estimate the direction of the rolling shutter and incorporate that into the calibration code. Certain camera sensors don't have the rolling shutter artifacts at all. This test can also identify if such a sensor is being used.
Now that we've stabilized the video, we can speed up (or slow down) the video much better. There are commercial packages that do similar tasks – now your OpenCV code can do it too!
You will have to calibrate every new device type you come across. If you move from one device type (say, a Samsung S5 to an iPhone 6), you'll have to run a calibration for this combination of lens and sensor. However, moving between different devices of the same kind does not require a re-calibration (such as moving from one iPhone 6 to another). If you're able to collect enough calibration results, your code can run perfectly on pretty much any device.
Currently, we're only using rotations. This means that if you shake the camera in a single plane, the algorithm won't do much. By using inputs from the accelerometer and using the translation of keypoints, it should be possible to compensate for translation as well. This should produce higher quality video.
Here are some additional things to keep in mind while working with Python and computer vision in general. They should help speed up your work and keep you safe from unexpected crashes!
Python gives us a neat way to store Python objects as files on disk. In our project, we have the gyroscope calibration class. This class stores information like the video dimensions and keypoints across different frames. Calculating this information from scratch everytime you want to test your code is cumbersome. You can easily pickle this object into a file and read back the data when required.
Here is some sample code for pickling the video object in our code:
import pickle
fp = open("/path/to/file.data", "w")
videoObj = GyroVideo(mp4)
pickle.dump(videoObj, fp)
fp.close()To read the object back into the script:
import pickle
fp = open("/path/to/file.data", "r")
videoObj = pickle.load(fp)
fp.close()This saves time when iterating on code and verifying if something works as expected.
When working with videos, you most often end up using something like the VideoWriter class from OpenCV. You feed it frames and it writes out a video file. While this is a perfectly valid way to get things done, you have more control if you write out individual frames to disk and then use a video encoder to combine the images into a video stream.
A simple way to combine multiple images is to use ffmpeg.
ffmpeg -i /tmp/image%04d.png -f image2 output.mp4
In the project, we're trying to stabilize video – thus we're calculating the delta between the actual gyroscope signal and a smoothed out version of the signal.
You might want to try it out with just the actual gyroscope signal; this will totally keep the video still. This might be useful for situations where you want the camera to appear completely still.