In this chapter, we've covered quite a bit: talking to your gyroscope, using that to find unknowns, negating the effects of camera shake and rolling shutter.

We started out by creating an Android app that uses background tasks to initiate recording media into a video file. While doing this, we figured out how to extend OpenCV's camera view class to incorporate custom UI and responsiveness. With this, you can now create very sophisticated UIs with an OpenCV backend. Along with this, we also captured the gyroscope trace and stored it in a single file. The sampling rate of the gyroscope and the media were different – however, we did not care about it at this stage. We'll let the app store a higher density of gyroscope traces (every few hundred microseconds versus every few dozen milliseconds for the media).

Once we had the media/csv pair, we used Python and the numpy/scipy libraries to calibrate the system. We had three unknowns initially: the pixel focal length of the camera, the gyroscope delay (the offset between the gyroscope recordings and the media timestamps) and the gyroscope drift.

We devised an error function that takes the expected keypoints and the transformed keypoints and returns the amount of errors. We then used this to calculate errors across the whole video. Using this, and Scipy's optimize method, we were able to find the values for these unknowns. This calibration needs to happen only once for each device type.

Then we added another parameter to our calibration—the rolling shutter. Estimating the value of this unknown was similar to the previous three, however, incorporating the undistortion was a bit tricky. We had to create a new method called meshwarp that takes a distorted grid. This method rectifies the grid and removes artifacts due to the rolling shutter. We worked on a vertical rolling shutter, however it should be easy to convert it to a horizontal rolling shutter.

We touched upon a lot of different areas: sensors, calibration, and geometric distortions. I hope this chapter gives you an insight into designing your own pipelines for working with images.