OpenCV stands for Open Computer Vision. OpenCV is an open source computer vision and machine learning software library that was built with C++ and has C++, Python, Java, and Matlab interfaces to support Windows, Linux, Android, and macOS.

OpenCV mainly focuses on real-time vision applications, although it can be used for machine learning very nicely. The library has many optimized algorithms and functions to compose or support such algorithms for state-of-the-art computer vision and machine learning, with roughly 2,500. To break down the ratio here, there are roughly 500 algorithms, and the rest are functions to compose or support these algorithms.

Talking about algorithms is fun and all, but I’m sure you are more interested in knowing what these algorithms are capable of doing. The algorithms are designed to be used to detect faces, recognize faces, identify objects, detect and classify human actions in video feeds, track camera movements, move object tracking, extraction of 3D models from objects, produce point clouds from stereo cameras in 3D, stitch images together for high resolution images, quickly find the same or similar images from an image database, red-eye removal, eye tracking, detect scenery, and establish markers for augmented reality overlay, and so much more.

OpenCV can work with TensorFlow, Caffe, Torch, Yolo, Vuforia, ARCore, and ARKit with ease. OpenCV can be used for Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). You can use visual scripting options, such as PlayMaker, and have access to all methods available in OpenCV. OpenCV also works with tons of different hardware, such as HoloLens, Oculus Rift, Telepathy, Kinect, and Raspberry Pi, to name a few.

We have two different options for gaining access to OpenCV. The first one is to build from source and create our own wrappers to create a Unity plugin for OpenCV. The second option is to purchase OpenCV for Unity from the Unity Asset Store.

If you want to go the easy route and purchase the Unity extension, it will cost 70 dollars, unless you have a Unity Plus/Pro membership, which will decrease the cost to 56 dollars. The good news is that it supports Unity Version 5.6 or greater and is based on OpenCV 3.4.1.

There are sample assets packs that you can download from the Unity Asset Store or from GitHub that will give you prebuilt examples for working with HoloLens Face Detection Overlay, FaceLandmark Detection, Facemasking, Real-time Face Recognition, Markerless AR. Face Swapping, Kinect with OpenCV, Facetracking, working with Vuforia and OpenCV, and marker-based AR.

Now, the other option, in other words, the cheap route, because, let's be honest, the Unity plugin is rather pricey, especially for younger developers or those who just want to try out OpenCV to see whether it is a good fit, is to build from source. What benefits do we gain from build from source, besides the low cost?

We get to decide which aspects of the library we need, for one. This is a huge bonus, as we can remove bloat to reduce the file and compile size. However, even when doing this, you can have the same features, or even more than the Unity plugin, since you are building from source.

The downside is that there are many things you need to get the full compilation of the source taken care of, which is dependent on whether you want to use prebuilt binaries or building your own library from scratch.

To build from scratch, you will need Visual Studio with C++ modules installed, CMake, Python Version 2.7 or later, Numpy, Intel Threading Building Blocks (TBB), Intel Integrated Performance Primitives (IPP), Intel IPP Asynchronous C/C++, Eigen, CUDA Toolkit, OpenEXR, OpenNI Framework, and Doxygen.