Previously, in Chapter 10, “Image Detection,” we looked at how we can get our AR mobile app to recognize and respond to predefined 2D images when they are detected in our scenes. Well, in a similar way, we can get our app to respond to predefined 3D objects. It’s a more complex process than 2D image recognition; however, ARKit makes this possible. All we need to do is plumb the functionality together.

This process requires two parts, the first to enable the user to use the app to scan a 3D object and store some of its “spatial data,” then a second part to use that spatial data again to detect the object in the scene.

While this chapter gives an overview of the concept, the code required to demonstrate this is too long to include in full. Fortunately, Microsoft have created an open source Xamarin Body Detection sample app that we can download and try out.

The example app and resulting screenshots discussed and shown in this chapter are from the Microsoft Xamarin.iOS Scanning App sample here:

https://docs.microsoft.com/en-us/samples/xamarin/ios-samples/ios12-scanninganddetecting3dobjects/

Scanning and Saving Object Spatial Data

During scanning, an instance of ARObjectScanningConfiguration configuration is used by the Session as can be seen in Listing 15-1.

var configuration = new ARObjectScanningConfiguration();

sceneView.Session.Run(configuration);

Listing 15-1

Using ARObjectScanningConfiguration

When the example app is ran, you will see that during the scanning stage a bounding box is used to denote the area in which the 3D object we wish to scan should be located as seen in Figure 15-1. By default, it detects a horizontal plane and places the bottom of the bounding box on top of it. It is possible to increase the size and location of the bounding box using pinch and pan touch gestures.

Once you are happy that your 3D object is located within the bounding box, press the Scan button to store the spatial data for later use. During the scan, the app asks you to move around the object to allow scanning and subsequent recognition from different angles. This process of scanning from different angles makes the bounding box walls go solid as shown in Figure 15-2. When you are happy that you have scanned the object from a sufficient number of different angles, press Finish.

Once the scan is complete, the scanned object is stored as an ARReferenceObject in the app for later reference.

Recognizing Scanned Objects

In order to recognize the 3D object in the scene, we need to retrieve (or at least reference) the previously scanned and saved spatial data for the 3D object and use it to allow the app to detect any objects that match it.

When you are ready, press the “Test” button in the app which will begin detection of the 3D object you have scanned in the scene.

If the object is detected in the scene (using the code in Listing 15-2), the app notifies you and tells you how long it took to detect it (pretty quick in my opinion) as shown in Figure 15-3.

public override void DidAddNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)

{

    if (anchor != null && anchor is ARObjectAnchor)

    {

        var objectAnchor = anchor as ARObjectAnchor;

        if (objectAnchor.ReferenceObject == referenceObject)

        {

            // Successful detection, do something

        }

    }

}

Listing 15-2

The code that fires when the object is detected

We could do anything once we have successfully detected the object, we could show a message as shown in Figure 15-3, or we could add additional nodes to or next to the detected object.

Things to Try

Here are some ideas of things to try using Object Detection.

Scan and store multiple objects.

See if you can scan and store multiple different objects.

Scan a product and retrieve/display product information upon successful detection.

Scan and save the 3D characteristics of a product (such as a cuddly toy); then when it is detected, display additional information next to it such as product details, description, price, and so on.

Scan someone’s head to see how accurate recognition is.

Try scanning someone’s head and seeing if Object Detection can recognize it.

See how small/big an object you can scan and detect.

Play around with scanning very small or very large objects to see if there are limitations to how well object detection works with very small or very large objects.

Change the color of the bounding box.

Try changing the color or other aspects of the bounding box used for scanning and detection.