Creating an Augmented Reality Game

In this chapter, we will be exploring augmented reality (AR) using Xamarin.Forms. We will be using custom renderers to inject platform-specific code, UrhoSharp to render the scene and handle input, and MessagingCenter to pass internal messages around in the app.

The following topics will be covered in this chapter:

  • Overview of the project
  • Setting up the project
  • Using ARKit
  • Using ARCore
  • Learning how to use UrhoSharp to render graphics and handle input
  • Using custom renderers to inject platform-specific code
  • Using MessagingCenter to send messages

Technical requirements

To be able to complete this project, you will need to have Visual Studio for Mac or PC installed, as well as the Xamarin components. See Chapter 1, Introduction to Xamarin, for more details on how to set up your environment.

You cannot run AR on an emulator. To run AR, you needa physical device, along with the following software:

  • On iOS, you need iOS 11 or higher, and a device that has an A9 processor or above.
  • On Android, you need Android 9, and a device that supports ARCore.

Essential theory

This section will describe how AR works. The implementation differs slightly between platforms. Google's implementation is called ARCore, and Apple's implementation is called ARKit.

AR is all about superimposing computer graphics on top of a camera feed. This sounds like a simple thing to do, except that you have to track the camera position with great accuracy. Both Google and Apple have written some great application programming interfaces (APIs) to do this magic for you, with the help of motion sensors in your phone and data from the camera. The computer graphics that we add on top of the camera feed are synced to be in the same coordinate space as the surrounding real-life objects, making them appear as if they are part of the image you see on your phone.

An overview of the project

In this chapter, we are going to create a game that explores the fundamentals of AR. We are also going to learn how to integrate AR control in Xamarin.Forms. Android and iOS implement AR differently, so we will need to unify the platforms along the way. We will do this using UrhoSharp, an open source 3D game engine, which will do the rendering for us. This is simply made up of bindings to the Urho3D engine, which allows us to use Urho3D with .NET and C#.

The game will render boxes in AR that the user needs to tap to make disappear. You can then extend the game yourself by learning about theUrho3Dengine.

The estimated build time for this project is 90 minutes.

Setting up the project

It's time to start coding! First, however, make sure you have your development environment set up, as described in Chapter 1, Introduction to Xamarin.

This chapter will be a classicFile | New | Projectchapter, guiding you step by step through the process of creating the app. There will be no downloads required whatsoever.

Creating the project

Follow these few steps in order to create the project:

  1. Open Visual Studio and click on File |New |Project, as shown in the following screenshot:

This will open up the Create a new projectwizard.

  1. Enter xamarin formsin the search field and select theMobile App (Xamarin.Forms)template, as shown in the following screenshot, and click Next:

  1. We will be calling our application WhackABoxin this project, so let's enter that into theProject namefield, as shown in the following screenshot, and clickCreate:

  1. Next, we select a project template. Select the Blank template option to create a bare minimum Xamarin.Forms app.
  2. Uncheck the Windows (UWP) checkbox under the Platform heading, since this app will only be supporting iOS and Android.
  3. Finish the setup wizard by clicking OK, and let Visual Studio scaffold the project for you. This might take a couple of minutes. You can see the aforementioned fields and options that you need to select in the following screenshot:

Just like that, the app has been created. Let's move on to updating Xamarin.Forms to the latest version.

Updating the Xamarin.Forms NuGet packages

Currently, the Xamarin.Forms version that your project has been created with is most likely a bit old. To rectify this, we need to update the NuGet packages. Please note that you should only update the Xamarin.Forms packages and not the Android packages; updating the Android packages might cause your packages to get out of sync with each other, resulting in the app not building at all. To update the NuGet packages, go through the following steps:

  1. Right-click on our solution in the Solution Explorer.
  2. Click Manage NuGet Packages for Solution..., as shown in the following screenshot:

This will open the NuGet Package Manager in Visual Studio, as shown in the following screenshot:

To update Xamarin.Forms to the latest version, go through the following steps:

  1. Click the Updates tab.
  2. Check the Xamarin.Forms checkbox and click Update.
  3. Accept any license agreements.

The update takes, at most, a few minutes. Look at the output pane to find information about the update. At this point, we can run the app to make sure it works. We should see the text Welcome to Xamarin.Forms! in the middle of the screen.

Setting the Android target framework version to 9.0

ARCore is available from Android version 9.0 and later. We will, therefore, verify the Target Framework version for the Android project by going through the following steps:

  1. In the Solution Explorer, double-click on the Properties node under the Android project.
  2. Verify that the Target Framework version is at least Android 9.0 (Pie), as shown in the following screenshot:

If there is an asterisk next to the Target Framework name, then you will need to install that software development kit (SDK) by going through the following steps:

  1. Locate the Android SDK Manager in the toolbar. You can also find it in Tools |Android|Android SDK Manager in the menu.
  2. Click the highlighted button to open the AndroidSDK Manager, as shown in the following screenshot:

This is the control center for all SDK versions of Android that are installed on the system.

  1. Expand the SDK version you want to install. In our case, this should be at least Android 9.0 - Pie.
  1. Select the Android SDK Platform<version number> node. You can also select which emulator images will be used by the emulator to run the selected version of Android.
  2. Click Apply Changes, as shown in the following screenshot:

Adding the camera permission to Android

In order to get access to the camera in Android, we must add the required permission to the Android manifest. This can be done by following these steps:

  1. In the Solution Explorer, open up the Android project node.
  2. Double-click the Properties node to open the properties for Android.
  3. Click the Android Manifest tab on the left, and scroll down until you see the Required permissions: section.
  4. Locate the CAMERA permission and check the box.
  5. Save the file by pressing Ctrl + S or File, and then Save. You can see the aforementioned fields and options that you need to select in the following screenshot:

Now that we have configured Android, we only have one small change to make on iOS before we are ready to write some code.

Adding a camera usage description for iOS

In iOS, you need to specify why you need access to the camera. The way to do this is to add an entry to the Info.plist file in the root folder of the iOS project. The Info.plist file is an Extensible Markup Language (XML) file that you can edit in any text editor. A simpler way to do this, however, is by using the Generic PList Editor provided by Visual Studio.

Add the required Camera Usage Description using the Generic PList Editor, as follows:

  1. Locate the WhackABox.iOS project.
  2. Right-click on Info.plist and click Open With..., as shown in the following screenshot:

  1. Select Generic PList Editor and click OK, as shown in the following screenshot:

  1. Locate the plus (+) icon at the bottom of the property list.
  2. Click the plus (+) icon to add a new key. Make sure that the key is in the root of the document and not under another property, as shown in the following screenshot:

The Generic PList Editor helps you to find the right property by giving it a more user-friendly name. Let's add the value we need to describe why we want to use the camera, as follows:

  1. Open the drop-down menu on the newly created row.
  2. Select Privacy - Camera Usage Description.
  3. Write a good reason in the values field to the right, as shown in the following screenshot. The field for the reason is a free-text field, so use plain English to describe why your app needs access to the camera:

That's it. The setup of both Android and iOS is complete, and we can now focus on the fun part—writing code!

You can also open the Info.plist file in any text editor, since it's an XML file. The key's name is NSCameraUsageDescription, and it must be added as a direct child of the root node.

Defining the user interface

We are going to start off by defining the user interface (UI) that will wrap the AR components. First, we will define a custom control that we will use as a placeholder for injecting an UrhoSurface that will contain the AR components. Then, we will add this control in a grid that will contain some statistics about how many planes we have found and how many boxes are active in the world. The goal of the game is to find boxes in AR using your phone and tap on them to make them disappear.

Let's start by defining the custom ARView control.

Creating the ARView control

The ARView control belongs in the .NET Standard project since it will be a part of both applications. It's a standard Xamarin.Forms control that inherits directly from Xamarin.Forms.View. It will not load any Extensible Application Markup Language (XAML) (so it will simply be a single class), nor will it contain any functionality other than simply being defined, so we can add it to the main grid.

Go over to Visual Studio and go through the following three steps to create an ARView control:

  1. In the WhackABox project, add a folder called Controls.
  2. In the Controls folder, create a new class called ARView.
  3. Add the following code to the ARView class:
using Xamarin.Forms;

namespace WhackABox.Controls
{
    public class ARView : View
    {
    }
}

What we have created here is a simple class, without implementation, that inherits from Xamarin.Forms.View. The point of this is to make use of custom renderers for each platform, allowing us to specify platform-specific code to be inserted at the place in the XAML code where we put this control. Your project should now look as follows:

The ARView control is no good just sitting there. We need to add it to the MainPage XAML code.

Modifying the MainPage

We will be replacing the entire contents of the MainPage and adding a reference to the WhackABox.Controls namespace so that we can use the ARView control. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the MainPage.xaml file.
  2. Edit the XAML code to look like the following code. The XAML in bold represents the new elements that must be added:
<?xml version="1.0" encoding="utf-8"?>
<ContentPage  
           xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml" 
           xmlns:local="clr-namespace:WhackABox" 
xmlns:controls="clr-namespace:WhackABox.Controls"
           x:Class="WhackABox.MainPage">

<Grid>
   <Grid.ColumnDefinitions>
      <ColumnDefinition Width="*" />
      <ColumnDefinition Width="*" />
   </Grid.ColumnDefinitions>

   <Grid.RowDefinitions>
      <RowDefinition Height="100" />
      <RowDefinition Height="*" />
   </Grid.RowDefinitions>

   <StackLayout Grid.Row="0" Padding="10">
      <Label Text="Plane count" />
      <Label Text="0" FontSize="Large" 
             x:Name="planeCountLabel" />
   </StackLayout>

   <StackLayout Grid.Row="0" Grid.Column="1" Padding="10">
      <Label Text="Box count" />
      <Label Text="0" FontSize="Large" 
             x:Name="boxCountLabel"/>
   </StackLayout>

   <controls:ARView Grid.Row="1" Grid.ColumnSpan="2" />
</Grid>
</ContentPage>

Now that we have the code, let's go through it step by step, as follows:

  • First, we define a controls namespace that points to the WhackABox.Controls namespace in code. This namespace is used at the end of the XAML to locate the ARView control.
  • We then define the content element by setting it to a Grid. A page can only have one child, which, in this case, is a Grid. The Grid defines two columns and two rows. The columns split the Grid into two equal parts, where we have one row that is 100 units high at the top and one row that takes up all the available space below it.
  • We use the top two cells to add instances of StackLayoutthat contain information about the number of planes and the number of boxes in the game. The location of those instances of StackLayoutin the grid is defined by theGrid.Row=".."andGrid.Column=".."attributes. Remember that the rows and columns are zero-based. You don't actually have to add attributes for the row or column 0 but it can sometimes be a good practice, to improve code readability.
  • Finally, we have theARView control, which resides in row 1 but spans both columns by specifyingGrid.ColumnSpan="2".

The next step is to install UrhoSharp, which will be our library for rendering graphics to represent the augmented part of our reality.

Adding UrhoSharp

Urho is an open source 3D game engine. UrhoSharp is a package that contains bindings to iOS and Android binaries, enabling us to use Urho in .NET. It is a very competent piece of software, and we will only be using a very small part of it to do the heavy lifting when it comes to rendering planes and boxes in the app. We urge you to find out more about UrhoSharp to add your own cool features to the app. One great place to start is at https://github.com/xamarin/urho.

All you have to do to install UrhoSharp is download a NuGet package for each platform. The iOS platform uses the UrhoSharp NuGet package, and Android uses the UrhoSharp.ARCore package. Also, in Android, we need to add some code to wire up life cycle events, but we will get to that later. Basically, we will set up an UrhoSurface on each platform. We will access this to add nodes to the node tree. These nodes will then be rendered based on their type and properties.

First, however, we need to install the packages.

Installing the UrhoSharp NuGet package for iOS and the .NET Standard project

For the WhackABox and the WhackABox.iOS project, we need to add the UrhoSharp NuGet package. This contains everything we need for our AR app. You can add the package as follows:

  1. Right-click on theWhackABoxsolution.
  2. ClickManage NuGet Packages for Solution..., as shown in the following screenshot:

  1. This opens theNuGet Package Manager window. Click theBrowselink on the top left of the window.
  2. Enter Urhosharp in the search box, and hit Enter.
  3. Select theUrhoSharppackage, check the WhackABox project and the WhackABox.iOS project, and click Install on the right side of the window, as shown in the following screenshot:

That's it for the WhackABox and the WhackABox.iOS projects. Android is a little bit trickier to set up, since it needs a special UrhoSharp package and some code to be written to wire everything up.

Installing the UrhoSharp.ARCore NuGet package for Android

For Android, we will be adding the UrhoSharp.ARCore package, which contains extensions for ARCore. It has a dependency on UrhoSharp, so we don't have to add that package specifically. You can add theUrhoSharp.ARCore package as follows:

  1. Right-click on the WhackABox.Android project.
  2. Click Manage NuGet Packages..., as shown in the following screenshot:

  1. This opens the NuGet Package Manager window. Click the Browse link on the top left of the window.
  2. Enter UrhoSharp.ARCore in the search box, and hit Enter.
  1. Select the UrhoSharp.ARCorepackage and clickInstallon the right side of the window, as shown in the following screenshot:

That's it—all your dependencies on UrhoSharp have been installed in the project. We now have to wire up some life cycle events.

Adding the Android life cycle events

In Android, Urho needs to know about some specific events and be able to respond to them accordingly. We also need to add an internal message using MessagingCenter so that we can react to the OnResume event later on in the app. We will get to that when we add the code to initialize ARCore. But for now, add the five required overrides for Android events, as follows:

  1. In the Android project, open MainActivity.cs.
  2. Add the five overrides from the following code anywhere in the MainActivity class and the unresolved references by addingusingstatements forUrho.DroidandXamarin.Forms, as shown in the following code block:
protected override void OnResume()
{
    base.OnResume();
    UrhoSurface.OnResume();

    MessagingCenter.Send(this, "OnResume");
}

protected override void OnPause()
{
    UrhoSurface.OnPause();
    base.OnPause();
}

protected override void OnDestroy()
{
    UrhoSurface.OnDestroy();
    base.OnDestroy();
}

public override void OnBackPressed()
{
    UrhoSurface.OnDestroy();
    Finish();
}

public override void OnLowMemory()
{
    UrhoSurface.OnLowMemory();
    base.OnLowMemory();
}

The events map one-on-one to internal UrhoSharp events, except for OnBackPressed, which calls UrhoSharp.OnDestroy(). The reason for this is for memory management so that UrhoSharp knows when to clean up.

The MessagingCenter library is a built-in Xamarin.Forms pub/sub library for passing internal messages in an app. It has a dependency on Xamarin.Forms. We have created a library of our own called TinyPubSub that breaks this dependency and has a slightly easier API (as well as some additional features). You can check it out on GitHub at https://github.com/TinyStuff/TinyPubSub.

Defining the PlaneNode class

In Urho, you work with scenes that contain a tree of nodes. A node can be just about anything in the game, such as a renderer, a sound player, or simply a placeholder for subnodes.

As we talked about earlier when discussing AR fundamentals, planes are a common entity that is shared between the platforms. We need to create a common ground that represents a plane, which we can do by extending an Urho node. The position and the rotation will be tracked by the node itself, but we need to add a property to track the origin and the size of the plane, expressed by ARKit and ARCore as the extent of the plane.

We will add this class now and put it to use when we implement the AR-related code on each platform. The code to do this is straightforward, and can be set up by going through the following steps:

  1. In the WhackABox project root, create a new file called PlaneNode.cs.
  2. Add the following implementation of the class:
using Urho;

namespace WhackABox
{
    public class PlaneNode :Node
    {
        public string PlaneId { get; set; }
        public float ExtentX { get; set; }
        public float ExtentZ { get; set; }
    }
}

The PlaneId property will be an identifier that allows us to track which platform-specific plane this node represents. In iOS, this will be a string, while in Android, it will be the hashcode of the plane object that is converted to a string. The ExtentY and ExtentZ properties represent the size of the plane in meters. We are now ready to start creating the game logic and hooking up our application to the AR SDKs.

Adding custom renderers for the ARView control

Custom renderers are a very smart way of extending platform-specific behaviors to custom controls. They can also be used to override behaviors on controls that are already defined. In fact, all of the controls in Xamarin.Forms use renderers to translate the Xamarin.Forms control into a platform-specific control.

We are going to create two renderers, one for iOS and one for Android, that initialize the UrhoSurface on which we are going to render. The instantiation of the UrhoSurface differs on each platform, which is why we need two different implementations.

For iOS

A custom renderer is a class that inherits from another renderer. It allows us to add custom code for important events, such as when an element in XAML is created when the XAML file is parsed. Since the ARView control inherits from the View, we will be using ViewRenderer as a base class. Let's create the ARViewRenderer class by going through the following steps:

  1. In the WhackABox.iOS project, create a folder called Renderers.
  2. In that folder, add a new class called ARViewRenderer.
  3. Add the following code to the class:
using System.Threading.Tasks;
using Urho.iOS;
using WhackABox.Controls;
using WhackABox.iOS.Renderers;using Xamarin.Forms;
using Xamarin.Forms.Platform.iOS;

 [assembly: ExportRenderer(typeof(ARView), typeof(ARViewRenderer))]

 namespace WhackABox.iOS.Renderers
{
    public class ARViewRenderer : ViewRenderer<ARView, UrhoSurface>
    {
          protected async override void 
          OnElementChanged(ElementChangedEventArgs<ARView> e)
        {
            base.OnElementChanged(e);

            if (Control == null)
            {
                await Initialize();
            }
         }

         private async Task Initialize()
         {
             var surface = new UrhoSurface();
             SetNativeControl(surface);
             await surface.Show<Game>();
         }
     }
}

The ExportRenderer attribute registers this renderer to Xamarin.Forms so that it knows that when it parses (or compiles) an ARView element, it should render it using this specific renderer. It takes two arguments: the first is the control that we want to register a renderer to, and the second is the type of the renderer. This attribute must be placed outside the namespace declarations.

The ARViewRenderer class inherits ViewRenderer<ARView, UrhoSurface>. This specifies which control this renderer is created for and which native control it should render. In this case, the ARView element will be natively replaced by a UrhoSurface control that in itself is an iOS-specific UIView.

We override the OnElementChanged() method that is called every time the ARView element changes, either when it is created or when it is replaced. We can then check whether the Control property is set. The control is of the UrhoSurfacetypesince we declared that in the class definition. If it's null, then we make a call toInitialize()to create it.

The creation is straightforward. We simply create a new UrhoSurface control and set the native control to this newly created object. We then call the Show<Game>() method to start the game by specifying which class represents our Urho game. Note that the Game class is not defined yet, but it will be very soon, right after we create the custom renderer for Android.

For Android

The custom renderer for Android does the same thing as it does for iOS, but with the additional step of checking permissions. Let's create the ARViewRenderer class for Android by going through the following steps:

  1. In the WhackABox.Droid project, create a folder called Renderers.
  2. In that folder, add a new class calledARViewRenderer.
  3. Add the following code to the class:
 using System.Threading.Tasks;
 using Android;
 using Android.App;
 using Android.Content;
 using Android.Content.PM;
 using Android.Support.V4.App;
 using Android.Support.V4.Content;
 using WhackABox.Droid.Renderers;
 using WhackABox;
 using WhackABox.Controls;
 using WhackABox.Droid;
 using Urho.Droid;
 using Xamarin.Forms;
 using Xamarin.Forms.Platform.Android;

  [assembly: ExportRenderer(typeof(ARView), 
  typeof(ARViewRenderer))]
  namespace WhackABox.Droid.Renderers
 {
     public class ARViewRenderer : ViewRenderer<ARView,  
     Android.Views.View>
     {
         private UrhoSurfacePlaceholder surface;
         public ARViewRenderer(Context context) : base(context)
         {
             MessagingCenter.Subscribe<MainActivity>(this,  
             "OnResume", async (sender) =>
             {
                 await Initialize();
             });
         }

         protected async override void 
         OnElementChanged(ElementChangedEventArgs<ARView> e)
         {
             base.OnElementChanged(e);

             if (Control == null)
             {
                 await Initialize();
             }
         }

         private async Task Initialize()
         {
             if (ContextCompat.CheckSelfPermission(Context, 
                 Manifest.Permission.Camera) != Permission.Granted)
             {
                 ActivityCompat.RequestPermissions(Context as  
                 Activity, new[] { Manifest.Permission.Camera },  
                 42);
                 return;
             }

             if (surface != null)
                 return;

             surface = UrhoSurface.CreateSurface(Context as 
             Activity);
             SetNativeControl(surface);

             await surface.Show<Game>();
         }
     }
 }

This custom renderer also inherits from ViewRenderer<T1, T2>, where the first type is the type of the renderer itself and the second is the native control that the renderer will produce. In this case, the native control will be a control that inherits from Android.Views.View. The renderer creates a UrhoSurfacePlaceholder instance, which it assigns as the native control. UrhoSurfacePlaceholder is a class that wraps some functionality of the Simple DirectMedia Layer (SDL) library that Urho uses on Android to access media functionality. The last thing it does is to start the game based on the soon-to-exist Gameclass. We will define this in the next section of this chapter.

Creating the game

To write an application that uses Urho, we need to create a class that inherits from Urho.Application. This class defines some virtual methods that we can use to set up the scene. The method we will use is Start(). Before that, however, we need to create the class. We will create it as an abstract class in the WhackABox project that will be used as a base class in each of the platform projects.

Adding the shared abstract Game class

We start by creating the Game.cs file that will contain shared code. Let's set this up by going through the following steps:

  1. In the WhackABox project, create a new file called Game.cs in the root of the project.
  2. Add the following code to the class:
using System;
using System.Linq;
using Urho;
using Urho.Shapes;

namespace WhackABox
{
    public abstract class Game : Application
    {
        protected Scene scene;

        public Game(ApplicationOptions options) : base(options)
        {
        }
    }
}

The Game inherits from Urho.Application, which will do most of the work regarding the game itself. We define a property called scene, of the Scenetype. A Scene inUrhorepresents one screen of the game (we could have different scenes for different parts of a game or for a menu, for example). In this game, we will only be defining one scene, which will be initialized later. A scene maintains a hierarchy of nodes that compose it, and each node can have any number of children and any number of components. It's the components that do the work. Later on, for example, we will be rendering boxes, which will be represented by a node that will have aBoxcomponent attached.

The Game class itself is instantiated from the custom renderers that we defined in the earlier section, and it takes an ApplicationOptions instance as a parameter in the constructor. This needs to be passed to the base class. We now need to write some methods that will be AR-specific and used by the code we will write later on.

InitializeAR

We need a common entry point for iOS and Android to hook up to. Since the initialization of ARKit and ARCore is platform-specific, we will simply define an abstract method that each platform can override. You can do this by following these steps:

  1. In the WhackABox project, open the Game.cs class.
  2. Add the following InitializeAR() method to the class:
protected abstract void InitializeAR();

Since the class is itself abstract, we can also define methods as abstract. This means that this method must be implemented by any class that inherits from the Game class.

CreateSubPlane

The next method is the CreateSubPlane() method. When the application finds a plane on which we can place objects, it will create a node. We will write that code specifically for each platform soon. This node also defines a subplane that will position a box, representing the position and size of that plane. We have already defined the PlaneNode class earlier in this chapter.

Let's add the code by going through the following steps:

  1. In the WhackABox project, open the Game.cs class.
  2. Add the following CreateSubPlane() method to the class:
protected void CreateSubPlane(PlaneNode planeNode)
{
    var node = planeNode.CreateChild("subplane");
    node.Position = new Vector3(0, 0.05f, 0);

    var box = node.CreateComponent<Box>();
    box.Color = Color.FromHex("#22ff0000");
}

Any class inheriting from Urho.Node, such as PlaneNode, has the CreateChild() method. This allows us to create a child node and specify a name for that node. That name will be used later on to find specific children to perform operations on. We position the node at the same position as the parent node, except that we raise it 0.05 meters (5 cm) above the plane.

To see the plane, we add a box component with a semi-transparent red color. The box is a component that is created with a call to CreateComponent() on our node. The color is defined using the AARRGGBB pattern, where AA is the alpha component (the transparency) and RRGGBB is the standard red-green-blue format. We use the hexadecimal representation of the colors.

UpdateSubPlane

Both ARKit and ARCore update planes continuously. What we are interested in are changes in the position of a subplane and the extent of it. By extension, we are referring to the size of the plane. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs class.
  2. Add the UpdateSubPlane() method in the code anywhere in the Game.cs class, as shown in the following code snippet:
protected void UpdateSubPlane(PlaneNode planeNode, Vector3 position)
{
    var subPlaneNode = planeNode.GetChild("subplane");
    subPlaneNode.Scale = new Vector3(planeNode.ExtentX, 0.05f, 
    planeNode.ExtentZ);
    subPlaneNode.Position = position;
}

The method takes the PlaneNode instance that we want to update, along with a new position for it. We locate the subplane by querying the current node for any node called "subplane". Remember that we named the subplane in the AddSubPlane() method. We can now easily access the node by name. We update the scale of the subplane node by taking the ExtentX and ExtentZ properties from the PlaneNode. The PlaneNode instance will be updated by some platform-specific code before we call UpdateSubPlane(). Finally, we set the position of the subplane to the passed position parameter.

FindNodeByPlaneId

We need a method to quickly find nodes. Both ARKit and ARCore keep an internal track of their planes, and to map those internal representations of planes to our PlaneNode, we have to assign a custom ID to a plane when it's created. This will be done in the platform-specific code, but we can still write the function to query the scene for PlaneNode.

The planeId argument is a string, since ARKit defines the plane ID in a form that resembles a Globally Unique Identifier (GUID). A GUID is a structured sequence of hexadecimal numbers that can be represented in a string format. Let's add a method that will find the node that we are looking for by using this ID, as follows:

  1. In theWhackABoxproject, open theGame.csclass.
  2. Add theFindNodeByPlaneId() method in the code anywhere in theGame.csclass, as shown in the following code snippet:
protected PlaneNode FindNodeByPlaneId(string planeId)
{
    return scene.Children.OfType<PlaneNode>().FirstOrDefault(e => 
    e.PlaneId == planeId);
}

The method queries the scene by using Linq, and looks for the first child with the plane ID that it was given. If it can't find one, it returns null, since null is the default value of a reference type object.

These are all of the methods that we need in the shared code before dropping down into ARKit and ARCore.

Adding platform-specific classes

It's now time to add platform-specific code. We will create two classes, one for iOS and one for Android, which will inherit from the abstract Game class.

In this section, we will simply set up the skeleton code for these files.

Adding the iOS-specific class

Let's start by creating the class for Game on iOS, as follows:

  1. In the WhackABox.iOS project root, add a new file called Game.cs.
  2. Add the code to the class, as shown in the following snippet:
using Urho;

namespace WhackABox.iOS
{
    publicclassGame : WhackABox.Game
    {
publicGame(ApplicationOptions options) : base(options)
        {
}
}
}

Since the abstract base class requires a constructor to be called, we need to extend that constructor into our platform-specific class. Now, let's do the same thing for Android.

Adding the Android-specific class

The same goes for Android: only the namespace changes. Let's set this up by going through the following steps:

  1. In the WhackABox.Android project, add a new file calledGame.cs.
  2. Add the code as shown in the following snippet:
using Urho;

namespace WhackABox.Droid
{
publicclassGame : WhackABox.Game
{
publicGame(ApplicationOptions options) : base(options)
{
}
}
}

As you see, it's identical compared to the iOS code except for the namespace. Let's now start adding some platform-specific code.

Writing the ARKit-specific code

In this section, we will write the platform-specific code for iOS that will initialize ARKit, find planes, and create nodes for UrhoSharp to render on the screen. We will be taking advantage of an Urho component that wraps ARKit in iOS. We will also be writing all the functions that will position, add, and remove nodes. ARKit uses anchors, which act as virtual points that glue the overlaid graphics to the real world. We are specifically looking for ARPlaneAnchor, which represents a plane in the AR world. There are other types of anchors available, but for this app, we only need to find horizontal planes.

Let's start off by defining the ARKitComponent field so that we can use it later.

Defining the ARKitComponent

We start by adding a private field to an ARKitComponent that will be initialized later on. Let's set this up by going through the following steps:

  1. In the WhackABox.iOS project, open Game.cs.
  2. Add a private field that holds an ARKitComponent, as shown in bold in the following code block:
using System.Linq;
using ARKit;
using Urho;
using Urho.iOS;

namespace WhackABox.iOS
{
publicclassGame : WhackABox.Game
{
protected ARKitComponent arkitComponent;

publicGame(ApplicationOptions options) : base(options)
{
}
}
}

Make sure that you add all the using statements to ensure that all the code we later use resolves the correct types.

Writing handlers for adding and updating anchors

We will now add the necessary code that will add and update anchors. We will also add some methods to help set the orientation of the nodes after ARKit updates the anchors.

SetPositionAndRotation

The SetPositionAndRotation() method will be used by both the Add and Update anchors, so we need to define it before creating the handlers for the events that will be raised by ARKit. Let's set this up by going through the following steps:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add the SetPositionAndRotation() method to the class, as shown in the following code block:
private void SetPositionAndRotation(ARPlaneAnchor anchor, PlaneNode 
                                    node)
{
     arkitComponent.ApplyOpenTkTransform(node, anchor.Transform, 
                                         true);

     node.ExtentX = anchor.Extent.X;
     node.ExtentZ = anchor.Extent.Z;

     var position = new Vector3(anchor.Center.X, anchor.Center.Y, -
                                anchor.Center.Z);
     UpdateSubPlane(node, position);
}

The method takes two parameters. The first is an ARPlaneAnchor object defined by ARKit, and the second is the PlaneNode instance that we have in the scene. The purpose of the method is to make sure that the PlaneNode is in sync with the ARPlaneAnchor object passed by ARKit. The arkitComponent object has a helper method called ApplyOpenTkTransform() to translate the position and rotation of the ARPlaneAnchor object into the position and rotation objects used by Urho. We then update the Extent (size) of the plane to the PlaneNode and get the anchor center position from the ARPlaneAnchor. Finally, we call a method that we defined earlier to update the subplane node that holds the Box component that will do the actual rendering of the plane as a semi-transparent red box.

We need one more method to handle the update and add functionality.

UpdateOrAddPlaneNode

The UpdateOrAddPlaneNode() method does exactly what the name implies: it takes an ARPlaneAnchor as an argument and either updates or adds a new PlaneNode instance to the scene. Let's set this up by going through the following steps:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add the UpdateOrAddPlaneNode() method, as shown in the following code block:
private void UpdateOrAddPlaneNode(ARPlaneAnchor anchor)
{
    var node = FindNodeByPlaneId(anchor.Identifier.ToString());

    if (node == null)
    {
        node = new PlaneNode()
        {
            PlaneId = anchor.Identifier.ToString(),
            Name = $"plane{anchor.GetHashCode()}"
        };

        CreateSubPlane(node);
        scene.AddChild(node);
    }

    SetPositionAndRotation(anchor, node);
}

A node is either already present in the scene or it needs to be added. The first line of code calls the FindNodeByPlaneId() method to query the scene for an object with the given PlaneId. For iOS, we use the anchor.Identifier property to track planes defined by iOS. If this call returns null, it means that the plane is not present in the scene and we need to create it. To do this, we instantiate a new PlaneNode instance, giving it a PlaneId and a user-friendly name for debugging purposes. We then create the subplane to visualize the plane itself by calling CreateSubPlane() (which we defined earlier) and add the node to the scene. Lastly, we update the position and rotation. We do this for every call to the UpdateOrAddPlaneNode() method since it's the same for both new and existing nodes. It's now time to write the handlers that we will eventually hook up to ARKit directly.

OnAddAnchor

Let's add some code. The OnAddAnchor() method will be called each time ARKit updates its collection of anchors that describe points that we will use to relate to within our virtual world. We are specifically looking for anchors of the ARPlaneAnchortype.

Add the OnAddAnchor()( method to the Game.cs class by going through the following two steps:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add the OnAddAnchor() method anywhere in the class, as shown in the following code snippet:
private void OnAddAnchor(ARAnchor[] anchors)
{
    foreach (var anchor in anchors.OfType<ARPlaneAnchor>())
    {
        UpdateOrAddPlaneNode(anchor);
    }
}

The method takes an array of ARAnchors as a parameter. We filter out the anchors that are of the ARPlaneAnchortypeand iterate through the list. For eachARPlaneAnchor object, we call the UpdateOrAddPlaneNode() method that we created earlier to add a node to the scene. Let's now do the same for when ARKit wants to update anchors.

OnUpdateAnchors

Each time ARKit receives new information about an anchor, it will call this method. We do the same as we did with the previous code and iterate through the list to update the extent and position of the anchor object in the scene, as follows:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add theOnUpdateAnchors()method anywhere in the class, as shown in the following code snippet:
private void OnUpdateAnchors(ARAnchor[] anchors)
{
    foreach (var anchor in anchors.OfType<ARPlaneAnchor>())
    {
        UpdateOrAddPlaneNode(anchor);
    }
}

The code is a copy of the OnAddAnchors() method. It updates all nodes in the scene based on the information provided by ARKit.

We also need to write some code to remove the anchors that ARKit has removed.

Writing a handler for removing anchors

When ARKit decides that an anchor is invalid, it will remove it from the scene. This does not happen very often, but it's a good practice to handle this call anyway.

OnRemoveAnchors

Let's add a method to handle the removal of an ARPlaneAnchor object by going through the following steps:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add the OnRemoveAnchors() method anywhere in the class, as shown in the following code snippet:
private void OnRemoveAnchors(ARAnchor[] anchors)
{
 foreach (var anchor in anchors.OfType<ARPlaneAnchor>())
 {
 FindNodeByPlaneId(anchor.Identifier.ToString())?.Remove();
 }
}

As with the Add and Remove functions, this method accepts an array of ARAnchor. We iterate through this array, looking for anchors of the ARPlaneAnchor type. We then look for a node that represents this plane by calling the FindNodeByPlaneId() method. If it's not null, then we call for that node to be removed. Note the null-check operator before the Remove() call.

Initializing ARKit

We've now come to the last part of the iOS-specific code, which is where we initialize ARKit. This method is called InitializeAR() and takes no parameters. It's defined as an abstract method in the Game base class and must be implemented by the Game class defined in the iOS project.

The code to initialize ARKit is straightforward, and the ARKitComponent class does a lot of work for us. Let's set it up by going through the following steps:

  1. In the WhackABox.iOS project, open the Game.cs file.
  2. Add the InitializeAR() method anywhere in the class, as shown in the following code block:
protected override void InitializeAR()
{
    arkitComponent = scene.CreateComponent<ARKitComponent>();
    arkitComponent.Orientation = 
    UIKit.UIInterfaceOrientation.Portrait;
    arkitComponent.ARConfiguration = new 
    ARWorldTrackingConfiguration
    {
        PlaneDetection = ARPlaneDetection.Horizontal
    };
    arkitComponent.DidAddAnchors += OnAddAnchor;
    arkitComponent.DidUpdateAnchors += OnUpdateAnchors;
    arkitComponent.DidRemoveAnchors += OnRemoveAnchors;
    arkitComponent.RunEngineFramesInARKitCallbakcs = 
    Options.DelayedStart;
    arkitComponent.Run();
}

The code starts by creating an ARKitComponent class. We then set the allowed orientation and create an ARWorldTrackingConfiguration class that states that we are only interested in horizontal planes. To respond to the addition, updating, and removal of planes, we attach the event handlers we created earlier.

We instruct the ARKitComponent to delay calling the callbacks to allow ARKit to initialize properly. Note the spelling error in the RunEngineFramesInARKitCallbakcs property. This is a good example of why you need to carry out a review of your code since it will be hard to change this name without breaking backward compatibility. Naming is hard.

The last thing is to tell ARKit to start running. We do this by calling the arkitComponent.Run() method.

Writing the ARCore-specific code

It's now time to do the same for Android with ARCore. Just as with iOS, we are going to put all Android-specific code in a file of its own. This is the Game.cs file in the Android project that we created earlier.

Defining the ARCoreComponent

First, we are going to add a field that stores a reference to the ARCoreComponent instance. This wraps up most of the interaction with ARCore. The ARCoreComponent is defined in the UrhoSharp.ARCore NuGet package that we installed at the beginning of the chapter.

Let's add some using statements and the arCore private field by going through the following steps:

  1. In the WhackABox.Droid project, open the Game.csfile.
  2. Add the arCoreprivate field, and also make sure that you add the using statements marked in bold in the following code block:
using Com.Google.AR.Core;
using Urho;
using Urho.Droid;

namespace WhackABox.Droid
{
publicclassGame : WhackABox.Game
{
publicGame(ApplicationOptions options) : base(options)
{
}

private ARCoreComponent arCore;
}
}

The using statements will allow us to resolve the types that we need in this file, and the arCore property will be a shorthand when we want to access ARCore functions.

We'll continue by adding some methods to this class.

SetPositionAndRotation

We need to add or update a PlaneNode whenever a plane is detected or updated. The SetPositionAndRotation() method updates the passed PlaneNode and sets properties on that node based on the content of the AR.Core.Plane object. Let's set this up by going through the following steps:

  1. In the WhackABox.Droid project, open the Game.cs file.
  2. Add the SetPositionAndRotation() method to the class, as shown in the following code block:
privatevoid SetPositionAndRotation(Com.Google.AR.Core.Plane plane,  
                                    PlaneNode node)
{
    node.ExtentX = plane.ExtentX;
    node.ExtentZ = plane.ExtentZ;
    node.Rotation = new Quaternion(plane.CenterPose.Qx(),
                                   plane.CenterPose.Qy(),
                                   plane.CenterPose.Qz(),
                                   -plane.CenterPose.Qw());

    node.Position = new Vector3(plane.CenterPose.Tx(),
                                plane.CenterPose.Ty(),
                                -plane.CenterPose.Tz());
}

The previous code updates the extent of the plane for the node and creates a rotation, Quaternion. Don't worry if you don't know what a Quaternion rotation is—few people do—but they seem to magically hold the rotation information of the model in a very flexible way. The plane.CenterPose property is a matrix that holds the position and orientation of the plane. Finally, we update the position of the node from the CenterPose property.

The next step is to create a method that handles frame updates from ARCore.

Writing a handler for ARFrame updates

Android handles updates from ARCore a little bit differently than ARKit, which exposes three different events for added, updated, and removed nodes. When using ARCore, these events get called whenever any changes occur, and the handler that will take care of this is the one we are about to add.

Let's add the method by going through the following steps:

  1. In the WhackABox.Droid project, open the Game.cs file.
  2. Add the OnARFrameUpdated() method anywhere in the class, as shown in the following code block:
privatevoidOnARFrameUpdated(FramearFrame)
{
    varall=arCore.Session.GetAllTrackables(
                  Java.Lang.Class.FromType(
                  typeof(Com.Google.AR.Core.Plane)));

    foreach(Com.Google.AR.Core.Planeplaneinall)
    {
        varnode=
        FindNodeByPlaneId(plane.GetHashCode().ToString());

        if(node==null)
        {
            node=newPlaneNode
            {
                PlaneId=plane.GetHashCode().ToString(),
                Name=$"plane{plane.GetHashCode()}"
};

            CreateSubPlane(node);
            scene.AddChild(node);
        }

        SetPositionAndRotation(plane, node);
        UpdateSubPlane(node,Vector3.Zero);
    }
}

We start by querying the arCore component for all the planes that it keeps track of. We then iterate through this list and see whether we have any nodes in the scene by calling the FindNodeByPlaneId() method, using the hash code of the plane as the identifier. If we can't find any, we create a new PlaneNode and assign the hash code as the PlaneId. We then create a subplane that contains the Box component to visualize the plane, and, finally, we add it to the scene. We then update the position and the rotation of the plane and make a call to update the subplane as well. Now that we have the handler written, we need to hook it up.

Initializing ARCore

To initialize ARCore, we will add two methods. The first one is a method that will take care of the configuration of ARCore, called OnConfigRequested(). The second one is the InitializeAR() method that will be called from the shared Game class later on.

OnConfigRequested

ARCore needs to know a few things, just as with iOS. In Android, this is done by defining a method that the ARCore component will call upon initialization. To create the method, go through the following steps:

  1. In the WhackABox.Droid project, open the Game.cs file.
  2. Add the OnConfigRequested() method anywhere in the class, as shown in the following code snippet:
private void OnConfigRequested(Config config)
{
    config.SetPlaneFindingMode(Config.PlaneFindingMode.Horizontal);
    config.SetLightEstimationMode(Config.LightEstimationMode.AmbientIntensity);
    config.SetUpdateMode(Config.UpdateMode.LatestCameraImage);
}

The method takes a Config object, which will store any configuration you make in this method. First, we set which type of plane we want to find. We are interested in Horizontal planes for this game. We define the kind of light-estimation mode we want to use, and, finally, we select which update mode we want. In this case, we want to use the latest camera image available. You can do a lot of fine-tuning during configuration, but this is out of the scope of this book. Be sure to check out the documentation for ARCore to learn more about its awesome power.

We now have all the code we need to initialize ARCore.

InitializeAR

As mentioned previously, the InitializeAR() method is defined in the abstract base class. We will now implement it for Android by overriding it, as follows:

  1. In the WhackABox.Droid project, open the Game.cs file.
  2. Add the InitializeAR() method anywhere in the class, as shown in the following code snippet:
protected override void InitializeAR()
{
    arCore = scene.CreateComponent<ARCoreComponent>();
    arCore.ARFrameUpdated += OnARFrameUpdated;
    arCore.ConfigRequested += OnConfigRequested;
    arCore.Run();
}

The first step is to create the ARCoreComponent provided by UrhoSharp. This component wraps the initialization of the native ARCore classes. We then add two event handlers: one for taking care of frame updates, and one that will be called during initialization. The last thing we do is call the Run() method on the ARCoreComponent to start tracking the world.

Now that we have both ARKit and ARCore configured and ready to go, it's time to write the actual game.

Writing the game

In this section, we will initialize Urho by setting up the camera, lighting, and a renderer. All the remaining code will now be written in the abstract Game class in the WhackABox project (the .NET Standard library project). The camera is the object that determines where objects will be rendered. The AR components take care of updating the position of the camera to virtually track your phone so that any object we render will be in the same coordinate space as what you are looking at. First, we need a camera that will be the viewing point of the scene.

Adding a camera

Adding a camera is a straightforward process, as shown in the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the camera property to the class, as shown in the following code snippet. A good practice is to place it right after the declaration of the class.
  1. Add the InitializeCamera()method anywhere in the class, as shown in the following code snippet:
private Camera camera;

private void InitializeCamera()
{
    var cameraNode = scene.CreateChild("Camera");
    camera = cameraNode.CreateComponent<Camera>();
}

In UrhoSharp, everything is a node, just as everything is a GameObject in Unity, including the camera object. We create a new node, which we call camera, and then we create a Camera component on that node and keep the reference to it for later use.

Configuring a renderer

UrhoSharp needs to render the scene to a viewport. A game can have multiple viewports, based on multiple cameras. Think of a game where you drive a car. The main viewport will be the game from the perspective of the driver. Another viewport might be the rear-view mirrors, which would actually be cameras themselves that render what they see onto the main viewport. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the viewportproperty to the class, as shown in the following code snippet. A good practice is to place it right after the declaration of the class itself, but placing it anywhere within the class will work.
  3. Add theInitializeRenderer()method anywhere in the class, as shown in the following code snippet:
private Viewport viewport;

private void InitializeRenderer()
{
    viewport = new Viewport(Context, scene, camera, null);
    Renderer.SetViewport(0, viewport);
}

The viewport property will hold a reference to the viewport for later use. The viewport is created by instantiating a new Viewport class. The constructor of that class needs a Context provided by the base class, the scene that we will create while initializing the game, a camera to know which point in space to render from, and a render path, which we default to null. A render path allows for postprocessing of the frame created while rendering. This is also outside the scope of this book, but it is worth checking out as well.

Now, let there be light.

Adding lights

To make objects visible, we need to define some lighting. We do this by creating a method that defines the type of lighting we want in the game. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the InitializeLights() method anywhere in the class, as shown in the following code block:
private void InitializeLights()
{
    var lightNode = camera.Node.CreateChild();
    lightNode.SetDirection(new Vector3(1f, -1.0f, 1f));
    var light = lightNode.CreateComponent<Light>();
    light.Range = 10;
    light.LightType = LightType.Directional;
    light.CastShadows = true;
    Renderer.ShadowMapSize *= 4;
}

Again, everything in UrhoSharp is a node, and lights are no exception to that rule. We create a generic node on the camera node by accessing the stored camera component and accessing the node it belongs to. We then set the direction of that node and create a Light component to define a light. The range of the light will be 10 units in length. The type is directional, meaning that it will shine from the position of the node in its defined direction. It will also cast shadows. We set the ShadowMapSize to four times the default value to give the shadow map some more resolution.

At this point, we have all we need to initialize UrhoSharp and the AR components.

Implementing the game startup

The base class of the Game class provides some virtual methods that we can override. One of these is Start(), which will be called shortly after the custom renderer has set up the UrhoSurface.

Add the method by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the Start() method anywhere in the class, as shown in the following code snippet:
protected override void Start()
{
   scene = new Scene(Context);
   var octree = scene.CreateComponent<Octree>();

    InitializeCamera();
    InitializeLights();
    InitializeRenderer();

    InitializeAR();
}

The scene that we have been talking about is created here in the first line of the method. This is the scene that we look at whenUrhoSharpis running. It keeps track of all nodes that we add to it. All 3D games in UrhoSharp need an Octree, which is a component that implements spatial partitioning. It is used by the 3D engine to quickly find objects in a 3D space without having to query every single one in each frame. The second line of the method creates this component directly on the scene.

Following this, we have the four methods that initialize the camera, the lights, and the renderer, and which make a call to one of the two InitializeAR() methods, based on which platform we are compiling for. If you start the app at this point, you should see that it finds planes and renders them, but that nothing more happens. It's time to add something to interact with.

Adding boxes

We are now going to focus on adding virtual boxes to our augmented world. We are going to write two methods. The first one is the AddBox() method, which will add a new box at a random position on a plane. The second is an override of the OnUpdate() method that UrhoSharp calls with each frame to perform game logic.

AddBox()

To add boxes to a plane, we need to add a method to do so. This method is called AddBox(). Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the random property to the class (preferably at the top, but anywhere in the class will work).
  3. Add the AddBox() method anywhere in the class, as shown in the following code block:
private static Random random = new Random();

private void AddBox(PlaneNode planeNode)
{
    var subPlaneNode = planeNode.GetChild("subplane");

    var boxNode = planeNode.CreateChild("Box");
    boxNode.SetScale(0.1f);

    var x = planeNode.ExtentX * (float)(random.NextDouble() -
      0.5f);
    var z = planeNode.ExtentZ * (float)(random.NextDouble() - 
      0.5f);

    boxNode.Position = new Vector3(x, 0.1f, z) +  
    subPlaneNode.Position;

    var box = boxNode.CreateComponent<Box>();
    box.Color = Color.Blue;
}

The static random object that we create will be used for randomizing the location of a box on a plane. We want to use a static Random instance since we don't want to risk creating multiple instances that may be seeded with the same value and therefore return the exact same sequence of random numbers. The method starts by finding the subplane child of the PlaneNode instance that we pass in by calling planeNode.GetChild("subplane"). We then create a node that will render the box. To make the box fit the world, we need to set the scale to 0.1, which will make it 10 cm in size.

We then randomize the position of the box using the ExtentX and ExtentZ properties, multiplied by a new random value between 0 and 1 that we first subtract 0.5 from. This is to center the position since the position of the parent node is the center of the plane. Then, we set the position of the box node at the randomized position and 0.1 units above the plane. We also need to add the subplane's position, since it might be a little bit offset from the parent node. Finally, we add the actual box to be rendered and set the color to blue.

Let's now add code to call the AddBox() method, based on some game logic.

OnUpdate()

Many games use a game loop. This calls an Update() method, which takes an input and calculates the state of the game. UrhoSharp is no exception. The base class of our game has a virtual OnUpdate() method that we can override so that we can write code that will be executed with each frame. This method is called frequently, usually about 50 times per second.

We will now override the Update() method to add game logic that adds a new box every other second. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the newBoxTtl field and the newBoxIntervalInSeconds field to the class at the beginning of the code as shown.
  3. Add the OnUpdate() method anywhere in the class, as shown in the following code block:
private float newBoxTtl;
private readonly float newBoxIntervalInSeconds = 2;

protected override void OnUpdate(float timeStep)
{
    base.OnUpdate(timeStep);

    newBoxTtl -= timeStep;

    if (newBoxTtl < 0)
    {
        foreach (var node in scene.Children.OfType<PlaneNode>())
        {
            AddBox(node);
        }

        newBoxTtl += newBoxIntervalInSeconds;
    }
}

The first field, newBoxTtl—where Ttl is time to live (TTL)—is an internal counter that will be reduced by the number of milliseconds that have passed since the last frame. When it falls below 0, we will add a new box to each plane of the scene. We find all instances of PlaneNode by querying the Children collection of the scene and returning only the children of the PlaneNode type. The second field, newBoxIntervalInSeconds, indicates how many seconds we will add to the newBoxTtl once it reaches 0. To know how much time has passed since the last frame, we use the timeStep parameterthat is passed into theOnUpdate()method by UrhoSharp. The value of this parameter is the number of seconds since the last frame. It's usually a small value, which will be something such as 0.016 if the update loop runs at 50 frames per second. It could vary, though, which is why you will want to use this value to carry out the subtraction fromnewBoxTtl.

If you run the game now, you will see that boxes appear on the detected planes. We still cannot interact with them, however, and they look pretty boring. Let's continue by making them rotate.

Making boxes rotate

You can add your own components to UrhoSharp by creating a class that inherits from Urho.Component. We will be creating a component that will make the boxes spin around all three axes.

Creating the rotate component

As we mentioned, a component is a class that inherits from Urho.Component. This base class defines a virtual method called OnUpdate() that behaves in the same way as the Update() method on the Game class itself. This allows us to add logic to the component so that it can modify the state of the node it belongs to.

Let's create the rotate component by going through the following steps:

  1. In the WhackABox project root, create a new class called Rotator.cs.
  2. Add the following code:
using Urho;

namespace WhackABox
{
    public class Rotator : Component
    {
        public Vector3 RotationSpeed { get; set; }

        public Rotator()
        {
            ReceiveSceneUpdates = true;
        }

        protected override void OnUpdate(float timeStep)
        {
            Node.Rotate(new Quaternion(
                RotationSpeed.X * timeStep,
                RotationSpeed.Y * timeStep,
                RotationSpeed.Z * timeStep),
                TransformSpace.Local);
        }
    }
}

The RotationSpeed property will be used to determine the speed of rotation around any specific axis. It will be set when we assign the component to the box node in the next step. To enable the component to receive calls to the OnUpdate() method on each frame, we need to set the ReceiveSceneUpdates property to true. If we don't do this, the component will not be called by UrhoSharp at each update. It's set to false by default, for performance reasons.

All the fun happens in the override of the OnUpdate() method. We create a new quaternion to represent a new rotation state. Again, we don't need to know how this works in detail, only that quaternions belong to the mystical world of advanced mathematics. We multiply each axis in the RotationSpeed vector by the timeStep parameter to generate a new value. The timeStep parameter is the number of seconds that have passed since the last frame. We also define the rotation as being around the local coordinate space of this box.

Now that the component is created, we need to add it to the boxes.

Assigning the Rotator component

Adding the Rotator component is as simple as adding any other component. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Update the AddBox() method by adding the code marked in bold in the following code block:
private void AddBox(PlaneNode planeNode)
{
    var subPlaneNode = planeNode.GetChild("subplane");

    var boxNode = planeNode.CreateChild("Box");
    boxNode.SetScale(0.1f);

    var x = planeNode.ExtentX * (float)(random.NextDouble() -
      0.5f);
    var z = planeNode.ExtentZ * (float)(random.NextDouble() - 
      0.5f);

    boxNode.Position = new Vector3(x, 0.1f, z) + 
    subPlaneNode.Position;

    var box = boxNode.CreateComponent<Box>();
    box.Color = Color.Blue;

    var rotationSpeed = new Vector3(10.0f, 20.0f, 30.0f);
    var rotator = new Rotator() { RotationSpeed = rotationSpeed };
    boxNode.AddComponent(rotator);
}

We begin by defining how we want the box to rotate by creating a new Vector3 struct and assigning it to a new variable called rotationSpeed. In this case, we want it to rotate 10 units around the x axis, 20 units around the y axis, and 30 units around the z axis. We use the rotationSpeed variable to set the RotationSpeed property of the Rotator component that we instantiate in the second row of the code we added.

Finally, we add the component to the box node. The boxes should now rotate in an interesting way.

Adding a box hit-test

We now have rotating boxes that keep piling up. We need to add a way to remove boxes. The simplest thing would be to add a feature that removes boxes when we touch them, but we are going to make it a little fancier than that: whenever we touch a box, we want it to shrink and disappear before we remove it from the scene. To do this, we are going to use our newly acquired knowledge of components, and then add some code to determine whether we are touching a box.

Adding a death animation

The Death component that we are about to add has the same template as the Rotator component that we created in the last section. Let's add it by going through the following steps and taking a look at the code:

  1. In the WhackABox project root, create a new class called Death.cs.
  2. Replace the code in the class with the following code:
 using Urho;
 using System;

 namespace WhackABox
 {
     public class Death : Component
     {
         private float deathTtl = 1f;
         private float initialScale = 1;

         public Action OnDeath { get; set; }

         public Death()
         {
             ReceiveSceneUpdates = true;
         }

         public override void OnAttachedToNode(Node node)
         {
             initialScale = node.Scale.X;
         }

         protected override void OnUpdate(float timeStep)
         {
             Node.SetScale(deathTtl * initialScale);

             if (deathTtl < 0)
             {
                 Node.Remove();
             }

             deathTtl -= timeStep;
         }
     }
 }

We first define two fields. The deathTtl field determines how long the animation will be, in seconds. The initialScale field keeps track of the scale of the node when the component is attached to the node. To receive updates, we need to set ReceiveSceneUpdates to true in the constructor. The overridden OnAttachedToNode() method is called when the component is attached to a node. We use this method to set the initialScale field. After the component is attached, we start getting calls on each frame to OnUpdate(). On each call, we set a new scale of the node based on the deathTtl field multiplied by the initialScalefield. When the deathTtl field reaches 0, we remove the node from the scene. If we don't reach zero, then we subtract the amount of time since the last frame was called, which is given to us by the timeStep parameter. All we need to do now is figure out when to add the Death component to a box.

DetermineHit()

We need a method that can interpret a touch on the 2D surface of the screen and figure out which boxes we are hitting using an imaginary ray traveling from the camera toward the scene we are looking at. This method is called DetermineHit(). Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the DetermineHit() method anywhere in the class, as shown in the following code block:
private void DetermineHit(float x, float y)
{
    var cameraRay = camera.GetScreenRay(x, y);
    var result = scene.GetComponent<Octree>
    ().RaycastSingle(cameraRay);

    if (result?.Node?.Name?.StartsWith("Box") == true)
    {
        var node = result?.Node;

        if (node.Components.OfType<Death>().Any())
        {
            return;
        }

        node.CreateComponent<Death>();
    }
}

The x and y parameters that are passed into the method range from 0 to 1, where 0 represents the left edge or top edge of the screen, and 1 represents the right edge or bottom edge of the screen. The exact center of the screen would be x=0.5 and y=0.5. Since we want to get a ray from the camera, we can use a method called GetScreenRay() directly on the camera component. This returns a ray from the camera in the scene in the same direction that the camera is set to. We use this ray and pass it to the Octree component's RaycastSingle() method, which returns a result that will contain a single node if one is hit.

We examine the results, perform multiple null checks, and—finally—check whether the name of the node starts with Box. If this is true, we check to see whether the box we hit is alreadydoomed by examining whether there is a Death component attached. If there is, we return. If there isn't, we create aDeathcomponent and leave the box to die.

This all looks good so far. We now need something to call the DetermineHit() method.

OnTouchBegin()

Touches are handled as events in UrhoSharp, and this means that they require event handlers. Let's create a handler for the TouchBegin event by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. Add the OnTouchBegin() method anywhere in the code, as shown in the following code snippet:
private void OnTouchBegin(TouchBeginEventArgs e)
{
    var x = (float)e.X / Graphics.Width;
    var y = (float)e.Y / Graphics.Height;

    DetermineHit(x, y);
}

When a touch is registered, this method will be called, and information about that touch event will be sent as a parameter. This parameter has an X and a Y property that represent the point on the screen that we have touched. Since the DetermineHit() method wants the values in the range of 0 to 1, we need to divide the X and Y coordinates by the width and height of the screen.

Once that is done, we call the DetermineHit() method. To complete this section, we just have to wire up the event.

Wiring up input

All that's left now is to wire up the event to the Input subsystem of UrhoSharp. This is done by adding a single line of code to the Start() method, as shown in the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. In the Start() method, add the code highlighted in bold in the following code block:
protected override void Start()
{
 scene = new Scene(Context);
 var octree = scene.CreateComponent<Octree>();

 InitializeCamera();
 InitializeLights();
 InitializeRenderer();

Input.TouchBegin += OnTouchBegin;

 InitializeAR();
}

This wires up the TouchBegin event to our OnTouchBegin event handler.

If you run the game now, the boxes should animate and disappear when you tap on them. What we need now is some kind of statistic that shows how many planes there are and how many boxes are still alive.

Updating statistics

At the beginning of the chapter, we added some controls to the XAML that displayed the number of planes and boxes that were present in the game. It's now time to add some code to update those numbers. We will be using internal messaging to decouple the game from the Xamarin.Forms page that we use to display this information.

The game will send a message to the main page that will contain a class with all the information we need. The main page will receive this message and update the labels.

Defining a statistics class

We are going to use MessagingCenter in Xamarin.Forms, which allows us to send an object along with the message. We need to create a class that can carry the information we want to pass. Let's set this up by going through the following steps:

  1. In the WhackABox project root, create a new class called GameStats.cs.
  2. Add the following code to the class:
public class GameStats
{
    public int NumberOfPlanes { get; set; }
    public int NumberOfBoxes { get; set; }
}

The class will be a simple data carrier that indicates how many planes and boxes we have.

Sending updates via MessagingCenter

When a node is created or removed, we need to send statistics to anything that is listening. To do this, we need a new method that will go through the scene and count how many planes and boxes we have and then send a message. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.csfile.
  2. Add a method called SendStats() anywhere in the class, as shown in the following code block:
private void SendStats()
{
    var planes = scene.Children.OfType<PlaneNode>();
    var boxCount = 0;

    foreach (var plane in planes)
    {
        boxCount += plane.Children.Count(e => e.Name == "Box");
    }

    var stats = new GameStats()
    {
        NumberOfBoxes = boxCount,
        NumberOfPlanes = planes.Count()
    };

    Xamarin.Forms.Device.BeginInvokeOnMainThread(() =>
    {
        Xamarin.Forms.MessagingCenter.Send(this, "stats_updated",  
        stats);
    });
}

The method checks all children of the scene object to find nodes of the PlaneNode type. We iterate through all of these nodes and count how many of the node's children have the name Box, and then indicate this number in a variable called boxCount. When we have this information, we create a GameStats object and initialize it with the box count and the plane count.

The last step is to send the message. We have to make sure that we are using the UI thread (the MainThread) since we are going to update the graphical user interface (GUI). Only the UI thread is allowed to touch the GUI. This is done by wrapping the MessagingCenter.Send() call in BeginInvokeOnMainThread() .

The message that is sent is stats_updated. It contains the stats information as an argument. Let's now make use of the SendStats() method.

Wiring up events

The scene has a lot of events that we can wire up. We will hook up to NodeAdded and NodeRemoved to determine when we need to send statistics information. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the Game.cs file.
  2. In the Start() method, add the code that is highlighted in bold in the following code block:
protected override void Start()
{
    scene = new Scene(Context);
 scene.NodeAdded += (e) => SendStats();
    scene.NodeRemoved += (e) => SendStats();
    var octree = scene.CreateComponent<Octree>();

    InitializeCamera();
    InitializeLights();
    InitializeRenderer();

    Input.TouchEnd += OnTouchEnd;

    InitializeAR();
}

Each time a node is either added orremoved,a new message will be sent to the GUI.

Updating the GUI

This will be the last method we add to the game. It handles the informationupdates and also updates the labels in the GUI. Let's add it by going through the following steps:

  1. In the WhackABox project, open the MainPage.xaml.cs file.
  2. Add a method called StatsUpdated() anywhere in the code, as shown in the following code snippet:
private void StatsUpdated(Game sender, GameStats stats)
{
    boxCountLabel.Text = stats.NumberOfBoxes.ToString();
    planeCountLabel.Text = stats.NumberOfPlanes.ToString();
}

The method receives the GameStats object that we sent and updates the two labels in the GUI.

Subscribing to the updates in the MainPage class

The last line of code to add will wire up the StatsUpdated handler to an incoming message. Let's set this up by going through the following steps:

  1. In the WhackABox project, open the MainPage.xaml.cs file.
  2. In the constructor, add the line of code that is highlighted in bold in the following code snippet:
public MainPage()
{
    InitializeComponent();
MessagingCenter.Subscribe<Game, GameStats>(this,  
    "stats_updated", StatsUpdated);
}

This line of code hooks up an incoming message with the content stats_updated to the StatsUpdated method. Now, run the game and go out into the world to hunt down those boxes!

The completed app looks something like the following screenshot, with spinning boxes popping up at randomlocations:

Summary

In this chapter, we learned how to integrate AR into Xamarin.Forms by using custom renderers. We took advantage of UrhoSharp to use cross-platform rendering, components, and input management to interact with the world. We also learned a bit about MessagingCenter, which can be used to send internal in-process messages between different parts of an application to reduce coupling.

Next up, we are going to dive into machine learning and create an app that can recognize a hotdog in an image.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.147.65.65