Chapter 8: Planets: Tracking Images

In this chapter, we will be using augmented reality for data visualization and education. We're going to build a project where users can learn about the planets in our Solar System. Suppose you have a children's science book on the Solar System with a companion mobile app. On the page about planet Earth, for example, the reader can point their mobile device at the picture on the page and a 3D rendering of the Earth will pop out of the page.

The AR mechanism we'll be using is known as image tracking. With image tracking, you prepare a reference library of images that may be recognized and tracked in the real world at runtime. When the user's device's camera detects one of these images, a virtual object can be instantiated at the image location.

I have provided you with "planet cards," which have pictures and unique markers on them for each planet that I created from free resources available on the web, for you to print yourself and use with the app. For rendering the planets' spherical surface skins, we will be using free texture images of the actual planets.

We will cover the following topics in this chapter:

  • Understanding AR image tracking
  • Specifying the Planets project and getting started
  • Defining and tracking reference images
  • Creating and instantiating a virtual Earth prefab
  • Rotating a planet on its axis
  • Expanding the project with multiple planets
  • Making a responsive UI

By the end of this chapter, you'll have a working app that detects images on the provided planet cards, renders a 3D model of the given planet, and offers additional information details about a planet.

Technical requirements

To implement the project in this chapter, you need Unity installed on your development computer, connected to a mobile device that supports augmented reality applications (see Chapter 1, Setting Up for AR Development, for instructions). We also assume that you have the ARFramework template and all its prerequisites installed. See Chapter 5, Using the AR User Framework, for more details. The completed project can be found in this book's GitHub repository at https://github.com/PacktPublishing/Augmented-Reality-with-Unity-AR-Foundation.

Understanding AR image tracking

Before we start building our project, let's take a moment to learn how AR image tracking works. In this section, I'll introduce some of the basic principles behind image recognition and tracking, and what makes some images better than others for this purpose.

As we know, the principles behind augmented reality involve using compute mechanisms to recognize features in the real world, determine their position and orientation in a 3D space, instantiate virtual objects relative to and anchored within this 3D space, and track the user as they move within this space. Modern devices can accomplish this using their video cameras and other sensors built into the device to performing real-time spatial mapping of the environment. A different approach is for the device to track predetermined images. That is what we will use for the project in this chapter.

Augmented reality technology was born in the 1990s, where QR code-like marker images were used for tracking. An example is shown in the following image:

Figure 8.1 – A basic AR marker

Figure 8.1 – A basic AR marker

Marker images can be used for triggering and positioning virtual objects in the real world. These simplistic yet visually distinct markers are easily detected, even by low-end devices. Such markers are readily detectable because of their distinctive details, high contrast edges, and an asymmetric shape – that is, it's an easily recognizable image with unambiguous top, bottom, left, and right sides. In this way, the detection software can determine which marker image is in view and the orientation of the camera relative to the marker in 3D space.

Taken to the next level, products such as Merge Cube (https://mergeedu.com/cube) have markers on each of its six faces, just like a physical cube that you can hold in your hand. Users can find companion apps with a wide gamut of games, learning, and exploration experiences. Merge offers a Unity package for developers so that you can build your own projects for it too. Merge Cube is depicted in the following image:

Figure 8.2 – Merge Cube provides a 3D tracking cube with markers on each face

Figure 8.2 – Merge Cube provides a 3D tracking cube with markers on each face

Markers can be combined with natural images to provide pleasing and informative yet visually distinct images that also act as AR markers. You'll often see this in AR augmented storybooks or even cereal boxes. This is the approach I have taken in this chapter.

While markers provide the highest reliability, they are not necessarily required for image tracking. Ordinary photographic images can also be used. In AR lingo, these are referred to as natural feature images. Images for tracking must have the same characteristics that make markers reliable – distinctive details, high contrast edges, and an asymmetric shape. Much has been written about the best practices for selecting images. For instance, the AR Core developer guide (https://developers.google.com/ar/develop/java/augmented-images/) contains additional tips about using reference images, including the following:

  • Use an image resolution of at least 300 x 300 pixels. However, a very high resolution does not help with recognition.
  • Color information is not used, so either color or grayscale images are just as good.
  • Avoid images with a lot of geometric features, or too few.
  • Avoid repeating patterns.

The AR Core SDK comes with an arcoreimg tool that can evaluate images and returns a quality score between 0 and 100 for each image, where a score of at least 75 is recommended. Likewise, Unity uses a similar tool when compiling the Image Reference Library in your builds (we'll learn more about this later in this chapter).

Given this general understanding of using image tracking in augmented reality applications, let's begin by defining a fun and interesting project – visualizing our Solar System's planets.

Specifying the Planets project

We are going to build a planet information app that allows users to scan planet cards to visualize a 3D model of each planet in the Solar System. Imagine this being part of a trading card collection or a companion app to a children's science book. When the user points the device's camera at one of the planet cards, they can see a 3D rendering of the planet. Upon pressing an Info button, the user can get additional information about that planet. In this section, I will define the general user experience flow, give you instructions for preparing the planet cards for your own use, and help you collect assets that you'll use in this project.

User experience flow

The general onboarding user workflow will play out as follows:

  1. Startup-mode: The app will start, check the device for AR support, and ask for camera permissions (OS-dependent). Once read, the app will go into Scan-mode.
  2. Scan-mode: The user is prompted to aim the camera at an image for detection and tracking. When at least one image is being tracked, the app goes into Main-mode.
  3. Main-mode: This is where the app responds to new or updated tracked images and allows the user to interact with the planet. When an image is tracked, it determines which planet corresponds to the image and instantiates the planet's game object. If tracking is lost, the app may go back to Scan-mode to prompt the user. If a different image is tracked, the current planet is replaced with the new image's planet.

This workflow is a bit simpler than the ones we implemented in the previous chapters. In that case, we needed the user to scan the environment for trackable planes before starting Main-mode. The user was then asked to deliberately tap the screen to place a virtual object in the scene. Furthermore, in the AR Gallery project, we added Edit-mode to modify pictures that had been added by the user. Much of that is unnecessary in this project; the process is more automated as we let the device detect an image and we instantiate a virtual object in response.

Preparing the planet cards

For this project, we are using printed planet cards as marker images so that we can choose a planet to visualize. You can find a PDF file that contains the cards in the project files for this chapter (in the folder named Printables/). To prepare the cards for this project, follow these steps:

  1. Print out the PlanetCards.pdf file.
  2. Then, cut the sheets into separate cards.
  3. I suggest that you print on thick paper stock or mount the printouts on paperboard to avoid warping, which may affect the software's ability to recognize the images at runtime.

The following photo shows getting these cards ready for use:

Figure 8.3 – Cutting the printed planet cards for this project

Figure 8.3 – Cutting the printed planet cards for this project

These cards were created from a combination of resources that can be found for free on the web. I found the original flashcards on the Kids Flashcards website. Upon going to https://kids-flashcards.com/en/free-printable/solar-system-flashcards-in-english, I downloaded the Solar System flashcards free PDF file.

First, I attempted to use the flashcards as-is, but the pictures were not distinctive enough to be detected individually. So, I decided to add ArUco markers to each one. ArUco is a square marker with a wide black border and inner binary matrix that determines its ID based on OpenCV (the Open Source Computer Vision library, which was developed at the University of Cordoba, Spain; see https://docs.opencv.org/3.2.0/d5/dae/tutorial_aruco_detection.html). I used the online ArUco marker generator at https://chev.me/arucogen/ to make separate markers for each planet.

Then, I used Photoshop to combine the markers with the planet flashcards to make our final planet cards for this project. (The Photoshop PSD file is also included with this chapter's files on GitHub.)

Each planet card is also a separate PNG image. These have been provided for you in the Image Library/ folder. Later in this chapter, we will create an image reference library and add these images. The images are named with the pattern [planetname]-MarkerCard.png; for example, Earth-MarkerCard.png. We'll take advantage of this naming convention in our code.

When the app detects a planet card, the application will instantiate a model of the planet. For this, we need texture images for the planet materials.

Collecting planet textures and data

We need texture images to use as the planet skins of the spherical mesh for each planet. The ones we're using I found at the interesting Solar System Scope project site (https://www.solarsystemscope.com/). These are included with the files for this chapter in this book's GitHub repository and can be downloaded from https://www.solarsystemscope.com/textures/. That said, you can find alternative assets in the Unity Asset Store (https://assetstore.unity.com/?q=solar%20system&orderBy=1), including the classic Planet Earth Free package (https://assetstore.unity.com/packages/3d/environments/sci-fi/planet-earth-free-23399) for Earth itself, which includes cloud cover.

For additional metadata about the planets, I found the Planetary Fact Sheet on the NASA.gov website (https://nssdc.gsfc.nasa.gov/planetary/factsheet/index.html) and more details at https://nssdc.gsfc.nasa.gov/planetary/planetfact.html. We could use some of these details directly while rendering and animating our models, such as the planet diameter (km), rotation period (hours), and tilt (obliquity to orbit in degrees).

With our planet cards, planet skin textures, and other planetary details in hand, we're ready to start building the project.

Getting started

To begin, we'll create a new scene named PlanetsScene using the ARFramework scene template. Follow these steps:

  1. Select File | New Scene.
  2. In the New Scene dialog box, select the ARFramework template.
  3. Press Create.
  4. Select File | Save As. Navigate to the Scenes/ folder in your Assets project, name it PlanetsScene, and click Save.

The new AR scene already has the following set up:

  • AR Session game object.
  • AR Session Origin rig with the raycast manager and plane manager components.
  • UI Canvas is a screen space canvas with child panels; that is, Startup UI, Scan UI, Main UI, and NonAR UI. It also contains the UI Controller component script that we wrote.
  • Interaction Controller is a game object that contains the Interaction Controller component script we wrote, which helps the app switch between interaction modes, including the Startup, Scan, Main, and NonAR modes. It also has a Player Input component that's been configured with the AR Input Actions asset we created previously.
  • The OnboardingUX prefab from the AR Foundation Demos project, which provides AR session status and feature detection status messages, as well as animated onboarding graphics prompts.

We can set the app title now, as follows:

  1. In the Hierarchy window, unfold the UI Canvas object and unfold its child App Title Panel.
  2. Select the Title Text object.
  3. In its Inspector window, change its text content to Planet Explorer.

Using this scene as a basis, we will replace the AR trackable components with an AR Tracked Image Manager one.

Tracking reference images

Our starter scene includes an AR Session Origin with components for Player Input and AR Raycast Manager. It also has a component we do not need in this project, for detecting and tracking planes, which we'll replace with AR Tracked Image Manager instead. Documentation on AR Tracked Image Manager can be found at https://docs.unity3d.com/Packages/[email protected]/manual/tracked-image-manager.html. Then, we'll create an image reference library for our planet card images.

Adding AR Tracked Image Manager

To configure the AR Session to track images, perform the following steps:

  1. In the Hierarchy window, select the AR Session Origin game object.
  2. In the Inspector window, use the 3-dot context menu (or right-click) on AR Plane Manager and select Remove Component.
  3. Using the Add Component button, search for AR and add an AR Tracked Image Manager component.

You'll notice that there is a Serialized Library slot on the component for the reference image library. We'll create that next.

Creating a reference image library

The reference image library contains records for each of the images that the application will be able to detect and track in the real world. In our case, we're going to add the planet card images. In the assets provided in the GitHub repository for this book, there is a folder named Image Library/ that already contains the planet card images we'll be adding to the library. We will start with just the Earth card here; we will add the other planets later in this chapter.

We can create the library by performing the following steps:

  1. In the Project window, find or create a folder named Image Library/.
  2. Right-click on the Image Library/ folder and select Create | XR | Reference Image Library.
  3. With the ReferenceImageLibrary assets selected, in the Inspector window, click Add Image.
  4. Drag the Earth-MarkerCard image from the Project window onto the square image texture slot.
  5. Check the Specify Size checkbox.
  6. If you printed the planet cards from the PDF provided, at scale, the width will be about 8 cm, or 0.08 meters. Otherwise, use a ruler to measure the Earth planet card you printed.
  7. Then, enter the width (0.08) in the X field. The Y value will be automatically updated based on the PNG image's pixel dimensions.
  8. Check the Keep Texture At Runtime checkbox.

The resulting Reference Image Library settings are shown in the following screenshot:

Figure 8.4 – Reference Image Library with the Earth added

Figure 8.4 – Reference Image Library with the Earth added

Now, we can update the AR Tracked Image Manager component, as follows:

  1. In the Hierarchy window, select the AR Session Origin object.
  2. Drag the ReferenceImageLibrary asset from the Project window onto the Serialized Library slot of AR Tracked Image Manager.
  3. Temporarily, while we get this project set up, we'll instantiate an existing prefab object when the image is detected.

    For example, drag the AR Placed Cube prefab from the ARF-samples/Prefabs/ folder onto the Tracked Image Prefab slot (or another similar object).

The AR Tracked Image Manager component should now look as follows:

Figure 8.5 – The AR Tracked Image Manager with the reference image library assigned

Figure 8.5 – The AR Tracked Image Manager with the reference image library assigned

You now have an AR scene that recognizes and tracks images that have been defined in a reference library. Currently, the library only contains the Earth-MarkerCard image. When the image is recognized while running the app, a simple cube will be placed on the Earth planet card.

We're almost ready to try this out. But first, let's configure the user framework's UI and modes.

Configuring the user interaction modes and UI

The scene template, ARFramework, where we started provides a simple framework for enabling user interaction modes and displaying the corresponding UI panels for a mode. This project will start in Startup-mode while the AR Session is initializing so that we can verify that the device supports AR. Then, it will transition to Scan-mode, where it will try to find one of the reference images. Once found, it will transition to Main-mode, where we can support additional user interactions with the app's content.

Scanning for reference images

In Scan-mode, we'll display an instructional graphic prompting the user to point the camera at a planet card with a planet and marker image. Perform the following steps to configure this:

  1. In the Hierarchy window, unfold the UI Canvas game object and unfold its child Scan UI. Select the child Animated Prompt object.
  2. In the Inspector window set Animated Prompt | Instruction to Find An Image.

This will now play the Find Image Clip we defined on the OnboardingUX object, which is provided by the Unity Onboarding UX assets and is already present in our scene hierarchy. What you can expect is shown in the following screen capture. On the left is Startup-mode, where the AR Session is being initialized. On the right is Scan-mode, where the user is prompted to find an image (you can't see the video feed because I'm covering the camera to make the prompt more visible in the screen capture).

Figure 8.6 – Screen captures of Startup mode (left) and Scan mode (right)

Figure 8.6 – Screen captures of Startup mode (left) and Scan mode (right)

Now, we need to set up the Scan-mode's script to know when an image has been found and transition to Main-mode. We'll replace the default ScanMode script with a similar one that references ARTrackedImageManager instead of ARTrackedPlaneManager, as follows:

  1. In the Project window, create a new C# script in your Scripts/ folder by right-clicking and selecting Create | C# Script. Name the new file ImageScanMode.
  2. Edit ImageScanMode and replace its content, as follows:

    using UnityEngine;

    using UnityEngine.XR.ARFoundation;

    public class ImageScanMode : MonoBehaviour

    {

        [SerializeField] ARTrackedImageManager imageManager;

        private void OnEnable()

        {

            UIController.ShowUI("Scan");

        }

        void Update()

        {

            if (imageManager.trackables.count > 0)

            {

                InteractionController.EnableMode("Main");

            }

        }

    }

  3. Save the script. Then, back in Unity, in the Hierarchy window, select the Scan Mode game object (under Interaction Controller).
  4. In the Inspector window, remove the original Scan Mode component using the 3-dot context menu and selecting Remove Component.
  5. Drag the ImageScanMode script onto the Scan Mode object, adding it as a new component.
  6. From the Hierarchy window, drag the AR Session Origin object into the Inspector window and drop it onto the Image Scan Mode | Image Manager slot.

The component will now look as follows:

Figure 8.7 – The Image Scan Mode component

Figure 8.7 – The Image Scan Mode component

Currently, we have created a new scene using the ARFramework template and modified it to use AR Tracked Image Manager and prompt the user to scan for an image accordingly. When an image is detected (for example, the Earth-MarkerCard), a generic game object will be instantiated (for example, the AR Placed Cube prefab). Let's test what we have accomplished so far on the target device.

Build and run

To build and run the scene on your target device, perform the following steps:

  1. Ensure you've saved the work you've done on the current scene by going to File | Save.
  2. Select File | Build Settings to open the Build Settings window.
  3. Click Add Open Scenes to add the current scene to the Scenes In Build list (if it's not already present).
  4. Uncheck all but the current scene, PlanetsScene, from the list.
  5. Then, click Build And Run to build the project.

When the app launches, point your device's camera at the printed Earth planet card. Your virtual cube should get instantiated at that location, as shown in the following screen capture from my phone:

Figure 8.8 – The Earth card has been detected, and the cube has been instantiated

Figure 8.8 – The Earth card has been detected, and the cube has been instantiated

We now have a basic AR scene with image detection set up to recognize the Earth planet card and instantiate a sample prefab at that location. Now, let's make a planet Earth model that we can use instead of this silly cube.

Creating and instantiating a virtual Earth prefab

In this section, we will create prefab game objects for each of the planets. Since each of the planets has similar behaviors (for example, they rotate), we'll first create a generic Planet Prefab, and then make each specific planet a variant of that one. In Unity, prefab variants allow you to define a set of predefined variations of prefabs, such as our planet one (see https://docs.unity3d.com/Manual/PrefabVariants.html). We'll write a Planet script that animates the planet's rotation and handles other behavior. Each planet will have its own "skin" defined by a material, along with a base texture map, which we downloaded earlier from the web.

In this section, we'll create a generic Planet Prefab object, create an Earth Prefab as a variant, add planet metadata by writing a Planet component script, and implement a planet rotation animation.

Creating the generic Planet Prefab

The Planet Prefab contains a 3D sphere that gets rendered with each planet's texture image. Planets spin along their axes, so we'll set up a hierarchy with an Incline transform that defines this incline axis. Follow these steps:

  1. In your Project window, right-click and select Create | Prefab (create the folder first if necessary). Name it Planet Prefab.
  2. Double-click (or select Open Prefab in the Inspector window) to open the prefab for editing.
  3. From the main menu, select GameObject | Create Empty and name it Incline.
  4. Right-click the Incline game object in the Hierarchy window and select 3D Object | Sphere. Name it Planet.
  5. It will be useful to have any planets we instantiate in the scene on a specific layer. I will name this layer PlacedObjects. (I introduced and discussed layers in a previous chapter). With its root Planet Prefab object selected, in the top right of its Inspector window, click the Layer drop-down list and select PlacedObjects.

    If the PlacedObjects layer doesn't exist, select Add Layer to open the Layers manager window. Add the name PlacedObjects to one of the empty slots. In the Hierarchy window, click the FramedPhoto Prefab object to get back to its Inspector window. Again, using the Layers drop-down list, select PlacedObjects.

    You will then be prompted with the question, Do you want to set layer to PlacedObjects for all child objects as well? Click Yes, Change Children.

  6. Save the prefab.

This is very simplistic right now (only a sphere child object is being parented by an Incline transform), but it will serve as a template for each planet prefab that we add. The Planet Prefab hierarchy is shown in the following screenshot:

Figure 8.9 – The Planet Prefab hierarchy

Figure 8.9 – The Planet Prefab hierarchy

Each planet will be rendered with a skin representing an actual view of that planet. Before creating the Earth prefab, let's take a moment to understand render materials and the texture images we are going to use.

Understanding equirectangular images

When Unity renders a 3D model, it starts with a 3D mesh that describes the geometry. Much like a fishing net, a mesh is a collection of vertices and vectors, with the vectors connecting these vertices, organized as triangles (or sometimes four-sided quads) that define the surface of the mesh. The following illustration shows a wireframe view of a sphere mesh on the left. On the right is a rendered view of the sphere, with a globe texture mapped onto its 3D surface:

Figure 8.10 – Sphere mesh (left) and rendered sphere with texture (right)

Figure 8.10 – Sphere mesh (left) and rendered sphere with texture (right)

A texture image is just a 2D image file (for example, a PNG file) that is computationally mapped onto the 3D mesh's surface when it is rendered. Think of unraveling a globe as a 2D map, like cartographers have been doing for centuries. A common 2D projection is known as equirectangular, where the center (equator) is at the correct scale and the image gets increasingly stretched as it approaches the top and bottom poles. The following image shows the equirectangular texture of the preceding globe (illustration by Stefan Kuhn):

Figure 8.11 – Equirectangular texture that defines the skin of a sphere

Figure 8.11 – Equirectangular texture that defines the skin of a sphere

Information – Equirectangular Images Are Also Used in 360-Degree Media and VR

Equirectangular images are also known as 360-degree images and used in virtual reality applications. In VR, the image is effectively mapped to the inside of a sphere, where you're viewing from the inside rather than the outside of a globe!

For our project, we have texture images for each of the planets. The Mars one, for example, is as follows:

Figure 8.12 – Texture map image for Mars

Figure 8.12 – Texture map image for Mars

To create a prefab for a specific planet, such as Earth, we'll need to create a material that uses the Earth texture image. We'll build that now.

Creating the Earth prefab

The Earth prefab will be a variant of the Planet Prefab, with its own Earth Material. Create it by performing the following steps:

  1. In the Project window, right-click Planet Prefab and select Create | Prefab Variant. Name the new asset Earth Prefab.
  2. Double-click Earth Prefab (or select Open Prefab in the Inspector window).
  3. In the Project window, right-click in your Materials/ folder (create one if necessary) and select Create | Material. Rename it Earth Material.
  4. Drag Earth Material from the Project window and drop it onto the Planet game object.
  5. Locate the Earth texture image (for example, in Planet Textures/earth) in the Project Assets and drag it onto the Surface Inputs | Base Map texture chip. The following screenshot shows Earth Material in the Inspector window:
    Figure 8.13 – Earth Material with the Base Map texture defined

    Figure 8.13 – Earth Material with the Base Map texture defined

  6. Let's pick a default size for our planet when added to the scene. Unless you want to place a 1-meter diameter planet into your scene (!), we need to set its Scale.

    Select the Planet child object in the Hierarchy window.

  7. In its Inspector window, set Transform | Scale | X, Y Z to (0.1, 0.1, 0.1).
  8. Likewise, to rest the planet on the image's surface, we could set its Y position to 0.05. But to let it hover a little above, we will set Transform | Position | Y to 0.075.
  9. Save the prefab and exit back to the Scene hierarchy.

Use this prefab instead of the AR Placed Cube prefab in the AR Tracked Image Manager component on the AR Session Origin object. Later, we'll manage this more correctly using a script but for now, let's just try it out:

  1. In the Hierarchy window, select the AR Session Origin game object.
  2. Drag Earth Prefab from the Project window into the Inspector window and drop it into the AR Tracked Image Manager | Tracked Image Prefab slot.
  3. Build and Run the scene.

This time, when you point the camera at the Earth planet card, the Earth prefab will appear, as shown in the following screen capture:

Figure 8.14 – While tracking the Earth planet card, the app instantiates an Earth prefab

Figure 8.14 – While tracking the Earth planet card, the app instantiates an Earth prefab

This looks pretty nice. The prefab could also include other information about the planet. We'll look at how to do this next.

Adding planet metadata

Each planet prefab can include additional information about that planet. We can capture this in the Planet script of the prefab, as follows:

  1. From the Project window, open Planet Prefab for editing.
  2. In the Project window's Scripts/ folder, create a new C# script named Planet.
  3. Drag the Planet script onto the root Planet Prefab game object, adding it as a component.
  4. Open the Planet script in your code editor and write the following:

    using UnityEngine;

    public class Planet : MonoBehaviour

    {

        public string planetName;

        public string description;

    }

  5. Save the script. Then, in Unity, Save the prefab.

    Although we have made all these changes to the Planet Prefab, the Earth Prefab inherits everything because it is a prefab variant.

  6. Now, open Earth Prefab for editing.
  7. In the Planet Name field, enter Earth.
  8. In the Description field, enter a text description that we'll use later in the project, such as Earth is the third planet from the Sun and the only astronomical object known to harbor and support life
  9. Save the prefab.

We also can ascribe behaviors to the planet prefab, such as rotation about its axis.

Animating the planet's rotation

Planets spin. Some faster than others. Mercury just barely – it rotates once every 59 Earth days, while it orbits the Sun in 88 Earth days! And a planet's axis of rotation is not perfectly vertical (relative to its orbit around the Sun). Earth, for example, is tilted by 23.4 degrees, while Venus rotates on its side at 177.4 degrees! OK, enough science trivia – let's animate our Earth model. We're going to add a Planet behavior script to the Planet Prefab that rotates the planet along its rotation axis. Follow these steps to do so:

  1. Open the Planet script in your code editor and add the following code:

        [SerializeField] private float inclineDegrees =        23.4f;

        [SerializeField] private float rotationPeriodHours =        24f;

        [SerializeField] private Transform incline;

        [SerializeField] private Transform planet;

        public float animationHoursPerSecond = 1.0f;

        void Start()

        {

            incline.Rotate(0f, 0f, inclineDegrees);

        }

        void Update()

        {

            float speed =           rotationPeriodHours * animationHoursPerSecond;

            planet.Rotate(0f, speed * Time.deltaTime, 0f);

        }

At the top of the class, we will declare variables for inclineDegrees (Earth is 23.4) and rotationPeriodHours (Earth is 24). We will also define references to the prefab's incline and planet child objects.

There's also a public animationHoursPerSecond, which sets the animation speed. I've initialized it to 1.0, which means the Earth will complete one rotation in 24 seconds.

The Start() function sets up the Incline angle by rotating along the Z-axis. This only needs to be done once.

The Update() function rotates the planet about its local Y-axis. Since the planet is parented by the Incline transform, it appears to rotate about the tilted incline axis. Multiplying the speed by Time.deltaTime each Update is a common Unity idiom for calculating how an object's Transform changes from one frame to the next, where deltaTime is the fraction of a second since the previous Update.

After saving the script, back in Unity, do the following:

  1. From the Project window, open Planet Prefab for editing.
  2. Ensure the root Plane Prefab game object is selected in the Hierarchy window.
  3. Drag the Incline game object from the Hierarchy window into the Inspector window before dropping it onto the Planet | Incline slot.
  4. Drag the Planet object onto the Planet | Planet slot.

The Planet component will now look like this in the Inspector window:

Figure 8.15 – The Planet component on the Planet Prefab

Figure 8.15 – The Planet component on the Planet Prefab

Please now Build and Run the project. When the Earth is instantiated, it will be tilted and rotating at the rate of one full rotation every 24 seconds.

At this point, we have a basic AR scene with image tracking. It lets the AR Tracked Image Manager instantiate our Earth Prefab directly when an image is detected. Currently, it doesn't distinguish what image is detected (supposing you had multiple images in the reference library) and always instantiates an Earth Prefab. We need to make the app more robust, and we can do this from the Main-mode.

Building the app's Main-mode

As you now know, AR Tracked Image Manager (on the AR Session Origin game object) performs 2D image tracking. But so far, we've being using the AR Tracked Image Manager incorrectly! We populated its Tracked Image Prefab property with our Earth Prefab. That's a no-no. According to the Unity documentation, "ARTrackedImageManager has a "Tracked Image Prefab" field; however, this is not intended for content" (https://docs.unity3d.com/Packages/[email protected]/manual/tracked-image-manager.html). Currently, when any reference image is recognized, the Earth Prefab will always be instantiated.

Rather, when the app is in Main-mode, we should determine which planet card image is being tracked and instantiate the corresponding planet prefab for that card. So far, we only have one planet, Earth, in the image reference library. However, later in this chapter, we'll expand the project for all the planets. We can start by removing the prefab from the AR Tracked Image Manager component, as follows:

  1. In the Hierarchy window, select the AR Session Origin game object.
  2. In the Inspector window, delete the contents of the AR Tracked Image Manager | Tracked Image Prefab slot, as shown in the following screenshot:
Figure 8.16 – AR Tracked Image Manager with the default prefab field cleared

Figure 8.16 – AR Tracked Image Manager with the default prefab field cleared

When no prefab is specified in AR Tracked Image Manager, an empty game object is created with an ARTrackedImage component on it. Now, we can instantiate the prefab as a child of that.

In our scene framework, the app starts in Startup-mode, then goes into Scan-mode once the AR Session is ready. When Scan-mode detects a reference image, it goes into Main-mode by enabling the Main Mode game object under Interaction Controller. This displays the Main UI panel. Let's build this panel now.

Writing the PlanetsMainMode script

In this section, we will write a new PlanetsMainMode script to replace the default MainMode one provided in the default scene template. Like other modes in our framework, it will display the appropriate UI panel when enabled. Then, when an image is tracked, it will find the corresponding planet prefab and instantiate it.

The script needs to figure out which image the AR software found and decide which prefab to instantiate as a child of the tracked image. In our case, we'll use the name of the detected image file to determine which planet card is recognized (by design, each card image is prefixed with the planet's name; for example, Earth-MarkerCard). The script will implement a serializable dictionary we can use to look up the planet prefab for each planet name, using the Serialized Dictionary Lite Asset package (you already have this package installed because ARFramework also requires it. See https://assetstore.unity.com/packages/tools/utilities/serialized-dictionary-lite-110992 for more information).

Begin by performing the following steps:

  1. In your Project Scripts/ folder, create a new C# script named PlanetsMainMode.
  2. In the Hierarchy window, select the Main Mode game object (under Interaction Controller).
  3. In its Inspector window, remove the default Main Mode component using the 3-dot context menu and selecting Remove Component.
  4. Drag the PlanetMainMode script from the Project window onto the Main Mode object, adding it as a new component.
  5. Double-click the PlanetMainMode script to open it for editing.
  6. Begin by adding the following using assembly declarations at the top of the file:

    using UnityEngine;

    using RotaryHeart.Lib.SerializableDictionary;

    using UnityEngine.XR.ARFoundation;

    using TMPro;

    using UnityEngine.UI;

  7. When an image is tracked, we need to find which planet prefab to instantiate. At the top of the file, define a PlanetPrefabDictionary as follows, and declare a planetPrefab variable for it:

    [System.Serializable]

    public class PlanetPrefabDictionary : SerializableDictionaryBase<string, GameObject> { }

    public class PlanetsMainMode : MonoBehaviour

    {

        [SerializeField] PlanetPrefabDictionary         planetPrefabs;

  8. When this mode is enabled, similar to the original MainMode script, we'll show the Main UI panel:

        private void OnEnable()

        {

            UIController.ShowUI("Main");

        }

  9. Likewise, we'll enter Main-mode after Scan-mode has determined it has started tracking an image. So, OnEnable should also instantiate planets for the tracked images. Add a reference to imageManager at the top of the class:

        [SerializeField] ARTrackedImageManager imageManager;

    Then, update OnEnable:

        void OnEnable()

        {

            UIController.ShowUI("Main");

            foreach (ARTrackedImage image in                  imageManager.trackables)

            {

                InstantiatePlanet(image);

            }

        }

    This loops through the trackable images and calls InstantiatePlanet for each one.

  10. Implement InstantiatePlanet, as follows:

        void InstantiatePlanet(ARTrackedImage image)

        {

            string name =            image.referenceImage.name.Split('-')[0];

            if (image.transform.childCount == 0)

            {

                GameObject planet =                Instantiate(planetPrefabs[name]);

                planet.transform.SetParent(image.transform,                false);

            }

            else

            {

                Debug.Log($"{name} already instantiated");

            }

        }

    The InstantiatePlanet function determines the planet's name from the tracked image filename (for example, Earth-MarkerImage) by assuming the images follow our naming convention. It makes sure we don't already have the planet object in the scene. If not, the planet prefab is instantiated and parented to the tracked image object. (We pass false as a second parameter so that the planet is positioned relative to the tracked image transform. See https://docs.unity3d.com/ScriptReference/Transform.SetParent.html.)

  11. Save the script.
  12. Back in Unity, make sure you have the Main Mode game object selected in the Hierarchy window.
  13. Drag the AR Session Origin object from the Hierarchy window into the Inspector window, dropping it onto the Image Manager slot.
  14. In the Inspector window, click the + button at the bottom right of the Planets Main Mode | Planet Prefabs list.
  15. Type the word Earth into the Id slot.
  16. Unfold the item and, from the Hierarchy window, drag the Earth Prefab object on the Value slot in the Inspector window.
  17. Use File | Save to save your work.

If you Build and Run now, the app will behave much the same as it did before – after Scan-mode detects an image, it enters Main-mode. But instead of AR Tracked Image Manager instantiating the Earth Prefab, instantiation is performed in PlanetsMainMode when it is enabled. Now, the code is ready to detect different planet card images and instantiate different corresponding planet prefabs. We will start by adding Mars.

Expanding the project with multiple planets

To add another planet to the project, we need to add its planet card image to the Reference Image Library, create its planet prefab, including a material for rendering the planet skin, and add the reference to the planetPrefabs list in PlanetsMainMode. Then, we'll update the script to handle tracking multiple planets. Let's walk through the steps for adding Mars.

Adding the planet card image to the Reference Image Library

Perform the following steps to add Mars to our Reference Image Library:

  1. Locate and select your ReferenceImgeLibrary asset in the Project window. (If you've been following along, then it should be located in the Image Library/ folder.)
  2. In its Inspector window, click Add Image.
  3. Locate and drag the Mars-MarkerCard image from the Project window and drop it onto the empty image Texture slot in the Inspector window.
  4. Check the Specify Size checkbox and enter the same Physical Size | X value you used for the Earth one. Mine measures at 0.08 meters (8 cm).
  5. Also, check the Keep Texture At Runtime checkbox.

The Reference Image Library should now look as follows:

Figure 8.17 – Reference Image Library with the Mars-MarkerCard image added

Next, we'll create the Mars Prefab and material.

Creating the planet prefab

To create the planet prefab, we'll copy and modify the Earth Prefab assets. Perform the following steps:

  1. In the Project window, locate the Earth Prefab asset (this will probably be in your Prefabs/ folder).
  2. Select Edit | Duplicate (or use Ctrl-D on the keyboard) to duplicate it. Rename the copy Mars Prefab.
  3. Open Mars Prefab for editing. Select the child Planet game object.
  4. In the Project window, right-click inside your Materials/ folder and select Create | Material. Name it Mars Material.
  5. Drag Mars Material onto the Planet object.
  6. In the Project window, locate the mars texture file (in the Planet Textures/ folder) and drag it onto the Mars Material | Base Map texture slot.

    The Mars Prefab Planet should now look as follows:

    Figure 8.18 – Mars Prefab with its Planet set to the Mars Material

    Figure 8.18 – Mars Prefab with its Planet set to the Mars Material

  7. Next, we'll set the Mars Planet metadata. In the Hierarchy window, select the root Mars Prefab game object.
  8. In the Inspector window, change the Planet component's parameters; that is, Planet Name: Mars; Incline Degrees: 25.2; Rotation Period Hours: 24.7. For its Description, you can add something similar to Mars is the fourth planet from the Sun and the second-smallest planet in the Solar System.
  9. Save the prefab and return to the scene Hierarchy (using the < button at the top left of the Hierarchy window).

Now, we can add the prefab to the Main-mode Planet Prefabs dictionary, as follows:

  1. In the scene Hierarchy, select the Main Mode game object (under Interaction Controller).
  2. In the Inspector window, click the + button at the bottom right of the Planets Main Mode | Planet Prefabs list.
  3. Type the word Mars into the Id slot.
  4. Unfold the item and, from the Hierarchy window, drag the Mars Prefab object onto the Value slot in the Inspector window.

The Planets Main Mode component should now look as follows:

Figure 8.19 – The Planets Main Mode component's Planet Prefabs dictionary with Mars added

Figure 8.19 – The Planets Main Mode component's Planet Prefabs dictionary with Mars added

If you Build and Run now, when in Scan-mode, point the camera at your Mars planet card. The Mars 3D object will be added to the scene, rotating in all its glory!

Unfortunately, after doing this, if you move the camera to scan the Earth planet card, nothing will happen. Let's fix that.

Responding to detected images

Your scripts can subscribe to events so that they're notified when an image is being tracked, updated, or removed. Specifically, we can implement an OnTrackedImageChanged function to handle these events. We can use this in the PlanetsMainMode script, as follows:

  1. Open the PlanetsMainMode script for editing again and add the following code:

        void OnTrackedImageChanged         (ARTrackedImagesChangedEventArgs eventArgs)

        {

            foreach (ARTrackedImage newImage in                  eventArgs.added)

            {

                InstantiatePlanet(newImage);

            }

        }

  2. Add the following line to your OnEnable function, adding a listener to imageManager:

            imageManager.trackedImagesChanged +=            OnTrackedImageChanged;

  3. Likewise, remove the listener in OnDisable:

        void OnDisable()

        {

            imageManager.trackedImagesChanged -=            OnTrackedImageChanged;

        }

    When ARTrackedImageManager detects a new image, the Main-mode script will kick in. It contains a listener for the events and will call InstantiatePlanet for any newly tracked images.

  4. If the app completely loses image tracking, we should go back to Scan-mode and display its instructional graphic, prompting the user to find a reference image. Add this check to Update, as follows:

        void Update()

        {

            if (imageManager.trackables.count == 0)

            {

                InteractionController.EnableMode("Scan");

            }

        }

    Tip – Tracking the State of Individual Trackables

    AR Foundation also provides you with the current tracking state of each trackable image individually. Given a trackable image (ARTRackedImage), you can check its trackingState for Tracking – image is actively tracking, Limited – image is being tracked but not reliably, or None – the image is not being tracked. See https://docs.unity3d.com/Packages/[email protected]/manual/tracked-image-manager.html#tracking-state. In this project, we will only go back to Scan mode when no images are being tracked, so we don't necessarily need this extra level of status monitoring.

OK – this is getting pretty robust. Build and Run the project again, this time scanning either (or both) the Earth and Mars planet cards. We've got planets! The following screen capture shows the app running, with the addition of the information UI at the bottom of the screen, which we will add in the next section:

Figure 8.20 – Earth and Mars rendered at runtime

Figure 8.20 – Earth and Mars rendered at runtime

Go ahead and add the rest of the planets to your project by following these same steps. As we mentioned earlier in this chapter, referencing the NASA data provided at https://nssdc.gsfc.nasa.gov/planetary/factsheet/index.html, use their Length of Day row for our Rotation Period Hours parameter, and their Obliquity to Orbit for our Incline Degrees parameter. You'll notice that some planets rotate imperceptibly slowly (for example, Venus has 2,802-hour days) and spin in a direction opposite to Earth (Venus and Uranus have negative rotation periods), whereas Jupiter and Saturn rotate more than twice as fast as Earth (9.9 and 10.7 hours per day, respectively). The Planet script already includes an animation speed scalar, animationHoursPerSecond, that you can use to modify the rotation rates that are visualized in the app.

Now that our application supports multiple planets, you might want to tell the user more about the specific planet that they are looking at. Let's add this capability to Main-mode so that it responsively updates the UI.

Making a responsive UI

In this section, we'll add an info panel to the bottom of the screen (as shown in the preceding screen capture). When you point the camera at one planet or another, we'll show the planet's name, as well as an Info button, which will cause a text box to appear that contains more information about that planet.

Creating the Main-mode UI

When the app is in Main-mode, the Main UI panel is displayed. On this panel, we'll show the name of the current planet and an Info button for the user to press when they want more details about that planet. Perform the following steps:

  1. In the Hierarchy window, unfold the UI Canvas object and unfold its child Main UI object.
  2. The default child text in the panel is a temporary placeholder, so we can remove it. Right-click the child Text object and select Delete.
  3. Create a subpanel by right-clicking on Main UI and selecting UI | Panel. Rename it Info Panel.
  4. Use Anchor Presets to set Bottom-Stretch. Then, use Shift + Alt + Bottom-Stretch to make a bottom panel. Then, set its Rect Transform | Height to 175.
  5. I set my background Image | Color to opaque white with Alpha: 255.
  6. Create a text element for the planet name. Right-click Info Panel and select UI | Text – TextMeshPro. Rename the object Planet Name Text.
  7. On the Planet Name Text TextMeshPro – Text component's Text Input, set the content with a temporary string such as [Planet name].
  8. Set the text properties; for example, Anchor Presets: Stretch-Stretch (and Shift + Alt + Stretch-Stretch); Text Align: Left, Middle; Rect Transform | Left: 50; Text Vertex Color: black; Font Size: 72.
  9. Create an Info button. Right-click Info Panel and select UI | Button – TextMeshPro. Rename it Info Button.
  10. Set the button properties; for example, Anchor Presets: Right-Middle (and Shift + Alt + Right-Middle); Width and Height: 150, 150; Pos X: -20.
  11. Unfold Info Button. On its child Text object, set Top: -50 and its text content to Info.
  12. Right-click Info Button and select UI | Text – TextMeshPro. On the new text element, set its text value to a question mark, ?, Font Size to 72, Color to black, Alignment to Center, Middle, and Pos Y to -15.
  13. We're going to use this button to toggle a details panel on and off. So, let's replace its Button component with a Toggle instead. With the Info Button object selected in the Hierarchy window, in the Inspector window, remove the Button component using the 3-dot context menu and selecting Remove Component.
  14. Select Add Component, search for toggle, and add a Toggle component.

My main Info Panel now looks as follows:

Figure 8.21 – Main UI's Info Panel with Planet Name Text and an Info Button

Figure 8.21 – Main UI's Info Panel with Planet Name Text and an Info Button

The Planet Name Text's content will be filled in at runtime. Let's add that code now.

Pointing the camera to show information

The plan is that with one or more virtual planets instantiated in the scene, the user can point the camera at a planet so that it displays the planet's name in the Info Panel. This can be implemented using a Physics Raycast. (Raycasts were introduced and explained in the previous chapters. See https://docs.unity3d.com/ScriptReference/Physics.Raycast.html). Recall that at the beginning of this chapter, we put the Planet Prefab on a layer named PlacedObjects. We'll make use of that here.

Make the following changes to the PlanetsMainMode script:

  1. Ensure the script file contains the following assembly references:

    using TMPro;

    using UnityEngine.UI;

  2. At the top of the class, declare and initialize references to the AR camera and layerMask variables, as follows:

        Camera;

        int layerMask;

        void Start()

        {

            camera = Camera.main;

            layerMask =             1 << LayerMask.NameToLayer("PlacedObjects");

        }

  3. Also, add references to the planetName and infoButton UI elements in the Info Panel:

        [SerializeField] TMP_Text planetName;

        [SerializeField] Toggle infoButton;

  4. We can initialize the UI settings when the mode is enabled. Please add the following lines to the OnEnable function:

            planetName.text = "";

            infoButton.interactable = false;

  5. Then, add the following highlighted code to the Update function:

        void Update()

        {

            if (imageManager.trackables.count == 0)

            {

                InteractionController.EnableMode("Scan");

            }

            else

            {

                Ray = new Ray(camera.transform.position,                  camera.transform.forward);

                RaycastHit hit;

                if (Physics.Raycast(ray, out hit,                 Mathf.Infinity, layerMask))

                {

                    Planet = hit.collider.                    GetComponentInParent<Planet>();

                    planetName.text = planet.planetName;

                    infoButton.interactable = true;

                }

                else

                {

                    planetName.text = "";

                    infoButton.interactable = false;              

                }

            }

        }

  6. Save the script. Back in Unity, select the Main Mode object in the Hierarchy window.
  7. Drag the Planet Name Text game object from the Hierarchy window (under UI Canvas / Main UI / Info Panel) into the Planets Main Mode | Planet Name slot.
  8. Drag the Info Button object onto the Info Button slot.

Go ahead and Build and Run the project one more time. While viewing one or more planets, as you point the device's camera at one of them, the planet's name will be shown in the Info Panel at the bottom of the screen.

Lastly, let's set up the Info Button and description display.

Displaying information details

When the user is pointing their camera at a virtual 3D planet in the scene, we show the name of the planet in the Info Panel. When the user clicks the Info button, we want to show more information about the planet, such as its description text. Let's add a text panel for that now by performing the following steps:

  1. In the Hierarchy window, select the Main UI game object (under UI Canvas) and right-click and select UI | Panel. Rename it Details Panel.
  2. It has already been set to Stretch-Stretch, which is what we want. But let's adjust its size. Set Left: 30, Right: 30, Top: 150, and Bottom: 200.
  3. Right-click Details Panel and select UI | Text – TextMeshPro. Rename it Details Text.
  4. Format the text area; for example, set its Anchor Presets to Stretch-Stretch (and Shift + Alt + Stretch-Stretch), its Text Vertex Color to black, its Font Size to 48, its Rect Transform Left, Right, Top, Bottom to 30, 30, 30, 30, and its Alignment: to Center, Middle.

Now, add control of this panel to the PlanetsMainMode script, as follows:

  1. Add references to detailsPanel and detailsText at the top of the class:

        [SerializeField] GameObject detailsPanel;

        [SerializeField] TMP_Text detailsText;

  2. Ensure the panel is hidden when the mode is enabled. Add the following line to the OnEnable function:

           detailsPanel.SetActive(false);

  3. Initialize the panel's content when a planet is being selected. That is, in Update, we must set detailsText at the same time we set planetName:

                if (Physics.Raycast(ray, out hit,                 Mathf.Infinity, layerMask))

                {

                    Planet = hit.collider.                    GetComponentInParent<Planet>();

                    planetName.text = planet.planetName;

                    detailsText.text = planet.description;

                    infoButton.interactable = true;

                }

                else

                {

                    planetName.text = "";

                    detailsText.text = "";

                    infoButton.interactable = false;

                }          

    Save the script. Back in Unity, we'll wire up the Info Button toggle.

  4. With Info Button selected in the Hierarchy window, in the Inspector window, click the + button at the bottom right of the Toggle | On Value Changed action list.
  5. From the Hierarchy window, drag the Details Panel game object onto the On Value Changed | Object slot.
  6. From the Function selection list, choose GameObject | Dynamic Bool | SetActive.
  7. Save the scene.

Now, when you Build and Run the project and view a planet, then press the Info button, the Details Panel will be displayed alongside the planet's description text, as shown in the following screen capture from my phone:

Figure 8.22 – Displaying description text about Mars in the toggled Details Panel

Figure 8.22 – Displaying description text about Mars in the toggled Details Panel

In this section, we added a responsive UI to the scene. When the user points their device camera at a virtual planet that's been instantiated in the scene, the name of the planet is displayed in the Info Panel at the bottom of the screen. If the user taps the Info button, a text panel is toggled, showing additional details about that specific planet.

Can you think of additional ways to improve this project?

Summary

In this chapter, you built an AR project that lets you visualize and learn about planets in our Solar System. The scene uses AR image detection and tracks the planet cards that you printed out from the PDF file provided with the files for this book. Each planet card image includes a distinct marker with unique details, high contrast edges, and asymmetric shapes, making them readily detectable and trackable by the AR system. You set up the AR Session to track images using the AR Trackable Image Manager component and built a Reference Image Library asset with the planet card images.

You then created a generic Planet Prefab with a Planet script that controls the rotation behavior and metadata for a planet. Then, you created separate prefab variants for each planet. You wrote a PlanetsMainMode script that instantiates the correct planet prefab when a specific planet card image is detected. This allows multiple tracked images and planets to be present in the scene. Then, you added a responsive UI where the user can point their device camera to an instantiated planet and get additional information about that virtual object.

In the next chapter, we'll explore another kind of AR application: flipping the device camera so that it's facing the user to make selfie face filters.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset