Body and Mind

The first time I turned on my Oculus Rift in 2018 I experienced a formative mental shift. On starting, the Oculus application mapped a translucent, blue, human-like hand to my own through my VR headset. As I moved my fingers along the touch controller, the Oculus system’s sensors seemed to track their individual movement. I could point with one finger, I could make a fist, and I could give a thumbs up—all in virtual reality (Figure 7-2). I lost the sense of the controller in my hand. I had identified my mind, my sense of self, with the digital representation of my body.

Other mixed reality systems, such as the Hololens 2, forgo representation completely. Inside-out tracking, which moves infrared sensors from external towers to sensors inside the headset, provides the ability to track a user’s hands without controllers. It likely won’t be long until both MR and VR headsets interpret the presence of a user’s body without controllers at all. Unconscious association of the self with a virtual avatar might become more seamless and more natural over time.

This is the next level of media to which VR evangelists refer when they speak of this great, new technology. The most powerful, most useful feature of VR is the synthesis it catalyzes between the outer and inner self. The moment of transcendence, the moment that clicks within a virtual experience when your body leaves your mind—that is the wonder of VR. It is also the foundation of what we explore in this chapter.

The Unified Whole

If the core of VR is immersion then its antithesis is disruption. Feedback determines the quality of an immersive experience. Does what the users see conform with their expectations? We can orchestrate these expectations through visual and audio cues in our experiences. For example, a highly stylized environment might allow a user to loosen their expectations of realistic physics. A cartoonish, modularly designed, voxel-based world, however, might not prepare a user for a gritty, violent, dramatic experience. At a more fundamental level, however, we find the success of immersion within the synchrony of the brain.

Identification of wholeness, of bodyness, we feel as humans when we watch our fingers move is a fulfillment of the feedback loop between the sensory, motor, and visual cortices of our brains (Figure 7-3). I will an action, I carry it out, and I validate its occurrence. A stuttering frame rate, an unrealistic response to physics, and inaccurate input feedback are the death knells of VR. The human mind is unforgiving of false reality. Any way the movement of our users does not match the visual feedback expected by their brain will tear them out of an immersive state we hope to design them into. How, then, do we capture a user’s hand movement to foster the experience of immersion?

Step 1: Project Setup

An easy reference for getting a Unity VR project setup with VRTK is available at https://github.com/ExtendRealityLtd/VRTK#getting-started.

Create a new VR-supported, 3D project in Unity. If you are using Unity 2019 navigate to the Window ➤ Package Manager console and install the XR Legacy Input Handlers.

Here, we follow the now familiar step of cloning the VRTK GitHub repository into our Unity Project’s Asset folder. Use the following procedure from the VRTK Getting Started documentation to clone the repo.

Before we drag the UnityXRCameraRig and TrackedAlias objects into our scene from the VRTK prefabs folder in our Project window, let’s create a demo of our scene in simple 3D.

To begin, we import our 3D model into our Assets folder. If you plan to use the 3D model of the building exterior I’ve provided for this exercise, you can download it from the exercise repo. Unzip the file and drag its contents into your project’s Asset folder. Unity might notify you of a warning about the model’s “normals” on its default mesh (Figure 7-4). If the warning appears yellow in your console you can ignore it for the purpose of this exercise. Select the model prefab that is identified by the name “model” preceded by a triangle identifying it as a parent of child objects. Drag the model into the Scene Hierarchy (Figure 7-5).

You might notice the orientation of the model is askew in our Scene view (Figure 7-6). This can occur often when importing 3D assets because the zeroed, default positioning of the model might be determined by the asset artist on exporting from 3D modeling software such as Blender, Maya, or Cinema4D. Change the values of the parameters in the Transform component of the model to set the scene to your liking.

If you’re using the 3D model provided for this exercise, you can also feel free to copy the Transform settings I’ve used for the game objects in my scene, which include a plane object for the ground (Figure 7-7). Listed here are the game objects in my Scene Hierarchy and the properties I’ve set on their transforms.

Alternatively, you can use whatever model you prefer to set your scene or simply construct your own building facade using object primitives (cubes, planes, cylinders) from within the Unity Editor. The purpose of this exercise is to learn the process of placing lights in a Unity scene and controlling the appearance of lighting through user input. As long as you follow along with the placement of lights and the scripting, the purpose of the project will not be lost on you.

Step 2: Swap the Skybox

Lighting is the primary technique we have at our disposal as Unity developers to manipulate the mood of our scene. Because the model Annie uses is of a decrepit, old inn, it doesn’t feel right to Elgin, her director, that the model sits beneath a blue sky lit by the sun. Instead, Annie wants the user, on entering her scene, to feel an ominous threat of possible danger. Anything could be lurking in the night.

To create this vibe, let’s swap out the default, sunny, blue Skybox in our scene with a black, starless night. To accomplish this, we simply follow the steps from Exercise 1. First, let’s create a new folder in our Assets folder called Materials. In the Project window, right-click inside the Materials folder and select Create ➤ Material. Name the material black. In the Inspector, set the black material shader to Skybox/Cubemap, and set its Tint Color value to black (Figure 7-8).

As we did in Exercise 1, let’s open up our Skybox properties and swap out its material. Navigate to Window ➤ Rendering ➤ Lighting Settings. Into the Skybox Material field beneath the Lighting ➤ Environment Property, drag and drop the Black material we created.

Step 3: Change the Value of the Directional Light

Speaking of light, let’s add some to our scene! First, let’s start by adding a little bit of moonlight. We can re-create the ambience of moonlight through the use of a Unity Directional Light. Right-click in the Scene Hierarchy and select Light ➤ Directional Light (Figure 7-9).

Unity places a default Directional Light object in our scene. For convenience sake, if you haven’t already, delete the original Directional Light object that comes as a default object with each new Unity Scene. Name our new Directional Light object MoonLight. The following are the Transform settings I set for the MoonLight object in my scene (Figure 7-10):

• Transform.Position: 9, 11, -16

• Transform.Rotation: 37, -25, 0

• Transform.Scale 1, 1, 1

In the Inspector, with the MoonLight object highlighted in the Scene Hierarchy, notice the component called Light attached to the game object (Figure 7-11). Remember, every object in our scene is an actor, ironically, even the lights. As actors, each game object has the capacity to hold components that distinguish it from other game objects. Of course, a Directional Light game object is distinct from an empty game object that only holds a Transform because of its component called Light.

Let’s manipulate the settings of our Light component to better simulate moonlight. In the Inspector, confirm that the Light.Type is Directional. Using the color picker, set the Light.Color property to a cool, whiteish blue.

Let’s wrap up the settings on our MoonLight game object’s Light component by setting its Intensity to 0.3 and its Shadow Type to Soft Shadows. These settings, like most of the others related to the appearance and location of game objects in our scenes, are to taste. Fiddle away at the knobs until you’ve tweaked the appearance of the scene to your liking (Figure 7-12). You’re learning Unity and VRTK to better express your creative vision, after all, so you might as well practice along the way.

Step 4: Add a Doorway Point Light

With the settings of our MoonLight object applied, let’s turn our attention to another kind of light provided by Unity. In the model I am using for this exercise, a doorway sits to the right of the building’s main entrance. The doorway looks as if it could lead to some kind of alley; appropriately spooky, I’d say. I’d like to apply a small light to the eave of the alleyway that the user can dim and lighten at will.

Unity offers a set of built-in lighting objects we can use to create an atmosphere in our scenes. They are a directional light, a spotlight, an area light, and a point light. There are also two options for objects called probes, but we discuss those later. In this exercise we’ve already worked with a directional light to mimic the soft light from our moon. Now, let’s add a point light.

In the Light component attached to the Point Light game object confirm the Light.Type is set to Point. Set the range to a value that you feel is appropriate for the mood you’re creating for your scene. Notice if you hold your mouse over the Range field, your cursor turns into a pointer with two opposite-facing arrows. When the two opposite-facing arrows appear around your cursor, you are allowed to click and hold your left mouse button to drag the field to the value you’d like to set as its parameter. I’ve chosen a float value of 4.45 for my PointLight.Light.Range value. Set the Shadow Type to Soft Shadows and select a warm, yellowish color to simulate the appearance of streetlights or glowing lamps (Figure 7-13).

By now your scene should be looking a lot more interesting than when we started the exercise. By adding three objects to our default scene—a 3D model, a directional light, and a point light—we’ve already created an interesting prototype for an immersive experience (Figure 7-14).

Step 5: Allow the User to Change the Point Light Through the Keyboard

Of course, as we’ve already discussed, the key to a convincingly immersive experience is—that’s right—feedback! Now, we’re going to create a script to empower our user to control the brightness of our doorway light.

Recall that when we introduce interactivity into our scene through an original script, we need a game object in the Scene Hierarchy on which we can attach our script as a component. By convention, we give this otherwise empty game object a name that ends with either “manager” or “controller.” Let’s create an empty game object in our Scene Hierarchy by right-clicking with our mouse. Name the new, empty game object LightController.

Like the script we wrote in our last exercise, this script stores a component of a game object in a variable, manipulates the value of the variable, and attaches the component back to its object. Two things you might not recognize in our Dimmer.cs script are the += and -= operators. These translated into English are “equals itself plus” and “equals itself minus,” respectively. In the code within the Update() function in Dimmer.cs the expressions within the conditional statements (if/else if) can be expressed in words as “set the point light range value equal to itself plus the value of the rate variable” and “set the point light range value equal to itself minus the value of the rate variable.” The value of the rate variable is a property on our Dimmer.cs script component we make public to allow us to adjust it within the Unity Editor.

An Aside on Magic Numbers

Storing known values like 0.08f (where f identifies the value as a data type float) in variables allows us to avoid a bad habit in programming called magic numbers. When developers tell you that you have a magic number in your code they are telling you that you have used a literal value, like 0.08f, instead of a variable. Although our code would still execute if we used the floating-point number 0.08f instead of the variable rate, the practice of avoiding magic numbers pays dividends over time. What if, for example, our Dimmer.cs script was but one component in a complex web of scripts communicating with each other, passing values back and forth? Under such conditions, a request from a project manager to change the rate at which our point light dims would require us to go through all of our code, changing each and every instance of the literal value 0.08f. Using a variable like rate, instead, gives us the freedom in the future, if we choose, to change the value of the point light’s rate of dimming by simply updating the value once at the moment we declare the rate variable in our script.

With the code in our Dimmer.cs script set, we save our changes and return to Unity. After waiting for the code to compile in the Unity Editor, attach the Dimmer.cs script as a component to the LightController game object in the Scene Hierarchy. Once you’ve attached the Dimmer.cs script to the LightController object, you will see two empty fields beneath the Dimmer (Script) in the Inspector. As we did with the cake object in our Interest Calculator exercise, drag and drop the Point Light object from the Scene Hierarchy on to the Dimmer (Script) Target Light field (Figure 7-15). Because we assigned the data type Light to this field in our script, Unity will only allow us to drag and drop a Light object into the Target Light field. Notice, too, that the value of our Rate field is already set to the 0.08 value we defined in our script. This is how we can define default values for public properties we create on objects through our scripts.

Step 6: Play-Test

Once you’ve set the properties on the Dimmer.cs script component attached to our Light_Controller, save the scene and give it a play. When you press the up-arrow key, the point light hanging in our alleyway should increase in brightness by a value of 0.08. Pressing the down-arrow key will diminish the brightness by 0.08. This is the work of the conditional statements we wrote in our Update() block (Figure 7-16).

An Aside on Errors

If you get an error in the Console window of the Unity project, click it to read the details. Double-clicking the error might take you to the line of code in the Dimmer.cs script where the error occurred if it is in fact the result of a coding mistake. If you do encounter an error notification connected to the code, refer back to the code printed in this book. Confirm that everything is entered as is with the same spelling, capitalization, and punctuation. Because VRTK 4 only became available for public use in March 2019 and both Unity and Visual Studio release frequent updates, the error might not be a result of anything you did. Such is the nature of developing in such a bleeding-edge space. Fortunately, as an open source project, VRTK allows users to submit errors through its GitHub page. Forums, too, keep developers abreast of the latest developments to a code base.

Step 7: Practice Moving Lights, Cameras, and Yourself Around the Scene

In Windows, holding the Alt key and the left mouse button in the Scene View window allows you to rotate your view around the scene. Holding the right mouse button allows you to tilt and pan across the scene. If your mouse has a button between its left and right buttons, then scrolling it allows you to zoom in and out of the scene, and holding it down and moving the mouse allows you to move vertically and horizontally in two dimensions within the scene.

Changing your view within a scene has no impact on the placement of your scene’s main camera. Camera objects, like other game objects, can be moved through the buttons on the toolbar or the keyboard shortcuts laid out earlier. If, however, you would like to reposition a scene’s camera to match the view you’ve found in your Scene window, then you can use the ever-handy shortcut Ctrl+Shift+F (in Windows) to align any game object highlighted in your Scene Hierarchy to the view currently displayed in your Scene View window. Other convenient functions to help you maneuver objects around your scene can be found on the GameObject menu (Figure 7-17).

Step 8: Change the Value of the Rate Variable in the Unity Editor

Finally, before we move on to converting our scripted dimmer function into a Unity Axis Action component using the VRTK framework, experiment with changing the value of the Rate variable in the Dimmer.cs script component in the Unity Inspector. Remember, changing the value of a game object property while the Editor is in play mode will not save it to the computer’s memory. If, however, you change the value of Rate while the Editor is stopped and then press Play, you will notice that the increment by which the alleyway point light brightens and dims changes, too. As you can see, defining malleable properties in our scripts with the public keyword allows us to prototype more quickly and efficiently in our editor.

Remaining in the Unity Editor as much as possible is a convenient and quick way to get ideas for our immersive experiences out of our imaginations and into a visual space. It’s one thing to creatively solve the problem of what code to write to execute an action you envision in your scene, but it’s another to write the code in an elegant, efficient way that allows for easy troubleshooting and smooth playback. Scripting a dimming function to respond not just to a touch controller’s trigger input, but to the trigger input of any brand of touch controller that has and is yet to exist, is the fastest, easiest way to lose steam on an immersive project.

In today’s fast-paced world where content ships quickly and can update within moments, getting our experiences out into the world is our goal. Although it’s tempting to curate every nook and cranny of our app to make sure our code doesn’t expose us as frauds, the unseemly secret of interactive applications is that we can’t know how they’ll work until someone interacts with them. To that end, rapid iteration is essential to VR development. The more often we can steamroll our way through the prototype of a vision, the more reps our application will get with real-life users and the more feedback we’ll receive to improve it. It is for this very reason that the VRTK framework can be such a valuable weapon in your creative arsenal. Let’s see how easy VRTK makes it to hook up our touch controller to a function that dims a light in our scene.

Step 1: Add a Second Point Light

Remember how earlier I said you could have optional practice with our broken building scene by placing a point light in one of its windows? Well, I lied. It’s not optional. If you’ve already done it, excellent! If you haven’t, please do it now. Refer to the steps we followed when placing the alley light in our scene for reference.

For your convenience, here are the transform settings for the bedroom pointlight I placed in my scene (Figure 7-18):

• Transform.position: -5.43, 4.88, 4.21

• Rotation and scale set to default.

Step 2: Write a Second C# Script for the New Axis Action

Now that you have a point light set up inside one of the building’s windows, let’s wire it up to a trigger action on our right touch controller. To accomplish this, we must use a Unity Axis Action component. Why a Unity Axis Action component? Well, buttons have two inherent states that developers use to communicate responses to yes/no, true/false questions. Triggers, as an axis action, allow us, as developers, to measure a user’s input on a spectrum, for example, between 0 and 1.

After you’ve set up a second point light in your scene, name it RoomLight. Create a new empty game object in the Scene Hierarchy and name it LightController2. Because this game object is going to be a controller in our scene, let’s add a script to it. The script connected to the controller as a component will provide the controller instructions for that which it will control. Let’s create a new C# script in our Scripts folder in our Project window and name it, creatively, Dimmer2. Once you’ve confirmed you’ve spelled and capitalized the script’s title as per your intention, double-click it to open it in Visual Studio.

Like the first statement we typed, the class definition appears by default in the Unity script. This is the name we give our script on first creating it in our Project window. Recall that a class is a blueprint for an object. We won’t be creating a Dimmer2 object in our scene. We still, however, need the script to exist for us to access the functions (verbs, the doers) in our script. It exists to hold properties and functions for us like a container.

After a moment your script will recompile. Do you see three red squiggly lines in your code as shown in Figure 7-19? If so, good! In Visual Studio, red squiggly lines tell us we have an error in our code.

At the bottom left of the Visual Studio window is a menu called Error List (Figure 7-20). If you click that menu, a window appears that lists the errors the compiler found in our code.

Now, let’s uncomment our using statement by removing the two forward slashes. The red squiggly lines disappear and our Error List shows no errors. Further, in our code the words MonoBehaviour, Light, and Debug have returned to the color green (Figure 7-21). We’ve just learned that these three words are classes in the UnityEngine namespace. We did not have to create object instances of these classes in our script because the statement using UnityEngine does that for us.

Here, after we’ve manipulated the desired property, we attach it back to the game object from which we originally took it. In our analogy about the mechanic, this code equates to the moment we return the car’s carburetor after fixing it, for example, back to the car’s engine.

Finally, to complete our script, we add two closing curly braces } } to mark the end of our DimLight function code block and Dimmer2 class (Figure 7-22). That’s it! Let’s save our script and return to Unity.

Step 3: Add a Unity Axis Action Component to a Game Object

After we allow Unity to compile the changes we’ve made to our Dimmer2 script, we’re ready to attach it as a component to our LightController2 game object. Select the controller object in the Scene Hierarchy, and in the Inspector add the Dimmer2 script as a component. Remember, you can either click Add Component and search for Dimmer2 or drag and drop the script from the Project window onto the controller object.

With our Dimmer2 script attached to our LightController2 object we can add the VRTK Unity Axis 1D Action component. With the LightController2 game object selected in the Scene Hierarchy, click Add Component in the Inspector and search for the Unity Axis 1D Action script (Figure 7-23). Notice that it has the same four events as the Unity Button Action component. Uniquely, though, the Axis 1D Action component has a field called Axis Name. This will be the axis to which our Value Changed event will listen. What’s the one-dimensional axis to which we’re interested in listening? It’s the right trigger button on the VR touch controller. How do we do that?

Step 4: Identify the Controller Button to Connect to the Unity Axis Action Through the Input Menu

Recall that when we imported VRTK into our Unity project, a window appeared asking us if we’d like to import the VRTK Input Mappings. If you didn’t click OK, it’s no problem. Simply navigate to Main Menu ➤ Window ➤ VRTK ➤ Manage Input Mappings (Figure 7-24).

When we open our Unity input settings (Main Menu ➤ Edit ➤ Project Settings ➤ Input) we see a drop-down list of all the axes to which Unity has mapped input.

Your list of inputs might look different than mine, but what’s relevant to this exercise is the presence of the VRTK Axis mappings (Figure 7-25). If they do not appear in your Input list repeat the import process described in the previous paragraph.

Let’s expand the heading VRTK_Axis10_RightTrigger. You’ll find a collection of fields defining the right trigger axis. Copy the value of the Name property to your computer’s clipboard (Figure 7-26). Close the Input menu and return to your project’s Scene Hierarchy.

With your LightController2 object selected, turn your attention to the Inspector. In the Axis Name field of the Unity Axis 1D Action component, paste the name of the VRTK axis you just copied (Figure 7-27).

Congratulations! You just mapped your controller game object and its attached Dimmer2 script to your touch controller’s right trigger button. Now, the float variable rate we defined as a parameter on our DimLight function in our Dimmer2 script will read the value communicated by the pressure placed on your touch controller’s right trigger. With our input into our Unity Axis 1D Action established, let’s define our output.

Step 5: Connect the Chain of Events

Because our right trigger event will control the intensity of our room light, we want to call our DimLight function whenever the user’s pressure on their trigger changes. To accomplish this, we’ll connect our DimLight function as the subscriber, or listener, on our value changed event of the Axis Action component (the publisher). Click the + button on the Unity Axis 1D Action component Value Changed (Single) event. In the field that presently reads None (Object), drag and drop the Dimmer2 script component from the same LightController2 object (Figure 7-28).

From the Value Changed function pull-down menu select Dimmer2 ➤ (Dynamic Float) DimLight (Figure 7-29). Although there is a DimLight(float) function listed beneath the Static Parameter heading, we want to use the DimLight function beneath the heading Dynamic Float. Confirm that the Dimmer2 script component in the LightController2’s Inspector window has a Light object from your scene identified in its Target Light field. In my scene, I placed a point light in the top left window of the building (in the negative x axis direction). I named the light RoomLight in my Scene Hierarchy and dropped it into my Dimmer2’s Target Light field. The Light object you select should be the one you created after placing the alleyway light in the scene. Before we move on from the controller object, confirm that you’ve properly named the Axis Name on the Axis 1D Action component and connected the Dimmer2 script component to the Value Changed event.

Step 4: Add a Virtual Camera and a TrackedAlias Prefab to the Scene

Back in the Scene Hierarchy, let’s add a UnityXRCameraRig object (if you plan to play-test with an external HMD) or a SimulatedCameraRig (if you plan to play-test inside the Unity Editor). Because we are listening to a trigger event from our VR touch controller, let’s also drag and drop a TrackedAlias game object into our hierarchy. Increase the size of the Element property on the TrackedAlias’s Tracked Alias Facade component and drag and drop the UnityXRCameraRig into the field Element 0 to complete the synchronization between the TrackedAlias game object and the UnityXRCameraRig (Figure 7-30).

Finally, in the Scene Hierarchy, also confirm that only one camera game object is active. If your scene includes a default Main Camera game object, then deactivate it in the Inspector. Further, confirm the two point light objects you’ve added in this chapter are active, too, and connected to their respective LightControllers.

Step 5: Play-Test

That’s it! Save your scene and give it a whirl!

If you play-tested your scene, did you find that when you pulled the right trigger button on your touch controller the point light you placed in one of the building’s windows dimmed on and off? If not, check the Unity console for any error messages. If the Axis 1D Action did work, then your Console window should show the string we passed into the Debug.Log() function in our Dimmer2 script for every time you pulled the right trigger on your touch controller (Figure 7-31). You now have the power to create VR apps that can read the sensitivity of a user’s input through touch.

As Annie, you just heard back from Elgin that the theater’s board of directors were impressed with his immersive presentation. With a clear understanding of Elgin’s vision they have no problem cutting the check required to fund the production. Congratulations, Annie! Because of you the show can go on!

The application of the tools you’ve learned in this exercise is, of course, not limited to dimming lights. With the knowledge you now have about VRTK, Unity, and C# scripting, you can create an assortment of actions influenced by the sensitivity of a user’s input. For example, you could create an interior design tool to help a user visualize what different shades of a certain color would look like in a bedroom. You could prototype an installation and test parameters of different ambient features in a space. The possibilities are defined by your imagination.