Partnership and collaboration

Despite how much I advocate for designers to know code, that doesn’t mean they should do all the programming. Developers are really good at what they do. Even if you can code, are you up-to-date on best practices in terms of scalability, security, and cross-platform development? And even just with coding, programmers are much more efficient at it, because it’s what they do all the time. Is diving into code really the best use of your time? Or should you be focusing your skills where your strengths are?

From a business standpoint, let’s say you make $100/hour designing or $800 per day. Think about what you could accomplish in that time and how many projects you could work on. Then, consider how long it would take you to program an experience. If it takes you five days to do what you could hire a programmer to do for less than $4,000, then you are missing out on producing income from your design work. What I have come to realize in my time as a freelancer and business owner is that you make more money doing the things you are best at. You can actually make more profit hiring others to help in the areas where you aren’t as efficient, even though you pay them for their time. Early on in my career, I thought “Why would I pay someone else to do it, if I can do it myself?” Now I realize that the answer is, “Because they are better at it than you.”

There is also another option. Being conversational in a specific programming language will help you become a better designer and will help you explore the potential of new technology to discover what is possible. Maybe you do want to try to create at least a test project on your own. Just as there are businesses and services that tailor to designers for design-first web development, there are also similar services for 3D and XR design. What is great about this kind of option is that you may not be responsible for writing full lines of code yourself, but rather will be editing existing code to customize an experience to match your design direction. Of course, you have to first learn enough about the code to start to understand it and edit it.

3D web frameworks

As you start to explore practicing code, there are a number of web-based frameworks that will allow you to explore with 3D objects and immersive experiences and their underlying code, often side by side. With this split-screen view, you can see the design and the code at the same time. This layout allows you to explore the relationship between the elements and the programing that is powering them.

In order for 3D and to work directly in a web browser, the code must follow web standards. The current standard is WebGL, which is a JavaScript API for rendering interactive 2D and 3D graphics without the need for installing any plug-ins. This has allowed stand-alone XR experiences and even 3D interactions to exist on the web. Some also use WebXR, which allows these experiences to connect to a VR or AR device for the user to interact with. These standards allow someone to visit a website and start interacting with the experience in seconds, without downloading an app or anything else to their device.

Although this implementation is exciting, the upcoming standard to watch is the implementation of the WebGPU standard, which has positioned itself to be the future of JavaScript to provide “modern 3D graphics and computation capabilities” according to the massive collaboration of developers and engineers in the W3C GPU for the Web Community Group. This group has members from such companies as Google, Apple, Microsoft, and Mozilla, to just name a few. It is something to keep an eye on as you explore creating for XR. As WebGPU starts to become the new web standard for all things 3D, it will certainly bring great web experiences along with it.

The key components for these web-based experiences are a combination of HTML and JavaScript. Because 3D integration on the web still faces a number of challenges, there are some different frameworks that have been created to help. These leverage, and often piggy-back on, what already exists to create a web-based experience.

Here are a few to get you started:

A-FRAME (www.aframe.io) A web framework for building 3D, AR, and VR experiences. It is built on top of HTML, making it easy to use, and it doesn’t require anything additional to get started. This is my recommended place to start as you can find a lot of great documentation to guide you. The documentation also is straightforward, making it easier to read through and start to understand the code and the relationships of objects. It works as a structure with three.js.

THREE.JS (www.threejs.org) A framework that creates and displays animated 3D computer graphics in a web browser using WebGL.

AMAZON SUMERIAN (aws.amazon.com/sumerian) A service that enables 3D, AR, and VR applications within the browser experience using WebGL and WebXR.

BABYLON.JS (www.babylonjs.com) A real-time 3D engine using a JavaScript library for displaying 3D graphics in a web browser using HTML. It uses WebGL, with some WebGPU support as well, but only for certain browsers that have this capability enabled. (But, it is a start!)

PLAYCANVAS (www.playcanvas.com) An open source HTML-based game engine built on WebGL for building VR and AR.

Powered by these frameworks, some software companies are offering integrated development platforms. The framework provides the foundation and the capability, but it is enhanced by tools to create AR/VR content, collaborate with others working on the project, and publish the content right to the web for instant access by the user. Three of these developer platforms to check out are MetaVRse 3D/XR engine, Amazon Sumerian, and 8th Wall (FIGURE 13.2). Of course, this convenience comes at a price, and launching this way will require a budget. However, it will still be less than designing and developing a full independent app—and you can do it with minimal to no code.

When it comes to code, you don’t need to know all the answers—just how to find them. That may mean that you work in partnership with an individual or a team who can. It is important to be familiar with the programming languages that your experiences are leveraging. Read lines of code; try to understand what is happening and how. Even explore debugging as a way to problem-solve and deepen your understanding of how the code affects the experience. But be sure to acknowledge your strengths and use those to help advance every project you work on first.

Agile versus waterfall

This agile workflow is the opposite of the traditional waterfall workflow, which, just as the name suggests, works from the top down. Each stage is completed before the next one begins. First, the research stage is completed. Then the design stage is completed, then the developmental stage. Once those three stages are complete, testing is done. The problem with this philosophy is that it doesn’t promote team collaboration, nor does it build in testing all along the way to identify problems. They will all come to the surface at the end of project, which can cause significant delays in delivery times. More significantly, you may identify issues in the design that may not even be possible in the final product. If you are working for a client, who has approved the design and the overall functionality of the project, then you could be in trouble when you later have to turn around and explain that what everyone agreed to isn’t possible. All of these outcomes are best to avoid.

The agile approach breaks a project into smaller segments that allow for each part of the team to work together and develop together. Research is done, and that influences the design. Some programming is done, and the testing is done for just that one small segment. Then the team reassesses before they go onto the next segment of work (FIGURE 13.3). This allows for pivots and adjustments to any and all parts of the process.

The concept of an agile workflow is something that has been adopted by many teams working with technology—specifically software development. Agile workflows have been especially embraced by teams working on website, mobile applications, and immersive and interactive experiences. The overall concept was first pitched as a manifesto by a number of software engineers in 2001 (agilemanifesto.org). As a group, they collaboratively list their key values as:

Working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.¹

¹ Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for agile software development. Agile Manifesto. agilemanifesto.org

As part of this manifesto they also list 12 principles. They vary in topic, but include key highlights such as: early and continuous delivery, embracing change, team work, including developers in decision-making (along with business partners and other key members of the project), and the concept of frequent delivery. However, the goal is the creation of a minimum viable product early in the process, which allows you to get a working product, even if basic, to the client to test, and build upon that.

This manifesto became the ripple that has spread into team workflows in interactive studios and agencies. People have mostly embraced this concept, as they have recognized the value it adds; it is the mindset that still holds the most value in the industry today. Working based on a team mentality has even changed the physical layouts of offices. Traditionally, there have been workspaces where all the designers sit in one group, all the developers sit in another group, and all the business and marketing teams are spaced out even more separately. As part of embracing an agile workflow, project teams sit together, with one person from each of those departments working in a shared space to collaborate through every step of the project.

This concept can work no matter the size of the project. If you are working independently, there are still many important reasons to adapt an agile workflow. The most important element of being agile is to continuously test how your designs will look on the device or platform where it will be viewed. This will allow for constant refinement of the experience through testing and analysis for each part of the design process.

Scrum

The agile methodology has been adapted in many different ways for different projects. The most common iteration that has been used effectively for designers is Scrum. This is a specific agile method that creates short design sprints that allow for quick exploration into one area. These sprints have to complete the full circle for them to be effective (FIGURE 13.4).

Let’s say, for example, that you wanted to create an AR experience that allows the user to walk around a neighborhood. As they reach different places they would be able to scan a house, or any other landmark, and then see historical photos of that exact space overlaid on the live image. The concept is to learn more about the history and narratives from each location on the street. How would you start this process using the Scrum method? Well, the idea is to use Scrum to problem-solve any challenges or conflicts, as well as see how each person on the team will contribute to the project. Some initial problems that might need to be solved in their own design sprints are:

• Determine what technology would be best for this kind of experience. Does it need to be downloaded as a stand-alone app, or can it be a web-based experience?

• Test what objects work best to scan as an image target. Signs, buildings, or something else?

• Is image recognition the best option, or would location services be more accurate?

• What are the privacy concerns with the user sharing their location with the experience?

• What kind of content will be displayed in AR? Videos, audio, photos, or 3D models?

• How will the experience change if people visit during different times of day?

• What is the traffic like on this street? Is it safe for people to be walking around this street distracted by their phones?

To start to find some of these answers out, it might be helpful to do some smaller sprints. Instead of trying to find the answers for the whole street at once, you could just focus on the experience for one house. Collectively as a team, you could determine what content will be displayed for that one house. Then based on that, and by visiting that exact location, you could determine the best spot for a person to stand to see the changes that have happened from the past to the present. From that location, you could determine what might work to use for image recognition. Is the house or landmark easily visible? Would there be room for a sign? What would make it easily seen so that the user would stop and know they could engage with the AR experience if they were just walking down the street?

Knowing what is possible with the technology at this stage is helpful, which is why it is great to have a programmer weigh in even at this ideation stage. Because these are sprints, you don’t have to be so worried about the details just yet. The first rounds can be used to test different options to see what will produce the most reliable results. Let’s say for the first round it is determined to use the house as the image target. The idea is that the user would hold their phone up toward the house, and the camera would recognize the house as an image target. This would activate an augmented menu to appear overlaid on the image of the house to explain about the various parts of the house. The user could select one area and have options: Choose to see what it looks like inside, watch a video, or see historic photos.

Okay. Now create that experience, but keep it small; just choose one room of the house. Design how the user will know what room is selected and what menu options are available. Then have this concept developed so it can be tested. Go to the physical space to see if the image target works and what it is like to be the user interacting with the proposed design in that way.

1. Take notes, and talk about it.

2. Decide what works and what doesn’t.

3. Keep what works, and throw out what doesn’t.

4. Come up with a new experience.

That completes one sprint. The next one will take what was learned in the first sprint and improve it. Depending on how well the first sprint went, you could move on to another room in the house, or you could stay in the same location with the same interaction, but built in a different way. You will be able to determine this based on how the first experience went. With each sprint, each person on the team should be able to assist in problem-solving, as well as making adjustments in a way that will continuously improve the experience for the user.

If you are working on a concept independently, than this is a great way to quickly test and rule out technology.

Qualitative research

This part of the process is when you become the student and the users become the teachers. It isn’t time to question each and every design decision made, but rather to gather information based on how the users interact with the experience. This can be done through a number of usability tests:

• Observation of use in a controlled environment

• Observation of use in a natural location

• Observation of use in a virtual location

• Interview

Before diving into these different approaches, let’s first consider the act of observation.

Observation based on location

You can learn a lot from just watching. There are different ways that observation research can be conducted, ranging from in-person to virtual sessions. For example, some observation could be done in a controlled environment. Having a user come to a specific location to do the testing does make the test easier to conduct. Depending on the type of experience, however, it may not be a true representation of how the user will experience your design in reality. If you are testing a VR experience, this option is ideal because you will need to make sure the user has access to the right equipment and software, and you will need to be in a place where you can observe both the user and the view from inside their headset (FIGURE 13.6).

This option may be less ideal for AR, however, because you want to test the experience in a variety of locations. In this case, doing observations from a more natural location, selected by the user, will provide more valuable feedback. Be sure to do testing both inside and outside, at different times of the day, with different lighting conditions, and with various Wi-Fi and cellular data connection strengths. These can help identify issues that may arise based on the physical environment. Some issues will have less to do with your specific design, but will provide insight on how the experience will change based on environmental factors.

You could also perform these observations in a virtual setting. These can be conducted via a video conference platform such as Zoom or Google Meet. However, this could be limiting as you will be able to observe from only the singular vantage point of the camera aimed at the user. If you are able, have the user share their device screen, perhaps by plugging in their phone to a computer to share what they are seeing. Having this information, in addition to be being able to see their face and body language, makes this a viable option. Using this virtual approach may also increase your reach to include users who wouldn’t be able to meet in person and will help diversify your results. This option also allows for easier video recording—no need of additional equipment. It might be helpful to record your observation for later review, and with this virtual option, you could do so easily (with the permission of your user).

Each option of observation has pros and cons that will need to be weighed based on the product you are testing. The best option is really to do a bit of each observation so that you can collectively observe users in different spaces and in different ways to best collect data that can be used to improve the experience.

It is important to note that someone who knows they are being observed will make more attempts to have a successful experience than someone who is testing it on their own. Each user will strive to accomplish their goal or perceived goal of the experience, but users who know they are being observed will spend more time than those who are testing the experience on their own accord.

Quantitative research

To support your observations and interviews, quantitative research relies on numeric and mathematical data that can then be analyzed. Because this data is more fact-based and not colored by an observer’s opinion, it can provide helpful information to suggest how usability can be improved.

Of course, although this data can provide insight to where users are looking, success and failure rates, and other statistical information, it can’t tell you what is wrong with the product and what can be done to improve it. This is why the two kinds of research work well together to provide rounded feedback. There are different ways to conduct this type of research for XR, including:

• Surveys (post experience or overlaid within the experience)

• Eye tracking

• Physiological response (motion and possible discomfort)

• App-generated reports

Eye tracking

As the technology for observing our gaze and focus improves for XR, it can provide great insight to where the user is looking during an experience (FIGURE 13.7). This concept has been used for websites to determine where on the screen people spend the most time looking. The difference here is that the user could already be wearing a device on their eyes that might be able to track their eye flow. This data could provide invaluable feedback to help improve the hierarchy and flow of the design within each of your scenes.

Layers of privacy

Imagine walking into a restaurant: Either you are handed a menu in digital form or one becomes available through your smartglasses. What makes this menu even more special is that it is completely tailored to you. It knows your dietary restrictions and taste preferences, plus it offers some suggestions or options for you. For someone who is gluten-free, for example, having a customized menu to just show which menu items are safe to eat is a luxury. Currently, determining a restaurant’s gluten-free choices requires asking for a special menu, going through the restaurant’s binder with each dish’s allergens listed, or even having the waitstaff relay questions to the chef or bring bottles of sauces to the table for you to check the ingredients—if such options are even available.

Having a personalized and customized menu option for each person would be impossible for a restaurant to do right now. But when each person who enters is wearing a device that has all their preferences saved, then this becomes a very real possibility. While this may be very helpful to some people, it might feel intrusive and undesirable to others. This is where choice comes in.

A user may value the privacy of different personal data at different layers. While it may not be considered private to share a food allergy, it would be to share something like a Social Security number. Privacy concerns when sharing your location with a computer vary widely based on the user. Some users would protect this highly, while others may decide the convenience outweighs the sharing of data. As you design a product, it is important to understand that each user has different layers of privacy that they want to protect. The outer layers comprise less personal information that many users are more willing to share, but as you go deeper, data will be, and should be, protected to a greater degree.

As you engage in a mobile AR experience, before the camera launches, a required message pops up asking for permission to use the camera to launch the AR experience. There will also be permission required to access the camera roll, share a location, and access the microphone on the smartphone (FIGURE 13.9). All of these are mandated by the phone manufacturer, and apps will not be approved to be released if they don’t follow these privacy protocols. For some of these options, there are different levels of privacy offered. In terms of sharing their location, for example, the user may have the choice to share it always, never, or only when the app is in use. As you design an experience, consider ways to place the control into the user’s hands. Let them decide how their data is used, as it is really their choice to make.

Users should be in charge of their own data. They should clearly know:

• What information they are sharing

• How it is being used

• What they get out of that as a result

That means that shared data can be used only for the purposes that a user has agreed to. Data is the new currency, and, as such, it should be protected accordingly. Luckily, actions are in motion already to make sure that data is being guarded in a way that will provide protection to the user.

AR Cloud

One of the ways that people are being proactive to ensure that this technology does better is by creating a set of standards in how to connect the physical and digital worlds to benefit the greater good. The largest initiative in this space is called the AR Cloud from the Open AR Cloud Association (OARC; openarcloud.org). The organization’s mission is to create standards, guidelines, and tools for how data stored in the AR cloud can be protected. Their main focus is to be respectful of the user’s rights to keep their different layers of information private. To do so, they are working to collaborate with businesses and XR professionals to create a cloud technology that would house and protect this information. Instead of data being held by large conglomerate business, who might share that data among partners and stakeholders, the data is held by this organization whose mission is to protect it.

The reason cloud data has emerged is that there isn’t enough space on our devices to hold all our digital information. So, we need to offload data to other places where we can access it as we need. This becomes very important for AR technology, especially as we look toward wearable AR glasses. With this wearable technology, physical storage space, weight, and even heat produced by processing are all big concerns. Having a place where information can be stored and accessed, as needed, provides a great opportunity (FIGURE 13.10). Many of these data clouds have emerged, but they are often created by the businesses that may also have a motivation to learn from their users. OARC’s idea of a separate space is a step toward moving in the right direction.

We are used to our smartphones knowing our shopping preferences and knowing our location, but the future of this XR tech means that our devices will be able to track everything we see and, specifically, what we look at. With cameras always in use and available, it becomes even more of a priority that we protect the privacy of the data collected. The cloud is promising a place that can hold digital clones of the spaces we interact with. To make sure users are okay with that, OARC is also creating standards to protect that digital information. OARC is looking to make trust the new commodity by placing information in a safe space and into the hands of the user.

Ethics

Because XR provides you the opportunity for users to have superpowers, what powers are you going to allow? As you design experiences, you have to determine what control to provide users based on what you feel comfortable with and what you don’t. With more and more features enhancing this space every day, there are many to choose from. While AR and VR are a component, they often work with other technologies such as machine learning, artificial intelligence, and facial recognition—to just name a few. Within each of these there are ethical considerations to consider.

With techniques like photogrammetry and 3D model scanning becoming faster, better, and more mainstream, it brings up the issue of what is real in our realities. If something looks like a photograph, how do we know if it has been edited? When is it okay to fictionalize an experience, and when is it not okay?

These lines have been drawn more clearly in the photography world, where the field of photojournalism emerged with a strong ethical commitment to document the world as they see it without alteration. But, just as in that field, instances have arisen where people have unethically edited a photograph for a wide variety of reasons.

When you create a digital scan of a physical space, there is editing that has to happen. Merging the images all together into one shape requires manipulation to make it work and to minimize any seams that occur. In 360-degree video, often the camera has to be edited out of the space. If that element has been removed, what else has been? What has been added or altered?

These are important conversations that need to take place throughout the project—and especially as you prepare to share your product with the world. The narrative you wrap around an experience will alter how people understand it. If you share a VR experience that promises to bring people to a place from history and experience it just as someone would have had years ago, then you ethically have an obligation to call out any editing that you have done that may have altered the original images. Even if it was only done for the purposes of the experience or to format it for the device, it is important to be transparent with the users. This narrative should also make it clear when you are fictionalizing an experience. People may make assumptions about digital content not being real, but if images in an experience are based on a real event, that is something that should be shared to clarify. As you present your experience with users, do the ethically conscious thing and also share what lens they should view it with: fiction, non-fiction, or reenactment.

Facial recognition

As we have discussed, cameras have evolved to do more than just capture a moment in time; they can also capture spatial information and data. This has led to the creation of what is called facial recognition (FIGURE 13.11). This technology can very useful in instances like unlocking a smartphone so that only a specific user can have access.

However, this technology has revealed a larger issue of computer bias. Based on the information released in a federal study The Washington Post reported that,

Asian and African American people were up to 100 times more likely to be misidentified than white men, depending on the particular algorithm and type of search. Native Americans had the highest false-positive rate of all ethnicities, according to the study, which found that systems varied widely in their accuracy.³

³ Harwell, D. (2019, December 19). Federal study confirms racial bias of many facial recognition systems, casts doubt on their expanding use. The Washington Post. www.washingtonpost.com/technology/2019/12/19/federal-study-confirms-racial-bias-many-facial-recognition-systems-casts-doubt-their-expanding-use

The study also showed that women were more likely to be misidentified than men. This is important information to pay attention to as many XR experiences use facial scanning and even identification as one of their core functionalities.

The accuracy of a facial scan may not be as significant when the purpose is just to apply digital content onto it, such as a makeup app or an app adding animated stickers. However, when the app is being used by law enforcement to identify suspects, accuracy becomes essential.

The Algorithmic Justice League is an organization led by Joy Buolamwini, a computer scientist and digital activist who has a mission to challenge companies from using this technology in decision-making software. As Buolamwini stated in her TEDxBeaconStreet talk titled “How I’m Fighting Bias in Algorithms,”

Computer vision uses machine-learning techniques to do facial recognition. So how this works is, you create a training set with examples of faces. This is a face. This is a face. This is not a face. And over time, you can teach a computer how to recognize other faces. However, if the training sets aren’t really that diverse, any face that deviates too much from the established norm will be harder to detect.⁴

⁴ Buolamwini, J. (2016, November). How I’m fighting bias in algorithms [Video]. TED. www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms

The Algorithmic Justice League (www.ajl.org) is working to create equitable technology that is for everyone.

When you are working with this kind of technology, there cannot be a buffer for accuracy. Either it has to work 100% accurately, or it does not work at all. Imagine if you opened your favorite map application and entered a destination. You expect the map to bring you to the correct place. If the app cannot show you the correct directions, then it doesn’t work. You’d deem it unusable. Why then would we accept anything less than that for other applications, such as facial recognition? If it is not 100% accurate, then it doesn’t work and should not be used.