Competence

Competence is the ability of the product to get the job done. Does it improve the experience or address the user’s needs satisfactorily? A good-looking product or one with many features that do not fulfill user needs is not competent. Strive for a product that provides meaningful value that is easy to recognize.² Google Search is an example of a competent product. It generally offers satisfactory results for your questions.

Reliability

Reliability indicates how consistently your product delivers on its abilities.³ A reliable product provides a consistent, predictable experience that is communicated clearly. A product that performs exceptionally well one time and breaks down the next time is not reliable. Apple’s iPhone is an example of a reliable product. It might not carry all the features its competitors have, but you can reasonably trust the ones it does have.

Predictability

A predictable interface is necessary, especially when the stakes are high. If the user comes to your product to perform critical, time-sensitive tasks, like quickly updating a spreadsheet before a client presentation, don’t include anything in your UI that puts habituation at risk.⁴ A probabilistic AI-based solution that can break the user’s habit is not ideal in such cases. However, suppose you think that users have open-ended goals like exploration. In that case, you can consider a dynamic AI-based solution that sometimes breaks user habits, for example, selecting a movie from dynamic AI-based suggestions.

Benevolence

Benevolence is the belief that the trusted party wants to do good for the user.⁵ Be honest and upfront about the value your users and your product will get out of the relationship. Patagonia is a clothing brand that makes jackets that does a great job with benevolence. While their products can be expensive, they encourage people to reuse and repair their Patagonia clothes, giving a percentage of their sales for environmental causes. The company is upfront about its value to the customer.

Explainability

If we don’t understand how AI systems work, we can’t really trust them or predict the circumstances under which they will make errors.⁷ Explainability means ensuring that users of your AI system understand how it works and how well it works. Your explanations allow product creators to set the right expectations and users to calibrate their trust in the AI’s recommendations. While providing detailed explanations can be very complicated, we need to optimize our explanations for user understanding and clarity.

Control

Decision-Makers

Decision-makers are people who use the AI system to make decisions. A decision-maker can be a bank officer deciding whether to sanction a loan or a radiologist diagnosing an ailment based on an AI’s recommendation. Decision-makers need explanations to build trust and confidence in the AI’s recommendations. They need to understand how the system works and sometimes require insights to improve their future decisions. Most decision-makers need simplified descriptions and have a low tolerance for complex explanations.

Affected Users

These are people impacted by recommendations that your AI system makes, like a loan applicant or a patient. Sometimes the decision-maker and the affected user are the same people. For example, giving Netflix your preferences is a decision, and getting movie recommendations is the effect. Affected users seek explanations that can help them understand if they were treated fairly and what factors could be changed to get a different result.¹⁴ They have a low tolerance for complex explanations, and outcomes need to be communicated clearly and directly.

Regulators

These are people who check your AI system. Regulators can be within the organization in the form of internal auditing committees or external in the form of government agencies. Regulators can enforce policies like the European Union’s General Data Protection Regulation (GDPR) and may require AI creators to provide explanations for decisions. Regulators need explanations that enable them to ensure that decisions are made in a fair and safe manner. They may sometimes use explanations to investigate a problem. Explanations for regulators can show the overall process, the training data used, and the level of confidence in the algorithm. These stakeholders may or may not have a high tolerance for complex explanations.

Internal Stakeholders

They are people who build the AI system, like ML engineers, product managers, designers, data scientists, and developers. Internal stakeholders need explanations to check if the system is working as expected, diagnose problems, and improve it using feedback. They generally have a high tolerance for complexity and need detailed explanations of the system’s inner workings.

A model diagram of complexity of the explanation has two queues, the simple and the complex. The commodities in simple are decision-makers and affected users, while complex has internal stakeholders and regulators.

Make Clear What the System Can Do¹⁶

Make sure your users understand the capabilities of the system. Users need to be aware of the full breadth of functionality of your feature. Providing contextual information on how the AI system works and interacts with data can help build and reinforce user trust. For example, if there is a search functionality within a grocery application, make sure all search possibilities are enumerated, like “search for fruits, vegetables, and groceries.” Clarify how input influences the results.

Try not to be opaque about how the system makes decisions, especially for high-stakes situations like loan sanctioning or predicting diseases. Unless your application can handle those, stay away from open-ended interaction patterns without proper guidance. For example, an AI running assistant asking questions like “Ask me anything?” or “What would you like to do today?” sets the wrong expectations about the system’s capabilities.

Two user interface images for Eat Well recipes. The selections in the first are: search for recipes; search by cuisine; by mealtime; and recommended recipes for you. The second has Chef A I. The first image is to aim for, and the second one is to avoid.

Make Clear How Well the System Does Its Job

AI systems are probabilistic. Help users understand when your AI system performs well and when and how often it makes mistakes. Set expectations of performance out of the box and clarify mistakes that will happen. In some cases, it is better to use uncertain language like “We think you’ll like this book” or indicate a level of confidence. When it is not feasible to provide this information directly, consider providing a help context that is easy to access or a universal help command like “Say help for more details.”

Don’t leave out setting expectations of performance and updating them when things change. For example, an increase in traffic can lead to higher wait times for a taxi service. AI systems operate with uncertainty. If your users expect deterministic behavior from a probabilistic system, their experience will be degraded.¹⁷

Two interface images for Flora. Two of the dialogue panes in each image read, "Click a picture of a plant to identify it." It works best with plants native to India that are italicized, and Hi! I'm Flora. Your A I power plant guide. The first image is to aim for, and the second one is to avoid.

Set Expectations for Adaptation

AI systems change over time based on feedback. Most AI systems adapt to changing environments to optimize their output or personalize to users. AI systems learn from their environments; sometimes, they keep track of whether or not an output was useful and adapt accordingly. For example, Netflix adjusts its recommendations based on your interactions with the service like movies watched, for how long, user ratings, etc. Indicate whether the product adapts and learns over time. Clarify how input influences results. Don’t forget to communicate changes in accuracy as conditions change. For example, a navigation system predicting the time to arrival can adjust depending on changes in traffic or weather conditions.

Two user interface images for EAT WELL recipes. The following selections are recommended for you: the information selection circle, and Chef A's suggestions. The first one is to aim for, and the second one is to avoid.

Plan for Calibrating Trust

AI systems will not perform perfectly all the time. Users shouldn’t implicitly trust your AI system in all circumstances but rather calibrate their trust correctly.¹⁸ People may sometimes overtrust your system for something it can’t do, or they may distrust the AI’s output. Either of these cases causes a mismatch in expectations and can lead to the failure of the product. You can help users calibrate their trust in the AI by explaining what it can and can’t do or displaying its output confidence. This uncertainty can lead to some level of distrust initially. For example, telling users that the prediction may be wrong can lead to them trusting the system less, but over time as the AI improves, they can start relying on your product more with the right level of trust.

The key to success in these human-AI partnerships is calibrating trust on a case-by-case basis, requiring the person to know when to trust the AI prediction and when to use their own judgment in order to improve decision outcomes in cases where the model is likely to perform poorly.¹⁹ It takes time to calibrate trust with users. Plan for trust calibration throughout the user journey over a long time. AI changes and adapts over time, and so should the user’s relationship with the product. For example, a music recommendation service may not be very accurate for new users. But over time, it improves by learning user preferences. Explaining that the service learns your preferences over time and will get better at recommendations can help users calibrate their trust at different stages of using the product. Planning for calibrating trust can help you in designing better explanations and build more effective human-in-loop workflows.

Your radio has an interface that has selections from top to bottom: Dismiss, Love, add to playlist, rewind, play, pause, next, and buttons. And foremost, the bottom part is the slider for where you want to start sound. On the right side is a dialogue that reads, "A personal playlist based on your listening habits." It improves as you listen more.

Be Transparent

It is difficult to trust those who appear to be hiding something. Transparency means operating in ways that make it easy for others to understand the actions of a system. It implies acting out in the open. There are legitimate reasons for organizations to have trade secrets or proprietary information. But when it comes to pertinent information of customers, privacy, data use, bias, other stakeholders, or the efficacy of the algorithm, transparency is central to earning trust. This is especially true for AI operating in high-stakes environments like medicine or driverless cars. A lack of transparency increases the risk and magnitude of harm when users do not understand the systems they are using or there is a failure to fix faults and improve systems following accidents.²⁰

Many AI systems will have to comply with privacy laws that require transparency about the collection, use, and storage of data and mandate that consumers have appropriate controls so that they can choose how their data is used.²¹ Product teams need to ensure that users are made aware of any data that is collected, tracked, or monitored and that it’s easy for them to find out how the data is collected, whether via sensors, user-entered data, or other sources.²² In necessary cases, use non–black box models so intermediate steps are interpretable and outcomes are clear, providing transparency to the process.²³

The level of transparency will be different for different stakeholders. Your explanations can help users understand why the AI made certain predictions through transparency of data, its use, and the underlying algorithm. Good explanations can help people understand these systems better and establish the right level of trust.

A h t t p s column slash slash jobfinder dot a i webpage. The header is an attached case logo, Jobfinder. The selections are: Search for job opportunities; Opportunities you might like, based on your resume, profile, current location, and industry; and to know more, and four selections of jobs based in Bangalore, India, that have Apply Now buttons.

Build Cause-and-Effect Relationships

People understand faster when they can identify a cause-and-effect relationship between their actions and the system’s response. Sometimes the perfect time to show an explanation is in response to a user action. Users might be confused if an AI system completes an action but does not respond about its completion. If it reacts unexpectedly, an explanation can help in calibrating or recovering trust.

On the other hand, when the system is working well, responding to users’ actions is a great time to tell the user what they can do to help the system continue to be reliable. For example, a user looks for personalized recommendations for breakfast places on a restaurant recommendation service like Zomato. If they only see recommendations for places they rarely visit or that don’t match their earlier preferences, they might be disappointed and trust the service a bit less. However, suppose the app’s recommendation includes an explanation that the system only recommends restaurants within a 2 km driving distance and that the user is standing in the heart of Cubbon Park in Bangalore. In that case, trust is likely to be maintained. The user can see how their actions affect the suggestion.

Building trust is a long-term process. A user’s relationship with your product can evolve over time through back-and-forth interactions that reveal the AI’s strengths, weaknesses, and behaviors.

Two user interface images for Food Monkey. The selections for the first image are: nearby restaurants you might like; your current location; for the second image, restaurants for you. The first image is to aim for, and the second one is to avoid.

Optimize for Understanding

Different stakeholders in your AI system need different levels of explanations. Your AI explanations need to be understandable. Simply publishing algorithms underlying AI systems or providing a dump of data used is not meaningful. A list of a billion operations is not an explanation that a human can understand.²⁴ The complexity of your explanations needs to be tailored for your users.

It can be challenging to explain how your AI system works. Providing a detailed explanation can sometimes confuse users. In such cases, the best approach is not to attempt to explain everything—just the aspects that impact user trust and decision-making.²⁵ Sometimes, it is required by law in certain regions like the EU for this information to be communicated “in a concise, transparent, intelligible and easily accessible form, using clear and plain language.”²⁶

The ability to explain can also determine the fate of your AI product. Product teams are responsible for making important judgment calls about which AI technologies might best be deployed for specific applications. A huge consideration here is accuracy vs. “explainability.”²⁷ Deploying a deep learning system in high-stakes environments like medicine or driverless cars that we can’t explain is risky. Such algorithms may require periodic regulatory scrutiny and sometimes might not be worth the cost of maintenance.

Explainability is critical for building user trust. Explaining your AI system so people can actually understand it is a fundamental human-centered AI design challenge.

Two user interface images for in-car navigation. The selection for the first image is a recommendation of the shortest route. For the second image, A I recommends the shortest route. The first image conveys the message aim for and the second one aims to avoid it.

Data Use Explanations

Types of Data Use Explanations

You can help build and reinforce user trust by enabling users to see what data is used and how it interacts with your AI system. Data use explanations can be categorized into three types:

1. 1.

   Scope of data use

    
2. 2.

   Reach of data use

    
3. 3.

   Examples-based explanations

    

Scope of Data Use

Scope refers to the type and range of data used to provide a result. When using scope-based explanations, show an overview of the data collected about the user and which aspects of the data are being used for what purpose.³⁴ Sometimes users may be surprised to see their information shown in the product; this can erode trust. To avoid this, explain to users where their data is coming from and how it is used. Your data sources need to be part of the explanation. For example, a personal assistant may offer to book a cab for your appointment. In this case, the personal assistant needs to communicate that it knows about your appointment because your calendar is linked and you have configured your preferred mode of transport as an Uber. However, remember that there may be legal, fairness, and ethical considerations for collecting data and communicating about data sources used in AI.³⁵ Sometimes this may be legally required in certain regions.

User interface image for Time to Leave notifications. The notification reads, "A I assistant, flight to Mumbai, Leave by 7:30 a.m. to arrive at the airport two hours before your flight, 6E1234. Image of a map, 30 minutes to Kempegowda International Airport, Bangalore, scope of data explanation, and two buttons for navigating or booking a cab.

Reach of Data Use

Reach-based explanations tell users if the system is personalized to them or a device or if it is using aggregated data across all users.³⁶ Your sources of data need to be part of reach-based explanations. For example, when Spotify creates personalized playlists for users, the description often explains how it was generated: “Your top songs 2020—the songs you loved most this year, all wrapped up.” When Amazon’s recommendation system on a product page shows aggregated suggestions, the explanation also explains the logic for the recommendation in a user-friendly manner: “Customers who read this book also read…”

Image A illustrates the aggregate suggestions for the different books that different customers bought for the same book that you might have purchased on Amazon. And Image B illustrates suggested personalized playlists on Spotify.

Examples-Based Explanations

Sometimes it is tricky to explain the reasons behind an AI’s output. In examples-based explanations, we show users examples of similar results or results from the training data relevant to the current interaction. Examples can help users build mental models about the AI’s behavior intuitively. These explanations rely on human intelligence to analyze the examples and decide how much to trust the classification.³⁷ Examples-based explanations can be generic or specific.

Generic Explanations

The system shows users examples where it tends to perform well and where it performs poorly. For example, an AI that detects lung cancer from X-ray images can show the types of images it performs well on, which may have a proper orientation, may be well-lit, etc. It can also display images on which it performs poorly, which may not have a correct orientation, may have too many organs in view, etc.

User interface image of Argos X-ray analytics. List the conditions under which it works best and those conditions under which it should be avoided.

Specific Explanations

With specific examples-based explanations, the system displays the most similar examples relevant to the current interaction. For example, a dog classification AI might show similar images from its training data along with the prediction when you show it a photo of a poodle. Displaying similar images can help users judge whether they should trust the “poodle” classification.

A user interface image of the dog detector. While using a camera that points to a dog that detects it as a Poodle, which is 97 percent match, widens the search about the Poodle's history and exhibits similar images of the Poodle breed.

Descriptions

Types of Descriptions

Designing user-centered explanations for your AI can significantly improve your product’s user experience. In general, we can categorize descriptions into two types.

Partial Explanations

When deciding the description type during an interaction that can increase or maintain trust in your AI, a partial explanation is most likely the best one. In these descriptions, we intentionally leave out functions that are unknown, too complex to explain and understand, or simply not useful. In most cases, you can help build user trust without necessarily explaining exactly how an algorithm works or why it made a prediction. Your partial explanations can be generic or specific:

1. 1.

   Generic explanations

   General system explanations talk about how the whole system behaves, regardless of the specific input. They can explain the types of data used, what the system is optimizing for, and how the system was trained.⁴¹ For example, a marathon training app can say “This app uses your height, weight, and past runs to find a workout” when suggesting exercises.

    
2. 2.

   Specific explanations

   They explain the rationale behind a specific AI output in a manner that is understandable. For example, a recipe recommendation system can say, “We recommended this recipe because you wanted to make a light meal with broccoli, pasta, and mushrooms,” or a dog recognition system can say, “This dog is most likely a German shepherd because of XYZ features.” Specific explanations are useful because they connect explanations directly to actions and can help resolve confusion in the context of user tasks.⁴²

    

A user interface image of the ebook reader consists of generic and specific explanations. The generic explains how the whole system behaves in general, regardless of the specific input, and the specific explains the rationale behind a specific A I output in a manner that is understandable.

Full Explanations

These are detailed explanations of how your AI system works or why it made a particular prediction. These can be in the form of research papers, blog posts, open source code, etc. In most cases, it is not advisable to give full explanations within your product’s user experience. Such explanations might be irrelevant for the task and can also confuse users. If you really want to provide full explanations, you can do it through progressive disclosure, tooltips with external links, etc. The best place for full explanations can be on your product’s blog, marketing collaterals, the company’s landing page, or even the careers page to attract the right candidates.

Confidence-Based Explanations

Confidence-based explanations are unique to probabilistic systems like data science reports or AI products. A confidence level is a statistical measurement that indicates how confident the AI system is about a particular outcome.⁴³ Confidence-based explanations provide answers to the following question: “How confident is the system about a prediction or an outcome?”

Confidence is a readily available output from an AI system, the value of which ranges from 0 (no confidence) to 1 (complete confidence). Because these systems are based on probability, in most real-world scenarios, you will never get a definite 0 or 1. Sometimes, we convert these values into percentages. Displaying confidence levels can help users gauge how much to trust an AI’s output. Rather than describing why the AI came to a particular decision, confidence-based explanations tell users how certain the AI is about the output. Sometimes it shows the alternatives the AI model considered.

Let’s assume that we build an AI system that detects donuts from images. We categorize results into high, medium, and low levels of confidence. A result with a value less than 40% has low confidence, while one with a value of more than 80% is categorized as high confidence, and anything between 40% and 80% has a medium level of confidence. The team building the AI decides these threshold values, which can vary across systems. Using these confidence levels and categories, we can customize the system’s output. For example, if you show an image of a donut that results in a high level of confidence, we can design it to say, “That’s a donut.” If the confidence about the image is medium, then it can say, “Maybe that’s a donut.” When you show the donut detector a picture of pasta that results in a low confidence, it can say, “I don’t think it’s a donut.”

A table has four columns and three rows. The headers are image, confidence value, confidence level, and system output. Row 1. a photo of flavored donuts, 98 percent, High, That's a donut. Row 2. a photo of a long john donut, 65 percent, medium, Maybe this is a donut. Row 3. a photo of noodles, 15 percent, Low, Not a donut.

Types of Confidence-Based Explanations

Displaying confidence can help users gauge how much trust to put in an AI’s output. If your research confirms that showing confidence improves decision-making, the next step is to choose the best type of explanation. Confidence-based explanations can be categorized into four types:

1. 1.

   Categorical

    
2. 2.

   N-best results

    
3. 3.

   Numeric

    
4. 4.

   Data visualizations

    

Categorical

Instead of showing a numerical value of the confidence (between 0–1), confidence levels are categorized into buckets like high/medium/low, satisfactory/unsatisfactory, etc. Generally, the product team determines the threshold confidence values and cutoff points in consultation with ML counterparts. It is important to think about the number of categories and what they mean. Determining categories will require trial and error and testing with your users. You can further use the category information to render the appropriate user interface, alter messaging, and indicate further user action. For example, when you ask a voice-enabled speaker to play a song, it could respond differently depending on its confidence in understanding your command. It might play the song for high-confidence cases, while it might ask you to repeat the command for low-confidence cases. Similarly, a search engine might tailor its UI based on its confidence about a query.

Three user interface images for the Voice Assistant. The scenarios are for high confidence, Hey Assistant. Play Adele, playing music by Adele. For medium confidence, Hey Assistant. Play Adele, Did you mean Adele? And for low confidence, Hey Assistant. Play Adele, Sorry! I could not understand. Can you repeat it?

N-Best Results

N-best means showing a specific or “n” number of best results that your AI outputs. We display multiple results in this method rather than presenting only one result with a particular confidence level. The confidence values of those results may or may not be indicated. This type of explanation is advantageous in low-confidence situations or in cases where it is valuable to provide users with alternatives. Showing multiple options prompts the user to rely on their own judgment. It also helps people build a mental model of how the system relates to different options.⁴⁶ Determining the number of alternatives will require testing and iteration.

Displaying N-best results is very common. For example, a smartphone keyboard with predictive text capabilities provides multiple choices of words. An application that detects animals from photos might show numerous options if it is unsure of what it’s seeing, for example, this photo may contain a goat, llama, or ram. The autosuggest capability in Google Search that suggests possible queries, Netflix recommending multiple movies that you might like, and Amazon recommending products based on past purchases are all examples of N-best explanations.

User Interface Image A illustrates the predictive text on the keyboard with the N best list. User Interface Image B illustrates the autosuggest in the search bar with the N best list. And User Interface Image C illustrates the search results in Audible for N best list of science fiction.

Numeric

In this method, numeric values of confidence are shown, often in the form of percentages. However, using numeric confidence explanations is risky since it assumes that users have a baseline knowledge of probability and an understanding of threshold values. Showing numeric explanations can even confuse users. It is difficult to gauge if an 87% confidence value is low or high for a particular context. Users might trust an AI that predicts a song they like with 87% confidence, but they would be wary of trusting it to drive their kids to school.

Make sure to provide enough context to users about your numeric explanations and what they mean. Remember that since AI systems are probabilistic, you will almost never get a 100% confidence value. There are two types of numeric explanations:

1. 1.

   Specific explanations provide users with a prediction along with the AI’s confidence value. For example, when you select a movie on Netflix, it shows a percentage match indicating your likelihood of enjoying it.

    
2. 2.

   General explanations present an average confidence of the system, for example, this app recognizes dogs with a 90% accuracy.

    

User Interface Image A illustrates the match score on Netflix titles that is 98 percent match for the Squid Game Series. And User Interface Image B illustrates a confidence score of 8.3 to 9.6 on a profile of a Digital Content Producer in Photofeeler.

Data Visualizations

When it comes to indicating confidence values, data visualizations are graphical indications of certainty over a span of time, types of results, or any other metric. For example, a stock price or a sales forecast could include elements indicating a range of possible outcomes based on your AI’s confidence. Your data visualizations can be static or interactive. Keep in mind that many data visualizations are best understood by expert users in specific domains. However, it is safe to assume that many people understand common visualizations like pie charts, bar graphs, trend lines, etc. Using a visualization and choosing the right type will require user research, testing, and iterations.

Using data visualizations to display confidence can be especially useful if you are designing AI systems for expert users. This type of user might appreciate your AI’s understanding of their domain and trust it more.

A user interface image of Google Ads, Keyword plan for the monthly searches of different brands and non-brands for December 2020 to November 2021 peaked at March 2021 by 30 percent.

Explaining Through Experimentation

People are often curious about how an AI-powered feature will behave. Users can be impatient and might want to jump into your product experience right away. They might skip any onboarding flows you might’ve designed to explain the AI system. It can help keep your onboarding short and suggest low-risk, reversible actions they can try out. For example, a document scanning app might ask the user to take a picture of a receipt and convert it to a scan with selectable text. It might allow them to copy text from the image, edit the scan area, apply image filters, etc. In this case, trying out the image-to-text extraction feature is much better than explaining through words. Such small, contained experimentation environments can help users build a mental model of your system, and you can start building user trust.

User interface images for the process of using the document scanner application. Step one is to open the application that will welcome you to an interface for the Scan your document first, press the scan button at the bottom middle part of the screen, and then step 2 for the actual scanning of documents. Press the scan button again to confirm.

You need to be intentional about letting users interact with the AI and test its limits. Building an experimentation-based explanation can be time-consuming and require a lot of effort. Developing such experiences will need multiple rounds of user testing.

Guidelines to Design Better Experimentation Experiences

Allowing users to try out and tinker with your AI product can help build trust and improve usability. Users can get up to speed with your product faster. The following are some guidelines to help you design better experimentation-based explanations:

1. 1.

   Allow experimentation with specific features.

   Asking users to try out the entire system as a way of experimentation is not explainable. Doing this can even confuse users about your product’s value. You need to tie your explanations to a specific output. For example, changing the lighting in a document scanning application can give a better outcome, or in a search engine, modifying your search query can provide better results. Point users to specific features where they can quickly understand the value of your product. Otherwise, they may find the system’s boundaries by experimenting with it in ways it isn’t prepared to respond. This can lead to errors, failure states, and potentially erosion of trust in your product.⁴⁷

    
2. 2.

   Consider the type of user.

   A user’s willingness to experiment will depend on their goals. For example, an average consumer might enjoy trying out different AI-enabled features on a smart TV as part of the explanation, but a person buying hundreds of TVs for an enterprise might find this frustrating.

    
3. 3.

   Build cause-and-effect relationships.

   People learn faster when they can build a cause-and-effect relationship between their actions and AI’s outputs. Within an experimentation environment, a specific user action should generate a response. The AI can respond to the success or even failure of the system in the form of errors. Providing no response can keep users waiting and frustrated and can erode trust in your system. For example, a smart speaker that offers no response and keeps the user hanging despite an error is worse than providing an error response.

    

Two user interface images for Voice Assistant. The first scenario is Hey Assistant. Play Adele, Sorry! I could not understand. Can you repeat? The first scenario is to aim for. And the second scenario is, Hey, Assistant. Play Adele, the Voice Assistant, did not answer, Hello? Are you there? The second scenario is to avoid.

No Explanation

In some cases, there is no benefit in explaining the AI’s function. If the way an AI works fits a common mental model and matches user expectations for function and reliability, then there may not be anything to explain in the interaction.⁴⁸ For example, in a camera application that automatically adjusts to external lighting, it might be irrelevant to explain every time it adjusts the image. Your users might already have a mental model of when and how that happens. Explanations can sometimes get in the way of a user’s actions. In the case of an AI writing assistant, it can be distracting if the system explains every time it corrects a word while you are drafting an email. It would also be wise to avoid explanations if they reveal private information or proprietary techniques.

Internal Assessment

User Validation

You should also validate your explanations with the users of your product. Your user group should reflect the diversity of your audience. You can use qualitative or quantitative methods to validate your explanations.

Balance Control and Automation

Hand Off Gracefully

When your AI makes mistakes, the easiest way is to return control to the user. It should be easy for users to take over the system. The level of control and override would depend on the type of situation. Sometimes, it is risky to hand off to a user immediately. For example, it can be dangerous if a self-driving car suddenly asks the driver to take over at high speed. You need to ensure that the handoffs are graceful. When this AI-to-manual control transition happens, it’s your responsibility to make it easy and intuitive for users to pick up where the system leaves off quickly. That means the user must have all the information they need in order to take the reins: awareness of the situation, what they need to do next, and how to do it.⁵⁰

Image A illustrates the chatbot support replies of automated messages. Image B illustrates the smart speaker with talk support to an artificial intelligence bot.

Data Control

Your users should have the ability to view, access, edit, and share their data that the AI system uses privately and securely. The product should empower them through an explicit opt-in system and explainable information of how the AI stores and uses their data. You can allow users to select the data that the system uses and disable parts of the data they don’t want to be used. In some cases, this might result in limited functionality, inaccurate results, or even the AI not working. Communicate these limitations, but don’t force users into providing data if they wish to opt out. Not allowing users control over their data erodes their trust in the system.

Here are some considerations when designing data control mechanisms.

Global Controls

Provide global controls by allowing users to customize what the system monitors and how it behaves.⁵³ Here are some guidelines to design better global controls:

1. 1.

   Let users enable or disable external inputs. For example, enabling or disabling location input in a shopping application can enable or disable a “trending near you” feature.

    
2. 2.

   Allow users to indicate preferences. For example, a running app can allow users to indicate if they want to avoid slopes, rocky trails, or traffic.

    
3. 3.

   Continue to clearly communicate permissions and settings throughout the usage of your AI. Ask permissions early in the user journey. Think about when the user might want to review preferences they’ve set in the past and consider reminding them of these settings when they shift into different contexts and may have different needs. They may also forget what they’re sharing and why, so explain the reasons and benefits.⁵⁴

    
4. 4.

   Allow users to deny service or data by having the AI ask for permission before an interaction or providing the option during an interaction. Privacy settings and permissions should be clear, findable, and adjustable.⁵⁵

    
5. 5.

   Make sure that your controls are not too broad. Simply enabling or disabling an AI feature is a very broad control that leads to your users either using the product or not, that is, they need to fully trust your AI or not at all. For example, an AI-based music recommendation service with controls that only allow enabling or disabling the service would not be useful for many users. Users who want to use the service will be forced to accept all data inputs even if they aren’t comfortable sharing parts of their data. Forced data sharing can erode your user’s trust in the AI and easily make them susceptible to switching to a competitor. However, if users can control which parts of their data are shared with the service, they can calibrate their level of control and trust in the system. For example, enabling users to disable location data in the music recommendation service can disable location-based playlists but allow them to use the rest of the application. Keep in mind that there is a risk of being too granular with the level of control you provide. Users can be confused if they have to choose granular controls like the precision and recall of your ML models.

    
6. 6.

   Don’t stop users from deleting or modifying information that the system has already collected. For example, a search engine should allow users to delete previous searches, or a running app should allow users to remove past runs. Users should always maintain control over what data is being used and in what context. They can deny access to personal data that they may find compromising or unfit for an AI to know or use.⁵⁶ In many cases, this might be a legal requirement.

    

Image A illustrates the i O S settings selections like Under Privacy, location services, tracking, contacts, calendars, reminders, photos, Bluetooth, and other selections. Image B illustrates the Nike Run App Settings with selections for Indoor, Auto Pause, Audio Feedback, Countdown, Orientation, and other selections.

Editability

Your user preferences may change over time. Consider giving users the ability to adjust their preferences and data use at any point in time. Even if your AI’s recommendations were not relevant initially, they might become better later as it gathers more information about the users’ preferences. Your product should allow for people to erase or update their previous selections or reset your ML model to the default, non-personalized version.⁵⁷ For example, a movie recommendation service might adapt to a user’s preferences over time, but it might not be as expected. As the user’s tastes in movies evolve, they might want to change the preferences that your AI captured previously. Editability allows them to calibrate their recommendations and thereby their trust in the system.

Image A illustrates the i O S setting for the Nike Run Application with Selections, Allow Location Access, Never, Ask Next Time or When I Share, and the button for Precise Location. And Image B illustrates the profile settings in the Nike Run Application with selections for About You, Gender, Height, Weight, and the button for use of default height and weight.

Removal and Reset

Tell users if they can remove their entire data or a part of it. Removal of data is different from disabling inputs. In the case of removal, all or a selection of previous data that the system collected is deleted. Additionally, you can allow users to reset their preferences entirely. The ability to reset is useful when your system provides a large number of irrelevant suggestions and the best way is to start from scratch. For example, a user logging into an online dating service might find the recommendations personalized for their past self, irrelevant after a few years. Their preferences for dates or partners might have changed. The current suggestions may not be desirable and can erode the user’s trust in the service. While this system can painstakingly calibrate its recommendations over time, in such cases, resetting data is the best course of action.

Image A illustrates the removal of past searches in the Google search bar. Image B illustrates the reset of YouTube activity data.

Opting Out

Your system should allow users to opt out of sharing certain aspects of their data and sometimes all of their data. Opting out of sharing their entire data can be equivalent to a reset. Consider allowing users to turn off a feature and respect the user’s decision not to use it. However, keep in mind that they might decide to use it later and make sure switching it on is easy. While users may not be able to use the AI to perform a task, consider providing a manual, nonautomated way to complete it.

Empower users to adapt your AI output to their needs.

Image A illustrates the contact permission per application in the i O S settings by use of the toggle button. And Image B illustrates the WhatsApp Application permission in the i O S setting by use of the toggle button to allow permission.

Control over AI Output

Your AI won’t be perfect for every user all the time. Empower users to adapt your AI output to their needs, edit it, ignore it, or turn it off. Here are some considerations when enabling users to control the AI’s outputs.

Provide a Choice of Results

If possible, allow users to select their preferred recommendations from a number of results, for example, a movie recommendation service that shows suggestions but allows users to choose which title to watch. You can ask users to pick which results they would like to see more, which can help the AI model improve over time. For example, a news app allows users to choose the topics they’d like to see more. Keep in mind that providing multiple recommendations is not always possible.

Image A illustrates the Netflix suggestion for a user named Akshay, through Top Picks. Image B illustrates the medium homepage recommended for the user based on their following. Image C illustrates Amazon book recommendations through different categories.

Allow Users to Correct Mistakes

Any AI system will inevitably be wrong. Your product can empower users to correct the AI’s mistakes through feedback. You might allow users to configure a list of results and their ranking, up-vote or down-vote results, or even provide detailed feedback. For example, a service that creates automatic playlists can allow users to like or dislike a song, change the order of songs, add or delete titles, etc. You can sometimes enable users to undo the inferences made by the system. Allowing a user to correct mistakes can establish the mental model that your system improves over time, and the user can play a part in this improvement. Being able to correct mistakes helps users to calibrate their trust over time.

Image A illustrates the Netflix thumbs-down button for a specific movie. Image B illustrates the Apple Music feedback through the heart button and the thumbs-down button. Image C illustrates the YouTube recommended videos with the options tab.

Support Efficient Dismissal

Make it easy to dismiss any irrelevant or undesired AI recommendations. Make sure to clarify how to dismiss a result, for example, swipe left on a dating app, saying “Cancel” in the case of a voice assistant, hiding or reporting ads in a search engine, etc. Don’t obfuscate the method of dismissal, for example, a dismiss button for a text autocomplete feature that is too small or not obvious enough.⁵⁸

Image A illustrates the Spotify daily mix interface and highlights the skip song button. Image B illustrates the LinkedIn job recommendation for a Senior User Researcher. Image C depicts Google News with the dismiss options selected on the page. And Image D illustrates the Tinder interface with like and dislike buttons.

Make It Easy to Ignore

Make it easy to ignore irrelevant or undesired AI recommendations. The ability to ignore a recommendation is especially useful in cases where the AI has a low level of confidence. You can do this by providing your recommendations unobtrusively. For example, a shopping application can show recommended products below the fold of the main product page that are easy to ignore.

People trust things that other people have trusted.

A user interface image of the Amazon product page with different book recommendations that highlights the section of customers who bought this item, as a section that is easy to ignore.

Set the Right Expectations

Introduce Features Only When Needed

Onboard in stages and at appropriate times. Avoid introducing new features when users are busy doing something unrelated. This is especially important if you are updating an existing product or a feature that changes the AI’s behavior. People learn better when short, explicit information appears right when they need it.⁶⁵ Introduce AI-driven features when it is relevant to the user. Avoid introducing AI-driven features as a part of a long list of capabilities.

Two user interface images for the application of Pedalytics. Image 1 has many options for a bike route and a start button in the middle bottom area. The image one is to aim for. Image 2 has only one choice for a bike route but has a next and skip button at the bottom area. Image 2 is the one to avoid.

Clarify Data Use

As you onboard users to a new feature, they might have various concerns around privacy, security, and how their data is used. Make sure that users are aware of any data that is collected, tracked, or monitored and that it’s easy for them to find out how the data is collected, whether via sensors, user-entered data, or other sources.⁶⁶ Ask for data permissions when relevant. For example, a running app might ask for location and GPS tracking permissions when it is creating route suggestions.

Image A illustrates the Google Assistant Onboarding terms of service and privacy policy. Image B illustrates the Onboarding for the Voice Match feature on Google Assistant. Image C illustrates the Apple Maps privacy policy in onboarding, where the data use explanation is highlighted.

Allow Users to Control Preferences

Give users control over their data and preferences as they get started. Give them the ability to specify their preferences and make corrections when the system doesn’t behave as expected, and give them opportunities to provide feedback.⁶⁷ This can also help set expectations that the AI will adapt to preferences over time.

Image A is for the preference selection on Headway application onboarding. The title is what your goals are, with a continue button. B illustrate the Spotify onboarding process with preference selection for different musicians. C is the Flipboard application. It has the title, pick topics to start reading and saving articles. Articles are listed below.

Design for Experimentation

People sometimes skip the onboarding process because they are eager to use the system. Make it easy for users to try the experience first. Your onboarding can include a “sandbox” experience that enables them to explore and test the product with low risk or initial commitment.

Three user interface images for the onboarding steps of the Otter dot a i app, starting from getting the started button, then recording sample audio. It has a pop-up. Let's record a conversation. Finally, recording and visiting the transcript update. It has a pop-up that says a few words.

Reboarding

Consider reboarding, that is, onboarding again, if there are noteworthy improvements in how a feature works or the data it uses. Ensure that these changes are significant enough for users to notice. You can also reboard existing users who are interacting with your system after a long time. In such cases, it is good to ask users if they want to be onboarded again and provide the option of skipping. Reboarding can also be used when a user forgets how to use your service and needs a refresher in the form of tutorials, wikis, etc.

Set the Right Expectations

Images A illustrates the search bar for LinkedIn, where the scope of search on the Jobs page and the scope of search on the homepage are highlighted. Image B illustrates Netflix's suggestions for Akshay, where the movie selected is Black Holes: The Edge of All We Know.

Clarify Data Use

Make sure your users understand what data is collected, tracked, and monitored by the system and how it is used. You should clarify which parts of their data are used for what purpose, what is private, and what is shared. You can also let your users know if the AI’s results are personalized for them or aggregated over lots of users.

Build Cause-and-Effect Relationships

You can strengthen your AI’s mental models by building cause-and-effect relationships between your user and the AI system. In many cases, the perfect time to show an explanation is in response to a user’s action. People understand better if they can establish a relationship between their response and the AI’s output. A user’s relationship with your product can evolve over time through back-and-forth interactions that reveal the AI’s strengths, weaknesses, and behaviors.

Allow Users to Choose, Dismiss, and Ignore AI Results

Users can build a more trustworthy relationship with your AI system if they can use their own judgment. Whenever possible, your AI system should provide a choice of results. If your AI’s results are not helpful, users should be able to ignore or dismiss them easily.

Provide Global Data Controls

Provide global controls by allowing users to customize any data that is collected, tracked, or monitored by the system. Let users enable or disable any external inputs like location, call information, contacts, etc. Clearly communicate permissions and privacy settings. Make sure they are findable and adjustable.

Clarify Data Use

In your system settings, make sure to communicate what data is collected, tracked, and monitored by the system and how it is used. You should clarify which parts of their data are used for what purpose, what is private, and what is shared.

Allow Editing Preferences

Your users might have provided their preferences during onboarding. Sometimes, your AI can learn user preferences over time. However, user choices can change over time. Allow users to specify or edit their preferences at any point in time. For example, a music recommendation service might adapt to a user’s likings over time, but it might not be as expected. As users’ tastes in music change, they might want to change the preferences that your AI captured previously.

Allow Users to Remove or Reset Data

Whenever possible, allow users to remove their entire data or a part of it. You can also provide users with the ability to reset their information and start from scratch. This is especially useful if the system has learned incorrect preferences leading to irrelevant predictions. In such cases, it is best to reset and start from scratch.

Allow Opting Out

You should allow users to opt out of the system even if it means that they can no longer use your service. Respect the user’s decision not to use your service and communicate appropriately. Remember that building trust is a long process, and your users might decide to use the service at a later point. You can also enable users to opt out of sharing certain aspects of their data partially.

Image A illustrates the Ad References for Goodreads. Image B illustrates the history setting for a particular Google account. And Image C illustrates the preference for editing on Flipboard.

Adjust User Expectations

Hand Off Gracefully

When your AI makes a mistake, the easiest method is to give users a path forward by giving them control. You can allow them to take over the system and manually complete the action. Address the error in the moment and let users know how you will resolve the problem. However, you need to design the AI-to-manual transition carefully to ease your users into taking control of the system.

Allow Users to Correct AI Mistakes

You can enable users to correct the AI’s mistakes through feedback. Providing feedback can also help set expectations for adaptation. Prevent the error from recurring: give users the opportunity to teach the system the prediction that they were expecting, or in the case of high-risk outcomes, completely shift away from automation to manual control.⁷⁰ Ability to correct mistakes helps users calibrate their trust in your AI system.

Allow Users to Choose, Dismiss, and Ignore AI Results

When the AI makes mistakes, users should be able to make their own judgments. If possible, allow users to choose from a number of results. When a result is irrelevant, users should be able to ignore or dismiss it. Allowing users to choose, dismiss, or ignore its results can help your AI build a more trustworthy relationship with your users.

Image A illustrates Google news where Dismiss options are selected. Image B illustrates the Spotify player interface where the skip song button is highlighted. Image C illustrates the LinkedIn Job recommendation where the dismiss recommendation button is selected. And Image D illustrates the Netflix suggestions for Akshay, where a movie is chosen.

Table 4-2

Opportunities for building trust in your user journey

Opportunities	Considerations
Onboarding	Set the right expectations.
	Introduce features only when needed.
	Clarify data use.
	Allow users to control preferences.
	Design for experimentation.
	Consider reboarding existing users periodically.
User interactions	Set the right expectations.
	Clarify data use.
	Build cause-and-effect relationships.
	Allow users to choose, dismiss, and ignore AI results.
Loading states and updates	Provide information about updates to the system.
	Inform users about any changes to the privacy policy or legal regulations.
	Inform users about any changes to how data is used and shared.
Settings and preferences	Provide global data controls.
	Clarify data use.
	Allow editing preferences.
	Allow users to remove or reset data.
	Allow opting out.
Errors	Adjust user expectations.
	Hand off gracefully.
	Allow users to correct AI mistakes.
	Allow users to choose, dismiss, and ignore AI results.

Don’t Pretend to Be Human

Clearly Communicate Boundaries

You should clearly communicate your AI’s limits and capabilities. When interacting with an AI with a personality and emotions, people can struggle to build accurate mental models of what’s possible and what’s not. While the idea of a general AI that can answer any questions can be easy to grasp and more inviting, it can set the wrong expectations and lead to mistrust. For example, an “Ask me anything” callout in a healthcare chatbot is misleading since you can’t actually ask it anything—it can’t get you groceries or call your mom. A better callout would be “Ask me about medicines, diseases, or doctors.” When users can’t accurately map the system’s abilities, they may overtrust the system at the wrong times or miss out on the greatest value-add of all: better ways to do a task they take for granted.⁸²

User interface images for HealthChat. Image one is the application suggesting questions and the user opting for a doctor's consultation. One should aim for image one. Image two should be avoided. In image two, the user asks to get groceries. HealthChat responds to only health-related concerns.

Consider Your User

Consider Cultural Norms

When deploying an AI solution with a personality, you should consider the social and cultural values of the community within which it operates. This can affect the type of language your AI uses, whether to include small-talk responses, the amount of personal space, the tone of voice, gestures, non-verbal communications, the amount of eye contact, the speed of speech, and other culture-specific interactions. For instance, although a “thumbs-up” sign is commonly used to indicate approval, in some countries this gesture can be considered an insult.⁸³

Designing Responses

Leveraging human-like characteristics within your AI product can be helpful, especially if product interactions rely on emulating human-to-human behaviors like having conversations, delegation, etc. Here are some considerations when designing responses for your AI persona.

Risks of Personification