Creating More Engaging Applications

If there’s one area commonly identified with AI, it is its ability to enable more engaging applications. Actually, one of the canonical examples of AI is directly targeted at increasing the time spent in the application—the recommender system.

Automatic recommendations are not new. We’ve been creating recommenders for years with traditional programming techniques, statistics, traditional machine learning algorithms, and more recently, deep learning. At every step the relevance and accuracy of these recommenders have been improving, and therefore their business impact has been growing. About 75% of viewer activity on Netflix is driven by its famous recommender system, and 35% of what consumers purchase on Amazon comes from product recommendations based on these algorithms.

At their core, these recommender systems are very similar. They rely on collaborative filtering techniques: the items recommended to a user are based on the preferences of similar users, who are clustered based on their preferences. The preferences might be explicit (such as ratings or likes), or they could be based on subtler implicit signals, such as visits to a product page, time spent on the page, search queries, etc. The possibilities are almost infinite, and the impact follows the Pareto principle—if you are not using recommendations today in your systems of engagement, you can easily get 80% of the benefit with a 20% effort.

Recommender systems are data-hungry, though. You will need an existing dataset of customer behavior to make the system relevant, which can be tricky, especially for long-tail cases like products that are not sold very often or movies that are not watched much. Some techniques can be used to avoid that shortcoming; actually, most recommender systems nowadays use a hybrid approach of collaborative filtering and content-based systems. With the latter, you classify your items statically (e.g., by gender, category, or type) and recommend items with similar attributes. For example, if the user likes horror movies (which wouldn’t be me, certainly!), you’ll show them other movies in the horror genre. This attribute extraction can be powered by AI as well. For example, Bing Ads uses computer vision to extract text and other information from advertisements in order to increase the applicability of the ads it displays to the search query performed by the user. Reuters automatically embeds videos in news articles by extracting the topics covered in the article with AI.

Another popular way to increase user engagement is personalization. Traditionally, application and web development relied on A/B testing techniques to optimize the user experience: a developer could create two different versions of the interface and release them simultaneously to different sets of users, and after a period of time, they could look at the engagement metrics to decide which user experience was more engaging.

AI can scale that process to the point that the interface is optimized individually: you can present slightly different experiences to users, and based on their behavior you can optimize the experience for each of them. The home screen in Xbox uses this approach. The system uses a reinforcement learning algorithm to optimize what tiles to show and in what order, based on what tiles a user has clicked in the past and what other users are clicking.

Machine learning techniques can also be applied to other user actions. Using the historical data on click-through for the notifications raised by your application, you can optimize what to show and when to show these notifications to increase their effectiveness. Windows 10 uses this technique to decide when to show system notifications or tips.

Other uses of AI are not directly connected to user engagement but can help in the long term. For example, it can be used to:

• Increase the security of an application, making users more likely to trust it and engage more often with it. For example, Uber uses computer vision as an authentication factor for the driver, increasing security for passengers.

• Scale the analysis of feedback coming from the user, with techniques like sentiment analysis or entity extraction. For example, travel agencies can use AI to analyze hotel user reviews and identify issues in real time. Geocaching, an entertainment company that provides an outdoor treasure hunting experience, uses sentiment analysis and key phrase extraction from its comment logs to uncover valuable insights and resolve issues quickly.

AI opens up new ways of interacting that can make our applications more engaging or accessible to more users. Today, prebuilt AI services can help us create systems that use text in multiple languages, human voice, gestures, ink, or even eye movement as the user input. You can use these techniques to make your existing applications more accessible and attractive to a broader set of users, but you can also use them to create entirely new form factors for your business, like retail kiosks, bank concierges, no-screen applications, and more. Equadex is a French company that commercializes the app Helpicto, a good example of an app that uses a natural interface to make it more engaging. Helpicto uses voice and pictograms to help people on the autism spectrum to communicate.

Augmented reality also makes heavy use of AI techniques, to perceive the world around the user and augment it. These augmented experiences represent a whole new level of user engagement and can be used in scenarios such as remote assistance, training, augmented processes, and many others. Chevron, for example, uses HoloLens to enable its refinery operators with remote expert assistance powered by augmented reality. The Oslo University Hospital is currently testing augment reality for a HoloLens-based surgery augmentation system, providing surgeons with a roadmap to successfully navigate around internal organs.

Creating More Productive Applications

Funnily enough, in many situations your goal is not to increase the time spent on your systems of engagement, but to reduce it. Another set of AI uses cases are aimed at reducing the time needed by the user to perform their tasks. These techniques are especially relevant for applications that represent a time-consuming activity in your organization, and can be a great source of cost savings in the short term.

Data entry is definitely a time-consuming activity in most internal applications. AI techniques like computer vision and optical character recognition (OCR) in particular can simplify the process dramatically. Form-based OCR is a mature use case of AI that can be applied to existing applications, such as bank forms or expense reports—for example, the Danish accounting specialist Dinero automates the submission of invoices for its customers by automatically extracting the key elements with OCR. Less obvious scenarios are also providing very promising results. For example, object detection can be used to accelerate data entry: car insurance claims, urban damage notifications, and equipment ordering can be greatly simplified by extracting data automatically from images.

In other scenarios, data has to be entered manually. In those cases, smart autocompletion powered by AI can accelerate the process: based on existing information and contextual information, the system can recommend potential content for the form automatically. For example, Outlook replaced its autocompletion system for contacts and search with an AI-powered algorithm that makes the autocompletion much more relevant to users. In an extreme application of this concept, Outlook, LinkedIn, and Gmail are also able to recommend several full answers to an incoming email based on the its content, learning from past messages. For developers, Visual Studio is able to provide automatic completion in your source code, learning from the code of millions of other developers on GitHub.

For tasks that are more creative, AI can assist the user and provide support in the creation process. PowerPoint, for example, provides design ideas based on the current content in a slide, and hints on how to become a better presenter by listening to your rehearsal. Excel can provide ideas to help you better understand your data by automatically identifying patterns.

Search is another common capability in systems of engagement that can be significantly improved with AI, reducing the time spent on finding information. Modern search services for your applications, such as Azure Search, have improved dramatically by leveraging core AI techniques such as word embedding. More important, AI is also enabling a new type of search, usually characterized as cognitive search. Cognitive search leverages AI models to extract additional attributes from your assets, enabling new scenarios. For example, using an OCR model you can extract text from your documents in physical format and perform a search over them. Using an object detection model, you could enable a search on a repository of images in your organization. Using speech recognition, you could search for something mentioned in a video or a voice recording. Cognitive search can bring together all the structured and unstructured assets in your company and make them searchable and accessible to users. At Microsoft, we did an interesting experiment with cognitive search that you can access online: we applied these techniques to all the documents released on the assassination of John F. Kennedy, allowing users to not only search through the content, but also understand the entities, relationships, and even photographs included in the documents.

Approaches to Your Conversational AI Strategy

A great source of use cases for your conversational AI strategy is the application portfolio inventory mentioned earlier. Existing customer and employee interactions associated with your applications and website are great candidates for conversational agents. For the evaluation process, consider the following aspects:

• Start with narrow use cases (chatbots) and set the right expectations for the user. Conversational agents can be very disappointing if the scope is not clearly understood by the user. Developers can use available design guidance documents to follow this principle, like the one published by Microsoft Research.

• Pick use cases that can specially benefit from a conversational AI. Decision flows (e.g., selecting a product), conditional data entry (e.g., describing an issue), and exploratory requests (e.g., learning options for an insurance contract) are good examples of use cases that can provide a better experience with a conversational interface.

• Prioritize use cases that are highly demanded but difficult to discover or time-consuming to perform. A good tactic to find these use cases is analyzing the traffic and search terms in your application or website, and the time spent finding the resource or performing the transaction itself.

This simple framework should give you a good list of low-hanging-fruit scenarios that you can address with narrow chatbots. Technical departments will learn in that process and you will get immediate return on your investment to keep the motivation high. In the case of Telefónica, this low-hanging fruit was a chatbot integrated with the primary consumer mobile application in its O2 branch in the UK. The chatbot was able to attend to the most frequent requests from users in the app, including checking their bill, getting an update on their current data consumption, and getting information about their plan. For Unilever, it was a chatbot that was able to deal with common requests that employees made to the HR department. In both cases, the systems were highly successful and provided a great experience for the technical teams.

However, you shouldn’t stop there. Narrow use cases for conversational AI are just the beginning. As your technical organization gets more experience, you can address more complex scenarios.

Omni-Bots

A common approach for this gradual evolution is to bring together multiple narrow use cases into a broader agent—an “omni-bot.” This concept allows easier management and maintenance, even by multiple teams. A dispatcher is able to redirect the requests appropriately to each individual chatbot based on the content, or it can send each request to all of them and pick the chatbot with the best match. This pattern is very common and supported by conversational AI platforms such as Azure Bot Service, and the resulting conversational agent can become a central hub for the entire organization. 

At that point, organizations usually go one step further and provide some identity to the agent. Unilever, for example, is now creating a much broader internal agent centralizing multiple processes, called Una. Telefónica has created Aura, a personified agent centralizing all its services to customers.

The “hub” approach is not the only one used in complex agents. A promising approach to bot development is based on applying deep learning to the conversation itself to enable richer dialogues. Traditionally, bots process the user input turn by turn. Each command sent by the user is mapped to an intent using natural language understanding techniques. For example, “I want insurance for my car” could be mapped to the intent “Purchase insurance.” The developer then manages the logic of what to do next, and the approach usually involves multi-turn dialogues that are designed manually. You can experience this by yourself if you follow the bot development exercise in Appendix A.

That technique is simple and works great for narrow bots or aggregations of narrow bots, but it doesn’t scale for complex scenarios. A different approach to address these scenarios is to use a deep learning model to manage the conversation itself, and train it with past conversations as examples. This technique is available already in services such as Project Conversation Learner and is an active area of development that will enable more complex agents.

Think of customer support, for example. You can use logs of previous real support chats to train a generic model that can mimic those conversations, under the guidance of a business user. An option to escalate to a customer agent is provided if necessary, and the virtual agent can then support the customer support staff, increasing their productivity. This approach is already used by several companies, including HP, which provides a chatbot on its support site, Macy’s, which handles user requests on its ecommerce site via a virtual customer support agent, and Microsoft, whose customer support chatbot covers hundreds of different use cases.

As more of these “omni-bots” are created, their level of sophistication will also increase. Users will expect these bots not only to attend to their commands, but also to be knowledgeable. Knowledge-based bots are able to answer questions in natural language, instead of just receiving commands. In the case of personal assistants these questions are usually related to personal life (movies, places, shopping, etc.), but in your corporate assistant, the questions will be related to your business. Several technologies are available today to help technical teams to create this kind of bot. At their core, these technologies are able to bring together unstructured or chaotic information from your internal systems and store it in a way that a bot can use to answer general questions in natural language. Azure QnA Maker, for example, is able to take FAQ files already available in your organization and use them to answer similar questions in your bot.

When the level of complexity of your bot gets to this stage, you will notice an interesting effect. The requests received from your users will now not only be command-based or knowledge-based; they will be social. As the user gets more confident about the capabilities of the bot, human curiosity will start demanding more social conversations. Who hasn’t asked Siri about a general topic?

When the agent gets to that level of interaction with your employees or your customers, you need to worry about a fascinating new topic: what personality do you want for your corporate presence? Do you want it to be casual? Funny? Serious? Companies at this level of maturity design bots with consistent and carefully planned personalities. Technology can help as well; Azure services like Project Personality Chat are providing promising results applying deep learning to chitchat conversations. Do you want your bot to behave like Harry Potter? You can train these services with Harry Potter’s lines from the books.

Omni-Channel Bots

As your organization creates broader use cases with conversational agents, your employees and customers will want to consume them on their channels of choice. Just like your corporate website can be accessed from any browser and device, conversational agents should be available on the platforms where your users are.

Omni-channel bots provide a consistent experience across all the channels, aligned with your corporate identity and services. Keep this in mind as your organization starts to create bots. It is easy to get started by creating just an Alexa skill or a web chatbot, but you will soon fall into the trap of creating a new bot for another channel, like Slack, Skype, or Google Assistant. Soon enough, you will be captive to that model and you will end up providing multiple inconsistent bots, each with an identity that aligns more with the provider of the channel than with your organization.

To avoid this, start with bots that are omni-channel from the get-go. Assume nothing about the channels. Today it may be web and Alexa. Tomorrow it could be WhatsApp and your custom device. Conversational AI platforms like Azure Bot Service or Google’s Dialogflow are designed to be cross-channel and will make that process easier.

This approach will ensure that you have full control over your customer experience. Telefónica recently released Movistar Home, an AI-powered device that replaces the landline devices in its customers’ homes. Because Telefónica’s digital assistant, Aura, is omni-channel, it could easily integrate with the device, owning the entire experience, services and data.

BMW also followed this approach for its in-car experience. Instead of integrating a third-party personal assistant in its cars, the company created its own conversational agent with its own name, voice, and personality, providing a fully branded BMW experience. This end-to-end agent can give you directions, provide information about the car (like engine settings and tire pressure), and respond to voice commands (e.g., for climate control), even with no internet connectivity.

Such branded experiences can also integrate as needed with third-party personal assistants like Google Assistant and Cortana, but you keep control of that integration at all times; you control the full experience and what gets exposed.

From infusing existing applications with AI to creating entirely new experiences powered by AI like those we mentioned here, bringing AI to your technical departments is extremely powerful and will provide the necessary foundation for the next step in the journey for an AI organization: transforming your business units.