Intelligence orchestration is a bit like car racing. A whole team of people build a car, put all the latest technology into it, and get every aerodynamic wing, ballast, gear-ratio, and intake valve set perfectly—they make an awesome machine that can do things no other machine can do.
And then someone needs to get behind the wheel, take it onto the track, and win!
Intelligence orchestrators are those drivers. They take control of the Intelligent System and do what it takes to make it achieve its objectives
. They use the intelligence-creation and management systems to produce the right intelligence at the right time and combine it in the most useful ways. They control the telemetry system, gathering the data needed to make the Intelligent System better. And they deal with all the mistakes and problems, balancing everything so that the Intelligent System is producing the most value it can for users and for your business. For example:
- If the intelligence gets better, the experience can be made more forceful to achieve better outcomes.
- If the problem changes, the intelligence management might need to be optimized to create and deploy intelligence differently or more quickly.
- If the system stops achieving its objectives, someone needs to figure out why and how to use the available systems to adapt.
This isn’t easy.
Building Intelligent Systems and orchestrating them are very different activities. They require very different mindsets. And they are both absolutely critical to achieving success.
This chapter describes what a well-orchestrated intelligence
looks like and introduces the elements of orchestrating an Intelligent System, including the life-cycle, the orchestration environment, mistakes, and abuse.
Properties of a Well-Orchestrated Intelligence
A well-orchestrated Intelligent System will
- Achieve its objectives reliably over time: That is, it will grow, achieving its objectives better and better, until it reaches the point of diminishing return. Then it will stay there, producing value, as the user base, the problem, and the broader world change.
- Have experience , intelligence, and objective in balance: That is, the experience will be as forceful as the intelligence will support so the net benefit to users and business is maximized.
- Have mistakes mitigated effectively: So that high cost mistakes are understood, are detected quickly, and are corrected. And so lower cost mistakes are mitigated and do not put too large a burden on users or on the business.
- Scale effectively over time: So that the cost of maintaining the system in good shape (all the balancing, intelligence creation, mistake mitigations, and so on) scales favorably as the number of users and the complexity of intelligence scales. In particular, this means that machine learning and automation are used effectively.
- Degrade slowly: In that it should not require a bunch of manual effort to hold in a steady state. The effort of orchestration should be focused on finding opportunities, not scrambling to keep the wheels on the bus.
Why Orchestration Is Needed
Orchestrating an intelligence involves tuning it so that it produces the most value possible throughout its life cycle.
And right now you might be saying—wait, wait. I thought machine learning was supposed to tune the system throughout its life cycle. What is this? Some kind of joke?
Unfortunately, no. Artificial intelligence and machine learning will only get you so far. Orchestration is about taking those tools and putting them in the best situations so they can produce value—highlight their strengths, compensate for their weaknesses—and react as things change over time.
Orchestration might be needed because:
- The Intelligent System’s objective changes.
- The Intelligent System’s users change.
- The problem changes.
- The intelligence changes.
- The cost of running the Intelligent System changes.
- Someone tries to abuse your Intelligent System.
One or more of these will almost certainly happen during the life-cycle of your Intelligent System. By learning to identify them and adapt, you can turn these potential problems into opportunities.
Objective Changes
When your objectives change, your approach is going to have to change, too, and that means rebalancing the parts of your Intelligent System. You might want to change your objective when any of the following happen:
- You understand the problem better: As you work on something, you’ll come to understand it better. You might realize that you set the wrong objective to begin with, and want to adapt. Recall that there are usually many layers of objectives. You might have a perfectly fine business objective, but come to understand that the user outcomes you are tracking are not contributing to the objective; or you might realize that the model properties you are optimizing aren’t heading you in the right direction. In these cases you will want to change the objectives and adapt the system as needed to achieve the new goals.
- You have solved the previous problem : You may find that you have achieved what you wanted to achieve. The Intelligent System is running, users are getting good outcomes, your business is doing well. At this point you might want to set a higher goal: and adapt the system accordingly. Or you might want to figure out how to run the system more cheaply (which we will discuss later).
- You realize the previous problem is too hard to solve: Sometimes you have to admit defeat. You might have a system that has been running for a while, and just isn’t getting to where you want it. You could shut down and find something else to do, or you could set an easier objective, change your approach, and build up more slowly.
- You discover some new opportunity : You might come up with a great idea. Something similar to your current Intelligent System, but just a little bit different. Maybe a totally new experience that can leverage the intelligence. Maybe a new type of intelligence you can build off of the existing telemetry. Sometimes chasing a new opportunity will require a new Intelligent System, but sometimes it can be accomplished with the systems you already have—by changing your objective.
Users Change
The user base of an Intelligent System will change over its life cycle, in various ways:
- New users come: And the new users might be very different than your existing users. For example, you might attract a new demographic, more casual users, users who exercise different parts of your Intelligent System, or users who just think differently. As your users change, you will have opportunities to learn and adapt. You will also have more telemetry to use to create intelligence and will have to deal with increasing sampling issues and intelligence-management complexity.
- Usage patterns change: As users learn to use your Intelligent System, they may change the way they interact with it. They will stop exploring how it works and become more focused on getting value out of it. Experienced users will benefit from different interactions than novice users, and so you might want your Intelligent System to change as your users become experts.
- Users change their perception of the experience: Over time users might become fatigued with forceful, frequent experiences. They might start to ignore them. They might get increasingly irritated with them. An experience that worked well for the first month of usage might be totally wrong in the long run—and it might need to change.
- Users leave: And as they do, you will probably cry a little bit. But life will go on. Depending on which users leave, you might have the opportunity to change the balance of intelligence and experience to optimizing for the users you still have. You will also have less telemetry, which might reduce your ability to create intelligence. Automated intelligence, especially, might need to be revisited—for example, to leverage more legacy data instead of relying on newer data.
Problem Changes
Intelligent Systems are for big, hard, open-ended, time-changing problems. These problems—by definition—change over time:
- Time-changing problems are always changing: And so the approaches and decisions you made in the past might not be right for the future. Sometimes a problem might be easy. At other times it might get very hard. Sometimes you will see lots of a particular type of context. At other times you won’t see those contexts at all. As a problem changes, there is almost always opportunity to adapt and achieve better outcomes through orchestration.
- Usage patterns changing : As your users change how they interact with your Intelligent System, the problems you need to solve can become very different. For example, users behave differently in the summer than they do in the winter; they behave differently on weekends than weekdays; they interact with new content differently from how they interact with old content.
- You start to solve the problem : In that the intelligence starts to catch up with a big, hard problem. In this case you might want to revisit the approach. Maybe you will want to start locking in behavior instead of continuing to grow, using more intelligence caches (lookup tables) and investing less in telemetry.
Intelligence Changes
The intelligence of your system will change, when you do any of the following:
- Get more data to build better intelligence: As users interact with your system, the intelligence will have more data, and should get better. When intelligence is better, experiences can become more forceful.
- Try a new approach that makes things a lot better: Data unlocks possibilities. Some of the most powerful machine learning techniques aren’t effective with “small” data, but become viable as users come to your Intelligent System and you get lots and lots of data. These types of changes can unlock all sorts of potential to try new experiences or target more aggressive objectives.
- Try a new approach that makes things worse: But you won’t be perfect. Sometimes intelligence evolutions go backwards. This can be a particular problem with intelligence that is combined in aggressive ways (tightly coupled or hard to comprehend). It can also be a problem when you rely on lots of human intervention in intelligence creation (for example with heuristics)—sometimes the complexity just crosses a tipping point and a brittle intelligence-creation plan can’t keep up. In these cases you might need to back off other parts of the system to give yourself time to get out of the problem.
Costs Change
Big systems will constantly need to balance costs and value. Changes to save money can require compromises in other places. Put another way, you might be able to change your experience or intelligence in ways that save a lot of money, while only reducing value to users or your business by a little. You might try to reduce either or both of these:
- Telemetry costs : By changing the way you sample data. This reduces your ability to measure your system and to create intelligence, but it might be a good trade-off. For example, you might collect less data when intelligence is not growing much, sample less from parts of the problem that are already solved, or remove parts of the contexts that aren’t contributing to useful features (and better models).
- Mistake costs : Managing mistakes can be very expensive. We’ll discuss dealing with mistakes more in Chapter 24. But you might find that you need to make fewer, less expensive mistakes, and you might need to change all sorts of things in the system to do this.
Abuse
Unfortunately, the Internet is full of trolls. Some of them will want to abuse your service because they think that is fun. Most of them will want to abuse your service (and your users) to make money—or to make it harder for you to make money. Left unmitigated, abuse can ruin an Intelligent System, making it such a cesspool of spam and risk that users abandon it.
In order to protect your Intelligent System from abuse you need to understand what abusers want, and then you need to do something to make your system less interesting to potential abusers. Sometimes a small tweak will do it. Sometimes combating abuse will become a major part of running your Intelligent System.
But abuse is a tough thing. Once one abuser figures out how to make a tenth of a penny per interaction off your users they will scale. They will tell their friends. Abuse can show up quickly and violently, so you need to be prepared to identify it and react.
We’ll discuss Adversaries and Abuse in detail in Chapter 25.
The Orchestration Team
Orchestrating an Intelligent System requires
a diverse set of skills. In particular, the team should:
- Be domain experts in the business of the Intelligent System, so they understand the objectives instinctively and know what users want from their interactions with the system.
- Understand experience and have the ability to look at interactions and decide how to make improvements in how intelligence is presented to users—what is easy, what is hard, what users will like, and what they won’t.
- Understand how the implementation works so they know how to trace problems and have some ability to improve the implementation as needed.
- Be able to ask all sorts of questions of data and understand and communicate the answers.
- Know how to do applied machine learning, to be able to create new intelligence and inject it into the system when needed.
- Get satisfaction from making a system execute effectively day in and day out. Not everyone likes this—some people like to invent new things. Orchestration is about excellence in execution—taking some of the most awesome machines in the world, getting them on the race track, and winning.
An orchestration team could consist of a small number of jacks-of-all-trades. Or it could be made up of participants with diverse skills who work well together.
Summary
An Intelligent System needs to be orchestrated throughout its life cycle to succeed. Orchestration is the process of keeping all the parts of the system doing what they are good at, and supporting them where they run into problems. A well-orchestrated Intelligent System will do all of the following
- Achieve its objectives reliably over time.
- Have experience, intelligence, and objective in balance.
- Have mistakes mitigated effectively.
- Scale effectively over time.
- Degrade slowly.
One key activity of orchestration is rebalancing the intelligence and the experience over the Intelligent System’s life-cycle. For example, making the experience more forceful when the intelligence improves; or making it less forceful when the problem gets harder.
Orchestration also involves dealing with mistakes, understanding what they are and when they are occurring, and then managing the system to make them less costly. Reasons orchestration would be needed include these:
- The objective changes.
- Your users change.
- The problem changes.
- Intelligence changes.
- The system needs to get cheaper to run.
Adversaries and abusers often become problems for successful Intelligent Systems. They must be identified and then convinced that your system isn’t worth their time.
For Thought…
After reading this chapter, you should
- Understand what it means to orchestrate an Intelligent System and why it is needed.
- Know what type of team is needed to orchestrate an Intelligent System.
You should be able to answer questions like these:
- Imagine that a system’s intelligence begins to behave poorly. What changes that aren’t intelligence-based could be made to mitigate the issue?
- Pick two of your favorite Intelligent Systems. Which would be harder to orchestrate? Why?
- Describe the types of problems an Intelligent System might have if its user base doubles—and all the new users are from a different country than the previous users.