This chapter discusses some of the common activities that contribute to success at intelligence orchestration. These are examples of the types of things that can help get the most out of an Intelligent System, and include the following:
- Monitoring the Success Criteria: To know if the system is achieving its goal and if it is getting better or worse.
- Inspecting Interactions: To experience contexts as the user experienced them and see the outcomes the user achieved.
- Balancing the Experience: To make it more or less forceful, maybe changing the type of interactions, or maybe changing the prominence and frequency of the interactions.
- Overriding the Intelligence : To correct for infrequent bad mistakes, or to help optimize common contexts.
- Creating New Intelligence : Managing intelligence and producing new models to deal with emergent issues as the problem, the users, or the scope of data changes.
Intelligent Systems can benefit from all of these activities and more. Some of them can be built as part of the implementation—for example, a metrics dashboard
that shows progress on the overall objectives. Some of them can be handled in a more ad hoc fashion—for example, querying raw telemetry sources
to piece together interactions that are causing problems.
This chapter discusses orchestration activities in turn and gives some ideas of how you can invest in each of the areas (if you choose to), along with some criteria for deciding how much to invest.
Monitor the Success Criteria
You must be able to tell if the system is achieving its goals. This involves taking telemetry, correcting for any sampling and outages, and producing
an answer that all participants can understand—maybe a few numbers, maybe a graph. Recall that this can involve any of the following:
- Business objectives and leading indicators, particularly showing how they vary between users who heavily use the intelligent part of the system compared to those who don’t.
- User outcomes, indicating how effective the Intelligent System is at helping users achieve the outcomes you want.
- Model properties, indicating how often the intelligence is correct or incorrect (no matter what outcomes the user ended up achieving).
Monitoring success criteria is critical. Every functional Intelligent System must have people who know how well it is doing, every day, and who care that it is achieving its goals.
Levels of investment for monitoring
success criteria include:
- Ad Hoc: Where a few people have the skill to measure the systems objectives and produce answers, maybe by doing some cross checking of current telemetry settings, tweaking a script, and running it, copying into a presentation form (like a chart), and distributing.
- Tool-Based: Where anyone on the team can run a tool that creates and then runs the correct query and outputs the answer in a suitable presentation format.
- Automated: Where the system automatically runs the queries on a regular basis and archives the results somewhere that all participants can find them.
- Alert-Based: where the system automates the metrics, but also monitors them and provides alerts to participants when something happens, including
- A major degradation: Where the metrics are far worse than they were at the previous measurement. For example, the number of errors went up 10% since yesterday.
- A significant, sustained erosion: Where the metrics have trended gradually downward long enough to cross some threshold. For example, the fraction of users getting good outcomes has gone down 10% over the course of the last month.
- Population Tracking: Where the system tracks metrics for important sub-populations as well as the overall population. This might include people from a particular location, with a particular usage pattern, with a particular demographic, and so on.
Criterion for investing in monitoring success criteria:
- Almost every Intelligent System should implement alert-based monitoring for success criteria and for critical factors that contribute to success criteria.
Inspect Interactions
Inspecting an interaction involves gathering all the telemetry related to a particular user interaction
and being able to see the interaction end-to-end. This includes the user’s context at the time of the interaction, the version of the intelligence running; the answer given from the intelligence, the experience that resulted from this intelligence answer, how the user interacted with the experience, and what outcome the user eventually got.
This is important for debugging problems, for tracking mistakes, for understanding how users are perceiving the Intelligent System, and for getting intuition about user experience options.
Levels of investment for inspecting interactions
include
- Ad Hoc: Where a few people have the skill to find all the parts of an interaction and understand how they relate. They may have to do some detective work to identify a specific interaction (or interactions with a certain property) and, based on sampling, may or may not be able to find any particular interaction.
- Tool-based: Where anyone on the team can identify an interaction (maybe by user and time) see the relevant telemetry. The tool may also support querying for specific types of interactions (maybe where the intelligence gave a particular answer and the user got a particular outcome) and inspect a sample of them.
- Browser-based: Where anyone on the team can find interactions as with the tool, but then experience the interaction as the user would have, seeing the experiences that the user saw, the buttons they clicked, the outcome they got, and so on.
- Getting Data from Users: Where orchestrators have the opportunity to flag specific types of interactions and ask users questions about their experience. For example, this can be done by specifying some contexts and then surveying a small fraction of users who hit that context. These surveys might ask the user to rate their experience, answer a question or two, or indicate if they had a good or bad outcome.
Criteria for investing in inspecting interactions:
- If you need a broad set of people to be able to understand interactions. For example, so experience creators and business stakeholders can understand how users are experiencing the Intelligent System.
- If you need to build support capabilities to help deal with mistakes. Tools can allow humans who aren’t experts in the implementation to participate.
- If your system has high cost mistakes and you expect a major part of orchestration will involve regular mitigation, then you will want to invest in inspecting interactions.
Balance the Experience
As the problem, the user base, and the intelligence change, they create opportunities
to optimize the experience. For example, when the intelligence is new—and poor quality—the experience might be very passive. Maybe it doesn’t show up very often. Maybe it shows up in ways that are easy for users to ignore. But over time the intelligence will get better, and users might have more positive outcomes with more forceful experiences.
Levels of investment for balancing the experience
include:
- Ad Hoc: Where changes to the experience are made by changing code and redeploying the software.
- Parameter Updates: Where the orchestrators can change parameters that affect the experience and push these parameters out relatively cheaply (maybe as intelligence is updated). Parameters might include:
- The frequency of interaction.
- The color or size of a prompt.
- The text to use in the experience.
- The threshold for automating an action.
- And so on.
- Experience Alternatives: Where multiple experiences are created and orchestrators have the ability to switch between them (using something similar to a parameter). For example, maybe you create one version of the experience with a subtle prompt, one with a prominent prompt, and one that automates an action. Then the orchestrator can switch between these to achieve objectives over time.
- Experience Language: Where orchestrators can author and deploy experiences out of band of code changes, similar to separating intelligence from implementation. This might be accomplished by specifying experiences in scripts that clients download and execute. Or it might be more curtailed, so that non-engineers can safely make changes; for example an experience mark-up language with a restricted runtime.
Criteria for investing in balancing the experience:
- If your orchestration team does not have engineering resources , then you probably want to create some controls for the experience that are not engineering-based.
- If it takes a long time to deploy new client code to your customers, you might want to invest in ways to control experience through parameters and data-file updates.
- If you think you’ll be changing your experience many times during the life-cycle of your Intelligent System, you might choose to invest in making it easy.
- If none of those are true—you do have engineering resources, you can easily deploy code, and you don’t expect too many changes—it will almost certainly be cheaper to take an ad hoc approach to experience updates.
Override Intelligence
Overriding intelligence involves identifying a few contexts and hand-coding the answer the intelligence
should provide for those contexts. You might want to do this to correct costly (or embarrassing) mistakes. You might want to do this to optimize a few common contexts. You might want to do this to support some business goal (like promote a piece of content in your system). You probably don’t want to override intelligence too much—but when you need to, you are going to really need to.
Levels of investment for overriding intelligence
include:
- Ad Hoc: Where you rely on intelligence creators to hack the learning process to try to achieve specific outcomes (which is very hard) or on engineers to hard-code the desired overrides into code and deploy them to customers.
- As an Intelligence Feed: Where overrides are treated as a very simple intelligence source that is deployed with the system’s other intelligence and has the highest priority in the runtime. Perhaps represented using a data file in some simple format that maps contexts to outcomes.
- Tool-based: Where you treat overrides as an intelligence feed but create tooling to support orchestrators. Tools should include functions like these:
- Ensuring the contexts to override are specified correctly.
- Giving feedback on the prevalence of the specified contexts and the current intelligence responses for those contexts.
- Giving feedback on existing overrides, including how many there are and how many users they are impacting.
- Tracking who is doing overrides and how good/bad they turn out to be.
- Managing expiration of overrides.
- Browser-Based: Where the tools are connected into a support suite, including ways to find interactions, view them, and override them all in one place.
Criteria for investing in overriding intelligence:
- If your system can make very costly mistakes you will almost certainly want to invest in overriding intelligence, possibly adding tracking and work-flow management around the tools discussed here. We’ll discuss this in more detail in the next chapter, on dealing with mistakes.
- Overriding intelligence can also be a very helpful buffer for intelligence creators. The last thing you want is for lots of participants in the Intelligent System to use the product, find zillions of little problems, and file them as bugs against the intelligence creation process. Intelligence is going to make mistakes. Beating up the creators for every flaw will not help. Having the ability to override a few of the more important problems can make everyone feel better.
Create Intelligence
An important part of orchestration is controlling
and creating the intelligence that drives your Intelligent System. This section recaps and summarizes some of the investments that can help intelligence grow and create impact.
Investments in creating intelligence
might include:
- Intelligent Management: This will include the following:
- Controlling when and how intelligence is updated and deployed.
- Controlling what intelligence lives where.
- Adding new sources of intelligence to the system.
- Controlling how intelligence is combined.
- Automated Machine-Learning Implementations: These produce new intelligence on a regular basis with little or no oversight. These systems may provide controls to change the way intelligence is produced, including the following:
- How much data to use.
- Which data to use.
- How often to run.
- How much time to spend searching for good models.
- Any other parameters of the modeling process that are tunable.
- Intelligence Creation Environments: Where the telemetry that captures context and outcome are gathered and made available for machine learning, including trying new machine-learning algorithms, new feature engineering, and incorporating new insights. These can produce ad hoc new intelligence, or find ways to improve existing automated learning systems.
- Support for Feature Engineering: Where intelligence creators are able to try new features very easily, and if they find ones that work are able to deploy them to customers quickly and safely.
- Telemetry Systems: Where the orchestrator can specify what data to collect and how much of it to collect.
Criteria for investing in creating intelligence:
Intelligence creation is kind of important for Intelligent Systems (it says so right in the name). You will probably end up investing in multiple systems to support this process.
The most effective tools tend to:
- Help prevent errors.
- Automate mundane tasks.
- Simplify multi-step processes.
- Leave some simple audit trail.
When building tools to support intelligence creation, avoid the temptation to mess with the core intelligence creation tools
(like machine-learning systems or runtimes). These are generally standard and innovating with them is unlikely to be core to your value proposition.
Summary
This book has introduced many tools that are important for running an Intelligent System. This chapter summarized some of the key ones, including some criteria and approaches for investing in them. These include:
- Monitoring the success criteria
- Inspecting interactions
- Balancing the experience
- Overriding intelligence
- Creating intelligence
You can build a successful Intelligent System without investing much in these areas during implementation. But you’ll probably need to do all of these activities during the system’s life-cycle. You can decide how labor intensive you want to be, and how much you want to invest in tools.
And keep in mind that having good tools allows you to change your staffing over time from the system’s creators (who often like to work on new exciting things) to skilled orchestrators (who need to be generalists with drive for sustained excellence). These are very different skill sets.
Often the orchestration environment will evolve as the Intelligent System does, with small investments being made where they add the most value over an extended period until no further investments make sense.
For Thought…
After reading this chapter, you should:
- Know the common tools needed to keep an Intelligent System healthy.
- Understand ways to start with low investment and scale the investment over time.
- Have some understanding of when the various tools will help, and which are most important for your Intelligent System.
You should be able to answer questions like these:
Imagine an important Intelligent System that exists today.
- Which area do you think it spends the most orchestration time: monitoring success, inspecting interactions, balancing the experience, overriding intelligence, or creating intelligence?
- What would you propose to reduce the cost in that area?