Thank you for reading this book. I’m glad you got this far. You should now have the foundation to execute on your own Intelligent System project, knowing:
- How to approach an Intelligent System: What it is good for, when you need one, and how to set objectives for one.
- How to design an intelligent experience: One that achieves your objectives and produces data to help grow the intelligence.
- What it takes to implement an Intelligent System: How to execute, manage, and measure intelligence.
- How to create intelligence: Including many approaches, but particularly using machine learning.
- How to orchestrate an Intelligent System: To bring these parts together throughout its life-cycle and achieve the impact you want.
This chapter will review some key concepts and provide a checklist for approaching your own Intelligent System project.
An Intelligent System Checklist
Intelligent Systems are changing the world by closing the loop between users and intelligence. Intelligent Systems solve important problems, delight users, and help organizations achieve success.
To create an Intelligent System there are many things you’ll need to do, and many decisions you’ll need to make. This section collects the key ones of these into one place, with references to the chapters where you can find more detail to help.
When approaching an Intelligent System project, I recommend that you consider the following steps.
Approach the Intelligent System Project
1. Begin by making sure an Intelligent System is right for you.
Intelligent Systems are useful when you expect to change the intelligence of your product many times through its life-cycle. This is commonly needed when your problem is:
- Large.
- Open-ended.
- Time-changing.
- Intrinsically hard.
In addition, an Intelligent System works best when:
- A partial solution is viable and interesting.
- You can use data from users interacting with the system to improve it.
- You can influence a properly scoped and meaningful objective.
- The problem justifies the effort of building the Intelligent System.
Chapter 2 discusses these topics in more details.
2. Define what success looks like for your Intelligent System.
Create consensus about what your Intelligent System should do in a way that communicates the desired outcome, is achievable, and is measurable.
Decide what organizational objectives (like profit or units sold) your Intelligent System will contribute to, and what leading indicators you can use to get quicker feedback that you’re making progress toward them. Then decide what your Intelligent System should optimize for on a day-to-day basis by identifying the user outcomes you want to create and the model properties that will be critical for success.
Come up with a plan for measuring that your Intelligent System is achieving its objectives, which might include: telemetry, waiting for outcomes to become clear, using human judgement, and asking users.
And be prepared to invest over time to keep your goals healthy. Chapter 4 contains more detail on setting objectives for Intelligent Systems
.
Plan for the Intelligent Experience
3. Decide how to present your system’s intelligence to users to achieve your goals.
The goal of an intelligent experience is to:
- Present intelligence to the user.
- Achieve the system’s objectives.
- Minimize intelligence flaws.
- Create data to help improve the intelligence.
To achieve these, you need to balance the experience, by trading off between:
- The forcefulness of the experience.
- The frequency of interactions.
- The value of success.
- The cost of mistakes (both discovering and recovering).
- The quality of the intelligence.
Balancing an intelligent experience is a process, in which an experience changes as intelligence changes. You’ll probably want to leave yourself some room to iterate. Chapter 7 talks about balancing intelligent experiences.
And Chapter 8 talks about specific modes of interaction between users and intelligence that you can use as part of this balancing, including:
- Automation
- Prompting
- Organizing
- Annotating
You’ll almost certainly use some hybrid of these in your Intelligent System, so that your experience is more forceful when the intelligence is certain, and less forceful when the intelligence is unsure.
4. Plan for getting data from your experience.
An ideal intelligent experience will produce data that lets you improve the Intelligent System (and specifically the intelligence itself) over time.
To be useful for improving the Intelligent System, data should:
- Include the context, user action, and outcomes of the interaction.
- Provide good coverage over the parts of your Intelligent System and problem space.
- Represent real interactions with your users.
- Be unbiased.
- Avoid feedback loops.
- Have sufficient scale to be useful for creating intelligence.
It is best—by far—when this data is created implicitly, as a natural byproduct of users interacting with your Intelligent System. But sometimes you need to ask for help understanding how your users perceive the outcomes they receive. Ways to get explicit user feedback include these:
- Letting users provide ratings in the experience (such as thumbs up, thumbs down, or 1-5 stars).
- Accepting user reports of bad outcomes in the experience (such as a “report as spam” button).
- Creating a support tier so users can escalate problems to get help (call a number for help).
- Prompting users to classify some of their outcomes you select.
But keep in mind that users won’t want to spend a lot of time and attention helping you build your product. Focus on implicit data and use explicit data sparingly. See Chapter 9 for more detailed discussion.
5. Plan to verify your intelligent experience.
You’ll need to plan how to:
- Know that the experience is functioning correctly (despite all the mistakes intelligence is making).
- Know that the experience is doing its part in helping your Intelligent System achieve its objectives.
To help with this, you might want to plan for tools to inspect the experience that will be generated for any particular context, and tools to help create and capture contexts where problems are occurring.
You might also want to plan for using different versions of the system’s intelligence in these tools, including the live intelligence; a snapshot of intelligence and some simple or stub intelligence
.
Chapter 10 covers verifying intelligent experiences.
Plan the Intelligent System Implementation
Implementing an Intelligent System requires all the work of implementing a traditional system, plus work to enable the system’s intelligence. These additional components include:
- The intelligence runtime.
- Intelligence management.
- Telemetry to help grow the intelligence.
- An intelligence creation environment.
- Intelligence orchestration tools.
These components provide important capabilities for improving an Intelligent System over its lifetime. Building the right implementation can make it much easier to create intelligence and to orchestrate an Intelligent System.
Chapter 11 is an overview of the components of an Intelligent System implementation.
6. Decide where your intelligence will live.
Intelligence can be any of the following:
- Static intelligence in the product.
- Client-side intelligence.
- Server-centric intelligence.
- Back-end (cached) intelligence.
Or probably some hybrid of these. You should select where your intelligence
will live based on the effect it will have on:
- The latency in updating your intelligence.
- The latency in executing your intelligence.
- The cost of operating your Intelligent System.
- Any requirement for offline operation.
Chapter 13 explores issues around where intelligence can live; it will have the detail you need to make good decisions for your application.
7. Design your intelligence runtime.
The intelligence runtime is where the intelligence is executed and experience is updated. An intelligence runtime must do all of the following:
- Gather the context you need so your intelligence can make good decisions.
- Extract features from context in a way that is safe, but that supports innovation in intelligence creation.
- Deal with model files and updating them.
- Execute models on contexts (and their features).
- Take the results and predictions produced by the execution and light up the intelligent experience.
Chapter 12 discusses intelligent runtimes in detail.
8. Plan for intelligence
management
.
Intelligence management is the system that lets you add new intelligence to the system and turn it on for users in a safe way. This includes:
- Ingesting intelligence into the Intelligent System.
- Sanity-checking intelligence to make sure it is not obviously harmful or corrupted.
- Combining intelligence when there are multiple intelligence sources, including making sure they are all in sync. Chapter 21 catalogs ways of combining intelligence.
- Lighting up intelligence for users in a controlled fashion. That can include silent intelligence, controlled rollout, flighting (A/B testing), and support for reversions.
Chapter 14 discusses methods for intelligence management.
9. Plan for
telemetry
.
Telemetry is a critical link to connect usage to intelligence, so an Intelligent System can improve over time. Telemetry is used to make sure the Intelligent System is working, understand the outcomes users are getting, and gather data to grow the intelligence.
There are more things in a system than you could possibly measure, so you need to make some decisions about how to control the scale of what you collect in telemetry. But you also need to provide tools to allow intelligence creators and orchestrators to adapt as the problem and Intelligent System change over time. These tools include:
- Sampling what is collected in telemetry, and turning up and down the sampling rate over time.
- Summarizing interactions in ways that allow the right information to be collected efficiently.
- Supporting flexible targeting of what contexts and users to sample telemetry from.
You can learn all about telemetry in Chapter 15
.
Get Ready to Create Intelligence
Intelligence is the part of the Intelligent System that makes the hard decisions, mapping from contexts to predictions, powering the intelligent experience, creating value for users and for your organization.
10. Prepare to evaluate intelligence.
With intelligence, evaluation is creation. You should be able to easily measure any intelligence to know:
- How well it generalizes.
- The types of mistakes it makes.
- Its mistake distribution.
To do this you will need evaluation data, hopefully from telemetry. This should be sufficient data to get statistically meaningful evaluations. It should also be independent from training data (which is sometimes tricky) and it should allow you to understand how your intelligence performs on important sub-populations.
Chapter 19 (the longest chapter in the book) discusses evaluating intelligence—it’s that important.
11. Decide how you will organize your intelligence.
Organizing intelligence creation
is important when working with large Intelligent Systems where you need to:
- Have multiple people collaboratively creating intelligence.
- Clean up mistakes that intelligence is making.
- Solve the easy part the easy way and focus more complex techniques on the hard parts of the problem.
- Incorporate legacy intelligence or intelligence acquired from external sources.
Here are some common ways to organize intelligence creation:
- Decouple feature engineering.
- Perform multiple model searches.
- Chase mistakes (which is probably the worst).
- Use meta-models.
- Perform model sequencing.
- Partition contexts.
- Use overrides.
Chapter 21 discusses all of these ways to organize intelligence creation
.
12. Set up your intelligence creation process.
Set it up to use all the tools and data available to create and improve intelligence. This involves:
- Choosing the representation for your intelligence (as described in Chapter 17).
- Creating simple heuristics as a baseline.
- Using machine learning to produce more sophisticated intelligence when needed (as described in Chapter 20).
- Helping understand the tradeoffs of various implementation options.
- And assessing and iterating, on and on, as long as the intelligence needs to improve.
Chapter 18 gives an example of the intelligence-creation process.
Orchestrate Your Intelligent System
You need to take control of all the tools discussed in this book and “drive the race car” to achieve success day in and day out for as long as the Intelligent System is relevant. A well-orchestrated Intelligent System will:
- Achieve its objectives reliably over time.
- Have experience, intelligence, and objective in balance.
- Have mistakes mitigated effectively.
- Scale effectively over time.
- Degrade slowly.
Orchestration is important for Intelligent Systems when:
- Objectives change.
- Users change.
- The problem changes.
- Intelligence changes.
- The system needs to be more efficient.
- Abuse happens.
Chapter 22 introduces orchestration in detail.
13. Plan for orchestration throughout your Intelligent System’s lifecycle.
This planning includes how to get started and how to invest over time. In order to orchestrate an Intelligent System, you’ll need to
- Monitor the success criteria.
- Inspect interactions.
- Balance the experience.
- Override intelligence.
- Create intelligence.
And more. Chapter 23 describes the activities of orchestration and how to decide when and where to invest.
14. Prepare to identify and deal with mistakes.
Because with intelligence, mistakes will come
.
You will need to identify why your Intelligent System is getting mistakes, including:
- System outages.
- Model outages.
- Intelligence errors.
- Intelligence degradation.
And when you do identify a source of mistakes you must decide how to react:
- Invest in intelligence.
- Balance the experience.
- Retrain intelligence differently.
- Implement guardrails.
- Override errors.
By embracing errors, finding them proactively, and mitigating the damage they cause, you can free up lots of space for intelligence to shine. Chapter 24 explores mistakes.
15. Get ready for abuse.
If someone can figure out how to make a buck by abusing your users or your Intelligent System, they will.
You should be prepared to identify abuse quickly if it starts to occur, by looking for:
- Groups of users with degenerate usage.
- Contexts that see spikes in activity.
- Contexts that see drastic changes in outcomes.
- Patterns in complaints and problem reports.
If abuse does happen, keep in mind that it is a business. You don’t need to block abuse. You just need to make the abuser think they can’t make money and they’ll stop. Here are some ways you can do this:
- Add costs to users of your Intelligent System.
- Become less interesting to abusers.
- Model with an adversary.
- Remove the adversary from the loop.
Chapter 25 is about adversaries and abuse.
16. Enjoy the ride.
And that’s not the end, but the beginning. Intelligent Systems can live for many years. Attract users. Achieve objectives. Create intelligence. Orchestrate
the thing—and win!
Summary
Thank you for reading this book. You should now have the knowledge to approach an Intelligent System project, with confidence.
I’m excited to see what you’ll accomplish!
For Thought…
After reading this chapter, you should:
- Feel great about the work you put into reading this book.
- Be prepared to participate in Intelligent System projects, confident you have the knowledge to contribute.
You should be able to answer questions like these:
- What type of Intelligent System would you like to build?
- How will it change the world?
- What’s stopping you? Get going!