One of the key challenges in intelligence creation is to produce intelligence that works well on things you don’t know about at the time you create the intelligence.

Consider a student who reads the course textbook and memorizes every fact. That’s good. The student would be very accurate at parroting back the things that were in the textbook. But now imagine the teacher creates a test that doesn’t ask the student to parrot back fact from the textbook. Instead, the teacher wants the student to demonstrate they understood the concepts from the textbook and apply them in a new setting. If the student developed a good mental model about the topic, they might pass this test. If the student has the wrong mental model about the topic, or has no mental model at all (and has just memorized facts), they won't do so well at applying the knowledge in a new setting. This is the same as the intelligence in an Intelligent System—it must generalize to new situations.

Let’s look at an example. Consider building intelligence that examines books and classifies them by genre—sci-fi, romance, technical, thriller, historical fiction, that kind of thing.

You gather 1,000 books , hand-label them with genres, and set about creating intelligence. The goal is to be able to take a new book (one that isn't part of the 1,000) and accurately predict its genre.

What if you built this intelligence by memorizing information about the authors? You might look at your 1,000 books, find that they were written by 815 different authors, and make a list like this:

When you get a new book, you look up its author in this list. If the author is there, return the genre. If the author isn't there—well, you're stuck. This model doesn't understand the concept of “genre” it just memorized some facts and won't generalize to authors it doesn't know about (and it will get pretty confused by authors who write in two different genres).

When evaluating the accuracy of intelligence, it is important to test how well it generalizes. Make sure you put the intelligence in situations it hasn't seen before and measure how well it adapts.

Intelligences can make many types of mistakes and some mistakes cause more trouble than others. We’ve discussed the concept of false positive and false negative in Chapter 6 when we discussed intelligent experiences, but let’s review (see Figure 19-1). When predicting classifications, an intelligence can make mistakes that

For example, suppose the intelligence does one of these things:

In order to be useful, an intelligence must make the right types of mistakes to complement its Intelligent System. For example, consider the intelligence that examines web pages to determine if they are funny or not. Imagine I told you I had an intelligence that was 99% accurate. Show this intelligence some new web page (one it has never seen before), the intelligence makes a prediction (funny or not), and 99% of the time the prediction is correct. That’s great. Very accurate generalization. This intelligence should be useful in our funny-webpage-detector Intelligent System.

But what if it turns out that most web pages aren’t funny—99% of web pages, to be precise. In that case, an intelligence could predict “not funny” 100% of the time and still be 99% accurate. On not-funny web pages it is 100% accurate. On funny web pages it is 0% accurate. And overall that adds up to 99% accuracy. And it also adds up to—completely useless.

One measure for trading off between these types of errors is to talk about false positive rate vs false negative rate. For the funny-page-finder, a “positive” is a web page that is actually funny. A “negative” is a web page that is not funny. (Actually, you can define a positive either way—be careful to define it clearly or other people on the project might define it differently and everyone will be confused.) So:

Using this terminology, the brain-dead always-not-funny intelligence would have a 0% false positive rate (which is great) and a 100% false negative rate (which is useless).

Another very common way to talk about these mistake-trade-offs is by talking about a model’s precision and its recall.

Using this terminology, the brain-dead always-not-funny intelligence would have an undefined precision (because it never says positive and you can’t divide by zero, not even with machine learning) and a 0% recall (because it says positive on 0% of the positive pages).

An effective intelligence must balance the types of mistakes it makes appropriately to support to the needs of the Intelligent System.

In order to be effective, an intelligence must work reasonably well for all users. That is, it cannot focus its mistakes into specific sub-populations. Consider:

These types of mistakes can be embarrassing . They can lead to unhappy users, bad reviews. It’s possible to have an intelligence that generalizes well, that makes a good balance of the various types of mistakes, and that is totally unusable because it focuses mistakes on specific users (or in specific contexts).

And finding this type of problem isn't easy. There are so many potential sub-populations, it can be difficult or impossible to enumerate all the ways poorly-distributed mistakes can cause problems for an Intelligent System.

Regressions return numbers. You might want to know what fraction of time they get the “right” number. But it is almost always more useful to know how close the predicted answers are to the right answers than to know how often the answers are exactly right.

The most common way to do this is to calculate the Mean Squared Error (MSE). That is, take the answer the intelligence gives, subtract it from the correct answer, and square the result. Then take the average of this across the contexts that are relevant to your measurement.

When the MSE is small, the intelligence is usually giving answers that are close to the correct answer. When the MSE is large, the intelligence is usually giving answers that are far from the correct answer.

A probability is a number from 0 – 1.0. One way to evaluate probabilities is to use a threshold to convert them to classifications and then evaluate them as classifications.

Want to know if a book is a romance? Ask an intelligence for the probability that it is a romance. If the probability is above a threshold, say, 0.3 (30%), then call the book a romance; otherwise call it something else.

Using a high threshold for converting the probability into a classification, like 0.99 (99%), generally results in higher precision but lower recall—you only call a book a romance if the intelligence is super-certain.

Using a lower threshold for turning the probability into a classification, like 0.01 (1%), generally results in lower precision, but higher recall—you call just about any book a romance, unless the intelligence is super-certain it isn’t a romance.

We’ll discuss this concept of thresholding further in a little while when we talk about operating points and comparing intelligences.

Rankings order content based on how relevant it is to a context. For example, given a user’s history , what flavor of soda are they most likely to order next? The ranking intelligence will place all the possible flavors in order, for example:

So how do we know if this is right?

One simple way to evaluate this is to imagine the intelligent experience. Say the intelligent soda machine can show 3 sodas on its display. The ranking is good if the user’s actual selection is in the top 3, and it is not good if the user’s actual selection isn’t in the top 3.

You can consider the top 3, the top 1, the top 10—whatever makes the most sense for your Intelligent System .

So one simple way to evaluate a ranking is as the percent of time the user’s selection is among the top K answers.

The data used to evaluate intelligence must be completely separate from the data used to create the intelligence.

Imagine this. You come up with an idea for some heuristic intelligence, implement it, evaluate it on some evaluation data, and find the precision is 54%. At this point you’re fine. The intelligence is probably around 54% precise (plus or minus based on statistical properties because of sample size, and so on), and if you deploy it to users that’s probably about what they’ll see.

But now you look at some of the mistakes your intelligence is making on the evaluation data. You notice a pattern, so you change your intelligence, improve it. Then you evaluate the intelligence on the same test data and find the precision is now 66%.

At this point you are no longer fine. You have looked at the evaluation data and changed your intelligence because of what you found. At this point it really is hard to say how precise your intelligence will be when you deploy it to users; almost certainly less than the 66% you saw in your second evaluation. Possibly even worse than your initial 54%.

This is because you’ve cheated. You looked at the answers to the test as you built the intelligence. You tuned your intelligence to the part of the problem you can see. This is bad.

One common approach to avoiding this is to create a separate testing set for evaluation. A test set is created by randomly splitting your available data into two sets. One for creating and tweaking intelligence, the other for evaluation.

Now as you are tweaking and tuning your intelligence you don’t look at the test set—not even a little. You tweak all you want on the training set. When you think you have it done, you evaluate on the test set to get an unbiased evaluation of your work.

The data used to evaluate intelligence should be completely separate from the data used to produce the intelligence. I know—this is exactly the same sentence I used to start the previous section. It’s just that important.

One key assumption made by machine learning is that each piece of data is independent of the others. That is, take any two pieces of data (two contexts, each with their own outcomes). As long as you create intelligence on one, and evaluate it on the other, there is no way you can be cheating—those two pieces of data are independent. In practice this is not the case. Consider which of the following are more (or less) independent:

Clearly some of these pieces of data are not as independent as others, and using data with these types of strong relationships to both create and evaluate intelligence will lead to inaccurate evaluations. Randomly splitting data into training and testing sets does not always work in practice.

Two approaches to achieving independence in practice are to petition your data by time or by identity .

Most Intelligent Systems will partition by time and by at least one identity when selecting data to use for evaluation.

Sometimes it is critical that an intelligence does not systematically fail for critical sub-populations (gender, age, ethnicity, types of content, location, and so on.).

For example, imagine a speech recognition system that needs to work well for all English speakers. Over time users complain that it isn’t working well for them. Upon investigation you discover many of the problems are focused in Hawaii—da pidgin stay too hard for fix, eh brah?

Ahem…

The problem could be bad enough that the product cannot sell in Hawaii—a major market. Something needs to be done!

To solve a problem, it must be measured (remember, verification first). So we need to update the evaluation procedure to measure accuracy specifically on the problematic sub-population. Every time the system evaluates a potential intelligence, it evaluates it in two ways:

This evaluation procedure might find that the precision is 95% in general, but 75% for the members of the pidgin-speaking sub-population. And that is a pretty big discrepancy.

Evaluating accuracy on sub-populations presents some complications. First is identifying if an interaction is part of the sub-population. A new piece of telemetry arrives, including a context and an outcome. But now you need some extra information—you need to know if the context (for example the audio clip in the telemetry) is for a pidgin-Hawaiian speaker or not. Some approaches to this include:

A second complication to evaluating accuracy for sub-populations is in getting enough evaluation data for each sub-population. If the sub-population is small, a random sample of evaluation data might contain just a few examples of it. Two ways to deal with this are:

So how much data do you need to evaluate an intelligence ? It depends—of course.

Recall that statistics can express how certain an answer is, for example: the precision of my intelligence is 92% plus or minus 4% (which means it is probably between 88% and 96%).

So how much data you need depends on how certain you need to be.

Assuming your data is very independent, the problem isn’t changing too fast, and you aren’t trying to optimize the last 0.1% out of a very hard problem (like speech recognition):

A starting point for choosing how much data to use to evaluate intelligence might be this:

But you'll have to develop your own intuition in your setting (or use some statistics, if that is the way you like to think).

One tool to help evaluate intelligences is to select an operating point . That is, a precision point or a recall point that works well with your intelligent experience. Every intelligence must hit the operating point, and then the one that performs best there is used. For example:

This reduces the number of variables by choosing one of the types of mistakes an intelligence might make and setting a target. We need an intelligence that is 90% precise to support our experience—now, Mr. or Mrs. Intelligence creator, go and produce the best recall you can at that precision.

When comparing two intelligences it’s often convenient to compare them at an operating point. The one that is more precise at the target recall (or has higher recall at the target precision) is better.

Easy.

But sometimes operating points change. For example, maybe the experience needs to become more forceful and the operating point needs to move to a higher precision to support the change. Intelligences can be evaluated across a range of operating points. For example, by varying the threshold used to turn a probability into a classification, an intelligence might have these values:

And on and on. Note that any intelligence that can produce a probability or a score can be used this way (including many, many machine learning approaches).

When comparing two intelligences that can make trade-offs between mistake types, you can compare them at any operating point. For example:

One model might be better at some types of trade-offs and worse at others. For example, one model might be better when you need high precision, but a second model (built using totally different approaches) might be better when you need high recall.

It is sometimes helpful to visualize the various trade-offs an intelligence can make. A precision-recall curve (PR curve) is a plot of all the possible trade-offs a model can make. On the x-axis is every possible recall (from 0% to 100%) and on the y-axis is the precision the model can achieve at the indicated recall.

By plotting two models on a single PR curve it is easy to see which is better in various ranges of operating points.

A similar concept is called a receiver operating characteristic curve (ROC curve) . An ROC curve would have false positive rate on the x-axis and true positive rate on the y-axis.

Statistics (for example, those used to represent model quality with a precision and recall) are nice, they are neat; they summarize lots of things into simple little numbers that go up or down. They are critical to creating intelligence. But they can hide all sorts of problems. Every so often you need to go look at the data.

One useful technique is to take a random sample of 100 contexts where the intelligence was wrong and look at them. When looking at the mistakes, consider:

In addition to inspecting random mistakes , it also helps to look at places where the intelligence was most-certain of the answer—but was wrong (a false positive where the model said it was 100% sure it was a positive, or a false negative where the model said the probability of positive was 0%). These places will often lead to bugs in the implementation or to flaws in the intelligence-creation process.

While looking at mistakes, imagine the user’s experience. Visualize them coming to the context. Put yourself in their shoes as they encounter the mistake. What are they thinking? What is the chance they will notice the problem? What will it cost if they don’t notice it? What will they have to do to recover if they do notice it?

This is also a good time to think of the aggregate experience the user will have. For example:

Putting yourself in your users' shoes will help you know whether the intelligence is good enough or needs more work. It can also help you come up with ideas for improving the intelligent experience.

And imagine the worst thing that could happen. What types of mistakes could the system make that would really hurt its users or your business? For example:

These are a little bit silly, but the point is—be creative. Find really bad things for your users before they have to suffer through them for you, and use your discoveries to make better intelligence, or to influence the rest of the system (the experience and the implementation) to do better.