Now that you have heard about the many ways human behavior can be biased on the one hand and how complex the development of an algorithm is on the other hand, you have probably started to appreciate in how many ways algorithmic biases can arise. In this second part of the book, we will examine in greater detail the different ways algorithmic biases can be introduced.
In this chapter, we will grab the bull by its horns and tackle first the most difficult type of algorithmic bias: the algorithmic bias caused by biased behaviors in the real world. It is the most difficult type of bias because in a sense this algorithmic bias is “correct”—the algorithm does what statistically it is supposed to do: it correctly mirrors and represents the real world. We therefore are not grappling just with technical issues but with deep philosophical and ethical problems. Importantly, we will conclude that algorithms can be as much part of the solution as they can be part of the problem.
I earlier stated that statistical algorithms are a way to remove bias from human judgment. However, algorithms sometimes fail to deliver on this promise. The reason is that real-world biases sometimes create facts, and these facts now are the reality that shapes the algorithm’s logic, thus perpetuating a bias.
To illustrate, let’s examine this fictitious example of confirmation bias: if the police are more likely to frisk passers-by with green skin (a.k.a. Martians) than passers-by with grey skin (a.k.a. Zeta Reticulans) even though both populations have exactly the same share of people carrying illicit drugs (e.g., 5%), then the police are likely to finish the day with a disproportionately large number of Martians caught with illicit drugs. Assume you want to overcome this biased behavior by programming a device that scores each person passing a police officer and beeps when the algorithm suggests frisking the person because the algorithm detects a high probability of carrying drugs. If both groups have the same propensity to carry illicit drugs, you would expect your device to also seek out members of both groups with the same frequency.
In order to construct the algorithm, you might collect data on all persons frisked by the police in the last 12 months. For each person, you collect a lot of different attributes as well as the results of the frisking—a binary flag if illicit drugs have been found. Based on the information above, you might expect to find that both 5% of Martians and 5% of Zeta Reticulans have carried illicit drugs—or maybe you expect a 50% success rate because you believe that the police are really good at spotting criminals. However, you find that 20% of Martian and 10% of Zeta Reticulans in the sample had carried drugs. What happened?
First of all, you can assume that the police somehow figured out a way to target people that are more likely to carry illicit drugs than the average person because their success ratio is significantly higher than 5%—the police therefore have some real insight that you would want to capture in your algorithm. However, why is the propensity of carrying illicit drug doubled for Martians?
It is possible that for the police it is somehow easier to spot illicit drugs on Martians than on Zeta Reticulans—maybe Martians prefer tighter-fitting clothes that make it easier to spot packages in cargo pockets. However, it is also possible that confirmation bias subtly changes the police’s behavior. If the police expect every Martian frisked to carry illicit drugs, they will not only be very diligent but if the first frisking doesn’t reveal any results, they might frisk once more, looking for hidden pockets. On the other hand, if many Zeta Reticulans are frisked only pro forma in order so as not to give an impression of being overly biased, a police officer may just frisk a couple of pockets and then let the Zeta Reticulan go. In other words, confirmation bias has compromised your data because some Zeta Reticulans carrying illicit drugs go undetected because of a lighter frisking practice.
This behavior occurs in many contexts. It also happens in hiring. Due to the anchoring effect, interviewers often have formed an opinion of an interviewee within the first couple of seconds of an interview, maybe as soon as the candidate walks into the door. This opinion will now inform the confirmation bias . If the candidate gives a mumbled answer to a quiz question, the interviewer’s brain might (subconsciously) “hear” the correct answer if it has already decided that this is a stellar candidate—and it may interpret the mumbling as proof that the interviewee isn’t a stellar candidate if the brain already has rejected the candidate.
Even more crazily, however, the interviewee might detect the interviewer’s biased judgment through body language and speech1—and subconsciously adjust to this. Interviewees who feel that the interviewer has a low opinion of them actually do perform worse. If you recorded the interview by video and developed the most advanced deep learning model on the planet to objectively score the interviewee’s performance, the score would still indicate a performance concordant with the interviewer’s bias even though it is entirely a psychological artifact.
This illustrates a major dilemma: human biases to an extent shape the world—and where biases have translated into factual differences in the behavior or appearance of intrinsically equal subjects, algorithms lose their power to correct the picture simply by applying statistical techniques to data.
Let’s push our thought experiment a bit further: realizing that the police uses different frisking protocols for Martians and Zeta Reticulans, you could work with a group of officers to run an experiment where the police agree to follow exactly the same approach for every person they frisk—maybe even use a portable body scanner (like the one used in airports) to complement the manual frisking. To your great surprise, even though now the rates of carrying illicit drugs have come a bit closer to each other, Martians still are found to carry drugs more frequently. You start to doubt your hypothesis. What if Martians really have more criminal energy?
In reality, you might be dealing with a much deeper issue. Years of bad press (the Zeta Evening Standard frequently runs headlines like “Another 15 Martian drug dealers arrested,” quietly ignoring the 7 Zeta Reticulans indicted the same day) may have influenced public opinion, and Martians therefore may struggle more than Zeta Reticulans to find jobs. As a result, more of them might end up dealing in drugs.
Biases therefore often have a “winner takes all” effect—an initial bias starts to tweak reality, and the effect becomes self-reinforcing and even self-fulfilling. Replacing human judgment with an algorithm at this point often enough can cement the status quo. If you developed a world class algorithm with the data you have collected, you are prone to achieve a much higher success rate than the judgmental approach of the police—thanks to your algorithm, the police may end up frisking overall less people but find 80% of them to carry illicit drugs, hence significantly increasing the number of drug dealers apprehended. Yet the vast majority of people flagged by your algorithm may be Martians, and following several lurid articles about the police’s work, one day the Zeta Evening Standard will publish the first Letter to the Editor openly musing whether the city should ban Martians.
Repeating the wrong answer thrice doesn’t make it right. If you face a situation like the one we are considering here, however, it is important to correctly identify the foe you are fighting: The algorithm is not biased—it is an unbiased representation of a reality that is horribly flawed due to human bias. To right the situation, it therefore is insufficient to just fix the algorithm—it is necessary to fix the world.
However, what should you do about your algorithm? There is a short answer and a long answer.
The short answer is that your algorithm right now is perpetuating a biased view of Martians that fuels ever-increasing discrimination and injustice. Your algorithm has become an accomplice. As a first step, you therefore will have to consider whether you want to stop using your algorithm. This is a difficult ethical decision and not within the scope of this book. However, I invite you to imagine what in our little story would happen if you took the algorithm away from the police. Would the author of said Letter to the Editor and others thinking similarly change their views of Martians, and would they accept if the police stopped frisking Martians? Or would they demand that the police double up their fight against drug dealers, driven by an honest fear for their safety and the future of their children? And what would the police do without your algorithm? Would they start to frisk Martians and Zeta Reticulans with the same frequency and utmost neutrality, or would they revert to even worse biases than before?
Google faced a situation like this in 2016 when a reporter found that research results propagated hate speech—for example, if a user started typing “jews are” in the search engine.2 Google found a simple solution: auto-complete now is blocked for potentially controversial subjects.3 Unfortunately, there is not always such an easy way out if an algorithm reflects a deep bias that has crept into society.
The long answer is that your algorithm could become part of the solution. In our little example, your algorithm now wields supreme power in deciding who gets frisked. If your algorithm was changed in a way that is more “just,” it would result in more just outcomes. The big issue is that here the definition of justice is outside of the realm of statistics.
In the third part of this book, we therefore will visit a wide range of options one could consider for this “long answer.” There might be ingenious ways to collect better data that trumps the bias, or there might be a democratic, political process to define what the electorate considers “justice” as a basis to inform a management overlay over the algorithm in order to drive actual decisions.
Summary
If biases in the real world have created their own reality, statistical techniques lose their power to remove such biases on their own.
In such situations, an algorithm arguably becomes an accomplice that can perpetuate biases and thus entrench ramifications deeper and deeper into reality.
Nevertheless, even a biased algorithm might be the smaller evil compared to human judgment that implies even worse biases.
In the long term, algorithms can even be the solution to real-world biases, which we will discuss in detail in part III of this book (in particular in Chapter 16).
In a way, you have made it through the darkest section of this book. There are many other ways algorithms can be biased, but they all tend to be easier to solve than the particular issue discussed in this chapter, and the many solutions at our disposal will be the subject of the third and fourth parts of the book. The sun therefore now will slowly rise again, and with each chapter you hopefully will feel more empowered to make the world a better place by understanding, managing, and preventing algorithmic bias.