Deepfakes

Perhaps the most infamous cases of AI working toward a harmful goal involve deepfakes. A deepfake is an image, video, or audio track in which a real person is combined with synthesized elements of another imaginary or real person (see Figure 3.1).

The most notable malicious use of deepfakes has been to swap the faces of (primarily) female celebrities onto pornographic images and videos. Other instances involve political figures—substituting the face of Adolf Hitler, for example, onto the image of the president of Brazil. Audio components can be forged as well if there is a repository of audio from which to learn a person's speech tones and patterns. Resemble.ai claims to be able to clone your voice, enabling you to create scripts that are then read in your voice. University of Washington researchers were able to learn and then synthesize lip movements to match speech patterns, supplying the video side of the equation.

Deepfakes leverage image and/or voice recognition technology to superimpose the face or voice of a specific person onto already existing media. The use of deepfakes to date has been primarily malicious or, at best, innocuous and created primarily for humor. Deepfake creators know that they are, in some cases, violating or skirting the law, and their express intent in these cases is largely to create mischief. Providing them with the ability to explain and interpret their algorithms does not deter them, making this a problem that belongs in the realm of public policy, not responsibility on behalf of the individual practitioner.

The methods outlined in this book, because they assume an algorithm's developer is seeking to avoid causing harm (or at least is not actively seeking to cause it), are of little value in cases like these. Public opprobrium, new regulation, or heightened legal consequences are the only way of controlling such harms. Such was the case with deepfakes for revenge porn in Australia, where a deepfake attack on a law student triggered the passage of specific anti-revenge porn laws.¹

Supporting State Surveillance and Suppression

Another controversial use of AI working as intended in the furtherance of harmful goals is government use of AI tools to control their citizenry, for example, the Chinese government's use of algorithmically derived “social credit” scores. The concept of social credit is an extension of the idea of an individual's financial creditworthiness, which has been tracked for more than a century. Financial creditworthiness is embodied in the familiar credit score, which is calculated for an individual by an algorithm that purports to predict the likelihood that money lent to the person will be repaid. Banks and other financial institutions can then use the credit score to help make lending decisions.

In China, the credit score has been extended beyond financial questions to social behavior issues, including:

• Failure to clean up after your dog
• Failure to pay traffic fines
• Failure to dim your high beams for oncoming traffic
• Failure to visit elderly parents regularly
• Failure to properly sort your waste and recycling

All these infractions can get factored into your social credit score in China, which, if it drops low enough, can result in consequences including restrictions on air travel, getting loans, pet ownership, and more.²

Using AI to track down and punish these offenses may not seem like a harmful goal; you might know some folks who would be happy to have a similar sanctioning regime in the United States or Europe. In China, though, it forms part of the scaffolding of the surveillance state. These social credit scores are an important component of on overall AI effort that also includes facial recognition algorithms and comprehensive data collection. The effort is particularly vigorous in provinces like Xinjiang that are inhabited by Uighurs, who are predominantly Muslim, serving at least in part to suppress that minority.

While deepfakes are a specific AI deployment that has no current purpose other than humor, mischief, or harm, the algorithms that lie behind the surveillance state also have more standard and legitimate purposes (e.g., maintaining local or national security). Nonetheless, the intent behind the usage of these tools as well as the balance of possible harms and possible benefits are relevant here. While cultural norms of the appropriate role government should take in securing peace and security may differ from country to country, we take the view that using the power of AI to abet ethnic repression or suppression of peaceful political dissent is an irresponsible use of the technology that ought to be curtailed.

Behavioral Manipulation

The Cambridge Analytica scandal is a famous example of standard AI algorithms deployed intentionally in furtherance of dubious goals. In a slick video presentation before a large audience at the Concordia Summit in 2016, Alexander Nix, Cambridge Analytica's CEO, reported how the company had gathered an enormous dataset on U.S. voters that characterized them in psychological and social terms. The data were sourced from public voter records, Facebook profiles, and other sources. Nix then described how the company was able to define individual voters in terms of their dominant personality traits and craft individually targeted political messaging that tapped into the motivations governed by those traits. In his talk, Nix implied that the messaging need not be true: for fear-motivated individuals, a “sharks sighted” sign would be more effective in warding off trespassers than a “private beach” sign, without regard to whether there were sharks present.

Cambridge Analytica was employed by the 2016 Trump campaign. In a 60 Minutes interview, Brad Parscale, the digital media director of Trump's campaign at that time, described how he supervised a massive message-testing and deployment campaign in which tens of thousands of Facebook ad experiments were conducted daily in an AI-driven search for the message that best produced the desired result (clicking the ad). Attributes like color, different photographs of Hillary Clinton, headlines, and various calls to action were continuously modified in an automated fashion. An algorithm tracked who responded to what combinations and learned what to present to given individuals to maximize a response. This was not something Parscale thought up on his own; it was the same technique Facebook used to maximize responses to commercial advertising.

There is nothing particularly new, or necessarily irresponsible, about personality-driven advertising; Madison Avenue has employed it since the 1950s. The automobile industry has used personality information in crafting car models and the ad campaigns that support them. The use of microtargeting in political campaigns is newer but still more than a decade old: Ken Strasma, who teaches political targeting methods at Statistics.com, pioneered its use as long ago as in the 2004 Kerry presidential campaign. Microtargeting combined with psychological manipulation is new and acquires even greater potency when combined with automated experimentation to maximize response. The combination of these three elements with a fourth, deception, is where trouble lies.

Automated Testing to Fine-Tune Targeting

Guided by Facebook staff, Parscale introduced a third element to the Trump campaign's advertising toolkit: continuous testing and optimization. Based in part on the statistical methodology termed multi-armed bandits, continuous testing and the optimization of ads is an important part of the Facebook business model. The more effective an ad is, the more it can be sold for. For important clients, i.e., those with large Facebook advertising budgets, Facebook will embed ad-targeting specialists with the firm to set up and tweak the AI algorithms. In his interview with 60 Minutes on CBS, Parscale reported that Facebook did this for the Trump campaign, and he came close to crediting Facebook's sophisticated AI-driven advertising strategy for the Trump victory.

DEFINITION   MULTI-ARMED BANDITS   In a traditional A/B test, a sample of subjects is gathered, and two treatment options are offered, say medical therapies. Typically, results from the entire sample are anticipated, and a statistical significance threshold must be reached to determine that a treatment really is superior and not the product of chance. This can mean that many subjects receive an inferior treatment. In digital marketing, though, there are hundreds, if not thousands, of potential “treatments” (digital ad variants) and millions of potential subjects. A multi-armed bandit experimentation process (see Figure 3.2) makes continuous decisions about the “winning treatment” to balance exploitation (rolling ahead with a winning treatment) with learning (continuing to try different options to see how they perform). Various optimization algorithms are used, depending on how much you value speed and are willing to risk losing out on a treatment that might ultimately be a winner but is an early underperformer.

A key feature of the continuous testing process is its huge number of automated experiments and decisions not requiring human review. The real basis for success or failure of an ad, say the presence or absence of particular ethnic groups in a photo, might be hidden. Parscale reported that he was conducting tens of thousands of ads daily at the height of the Trump 2016 campaign.

It remains uncertain how successful the psychographic aspect of targeting was. A competing Democratic digital marketer doubted its effectiveness and also doubted that Cambridge Analytica really had all the Facebook data that Nix claimed it did. Facebook, he said, would not allow this. As it turned out, Nix did have Facebook data from more than 80 million users, data that was scraped illicitly through the Facebook app This is Your Digital Life. That app was developed by Aleksandr Kogan, a research associate at Cambridge University, for the express purpose of collecting data. The user filled out a short quiz, which enabled the app to access not just their own data but the data of all their Facebook friends as well. It was this unauthorized theft of personal data from Facebook that ultimately got Cambridge Analytica into trouble more than the behavioral manipulation it enabled.

Does psychographic behavior manipulation work? Two researchers, Chris Sumner and Matthew Shearing, conducted a study of digital ads aimed at more than 2,000 internet users who were scored either high or low for the personality trait of authoritarianism. They then crafted ads with a call to action to support mass surveillance. The ads came in two flavors.

• Messaging designed to appeal to authoritarians
• Messaging designed to appeal to anti-authoritarians

They found that the messaging flavor mattered. This is an extension of the well-known phenomenon in survey design where it matters how you word a question. What Sumner and Shearing found was that flipping the messaging tone would flip the person's position on the surveillance question.

Statistics and machine learning have brought important new dimensions to the long-standing practice of mass consumer and voter manipulation:

• Massive experimentation, which, coupled with big data, can lead to effective microtargeting at the individual level
• Concealing or obscuring the basis on which particular ads are targeted at particular individuals

Is there anything wrong with this? After all, truth has never been of paramount importance in commercial and political promotion. However, in the old regime of mass advertising, at least messaging was exposed to everyone on a wide basis and available for public viewing and broad judgment. Microtargeting and experimentation with thousands of messaging combinations have given deception and outright lies hidden pockets in which to operate out of sight. Moreover, the focus on individuals as targets can allow broader bias to prevail: The Washington Post reported on September 29 that the Cambridge Analytica database prepared for the Trump campaign ended up disproportionately targeting African American voters for messaging aimed at deterring voting.

These three preceding examples illustrate a range of cases where the intent is ill. Why then would the methods we discuss in this book help? In the case of a lone wolf who is an expert coder with mischievous or malicious intent, they might not. But many of the examples of harmful use arise from the development and/or implementation of AI in an organizational context. A data scientist trained from a perspective of transparency, interpretability, and ethical auditing is more likely to spot the potential misuse of the AI they are asked to develop and spread that perspective within the organization. Some individuals concerned with the Cambridge Analytica scandal now say they regret having developed the tools that later led to the company's ruin. A data scientist who is engaged with the broader ethical and societal implications of their work, as embodied in the Responsible Data Science framework we outline in the next chapter, and not simply with the “cool factor” of data science, is less likely to contribute to its misuse. They are also more likely to imbue their data science colleagues, as well as those in their organization who are less technical, with the same ethos.

AI and Unintended Consequences

Let's now turn to cases where the processes that we lay out in the next chapter can be much more helpful: cases where algorithms developed with good or benign intentions produce harmful outcomes (sometimes in spite of otherwise reasonable results). In those circumstances, developers and users of algorithms would have benefited from a process to render the AI models they are working on more transparent and interpretable. This would have provided them the knowledge and tools to anticipate such problems and take steps to reduce the risk of harm.

Healthcare

We already saw, in Chapter 1, the case of Optum, whose algorithm for predicting the need for follow-up care or rehospitalizations boomeranged and ended up discriminating against African Americans. Optum did not set out to discriminate against African Americans; the discrimination was an unanticipated by-product of the algorithm.

The Optum case is not unique. It just happened to generate a lot of attention. According to a study from the Society of Actuaries,³ more than two dozen risk scoring models are used by healthcare providers and insurance companies. The purpose of the risk score is to predict future health. A research team led by Ziad Obermeyer studied these algorithms and found that, for any given risk score, African American patients consistently experienced more chronic health conditions than did white patients (see Figure 3.3).

Another example comes from the field of genetic testing. Hypertrophic cardiomyopathy is a condition in which the heart wall thickens, impeding its ability to pump blood. It affects roughly 1 in 500 people. There is a genetic component to the disease, which can be useful in diagnosis and also in the screening of relatives of a confirmed patient. A person diagnosed with hypertrophic cardiomyopathy may be prescribed treatments ranging from medication to surgery. For African Americans, though, the predictive power of the genetic screening was very low compared to others. The reason? The genetic data used to train the model had practically no gene sequence data from African Americans. Without this data to train on, the model missed the fact that African Americans were far more likely than other groups to have a benign version of the genetic mutation associated with the disease. Hence, they were more likely to be referred for further testing and for treatments that carried at least some degree of unnecessary risk.

Interestingly, bias based on race is still explicitly included as part of medicine. In June 2020, the New England Journal of Medicine catalogued a number of treatment scenarios where race is explicitly included as a risk factor. For example, guidelines promulgated by the American Heart Association assign lower cardiac risk scores to African American patients than to white patients. This means that an African American patient presenting with a given medical history and condition is likely to be assigned a less proactive therapy than a white patient with the same history and condition. Similar examples were cited in nephrology, obstetrics, urology, and other areas. These race-based risk scores most likely are derived from minimally adjusted population estimates and reflect long-standing beliefs in a genetic basis for race covering many biological facets. The scores probably do not reflect sophisticated statistical analysis that allows for adjustment based on “other things being held equal.”

Finance

In 2019, Apple introduced a new credit card in conjunction with Goldman Sachs. Shortly after the launch, prominent software developers started to complain publicly on social media that the card was biased against women in setting credit limits. David Heinemeier Hansson, the creator of Ruby on Rails, reported that he had been granted a credit limit that was much greater than that of his wife, Jamie Hansson, although she had a better credit score. On Twitter, he stated:

Steve Wozniak, cofounder of Apple and coinventor of the original Apple computer, chimed in with a similar complaint:

If Steve Wozniak finds it hard to get Apple's attention to correct this problem, we can be sure that the issue is not simply with algorithmic bias alone. Here, we are dealing with an issue of power, where users are denied the due process or recourse to appeal a decision made about them by an automated black-box algorithm.

Law Enforcement

A notable example of AI bias in the law enforcement arena is the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm to predict recidivism (propensity for a convicted criminal to re-offend). Courts have started relying increasingly on AI recidivism algorithms to inform decisions on sentencing. The COMPAS algorithm is among the most prominent of them, and its advocates point to its overall good predictive performance. They have responded to allegations of bias by noting that this predictive performance, as measured by the area under the ROC curve, is similar for African American and white defendants. The trouble is, the errors made are quite different.

• African American defendants are overpredicted to re-offend (leading to tougher sentences).
• White defendants are underpredicted to re-offend (leading to lighter sentences).

The errors are decidedly unfavorable to African American defendants, but they balance out the ones made in favor of white defendants, so the overall error rate is the same for both. We discuss more about the specifics of this case in Chapter 6, “Beginning a Responsible Data Science Project,” and Chapter 7, “Auditing a Responsible Data Science Project.”

There are other examples of obvious algorithmic predictive bias based on race or gender, but there are also interesting examples of more subtle unintended effects of AI in the area of law enforcement.

Around 2016, undocumented immigrants from Mexico in the Washington State town of Nahcotta, a shellfishing center, began disappearing. Boat workers had occasionally been picked up and deported before, but the pace picked up dramatically in 2016. The political climate had changed, and the tools of big data and AI were being brought to bear.

In the first decade and a half of the 2000s, US immigration policy was highly polarized between those favoring a relaxed, more open-door policy and those favoring a more restrictive policy. The general consensus, though, was that the 10 million people who had lived large portions of their lives here in the United States, albeit without documentation, should not be subject to roundup and deportation. Such a policy might be considered law enforcement in a technical sense, but it would have been seen by most Americans, a quarter of whom have at least one foreign-born parent, as excessive and draconian. Of particular concern was the fate of children who had lived nearly all their lives in the United States but who were technically undocumented aliens.

In the 2013 debate over proposals for immigration reform, the status of undocumented aliens was a principal point of contention. Open-door advocates favored a rapid path to citizenship. Some restrictionists favored ultimate citizenship, while others supported legalized status but not a rapid path to citizenship. Still others opposed nearly all legalization efforts. No significant political voice, however, advocated rounding up long-term undocumented aliens and deporting them.

The advent of AI-powered law enforcement techniques disrupted the consensus on this latter point and allowed the US Immigration and Customs Enforcement Agency (ICE) to pursue a more aggressive policy of finding and deporting undocumented aliens, even longtime US residents. Deportations in the first 14 months of the Trump administration tripled from the previous 14 months. Big data and multiple AI tools played an important role.

• Computer vision has contributed two key components: facial recognition and license plate readers.
• Location prediction algorithms can help predict where someone is going to be.
• Entity resolution algorithms enhance identification efforts by bringing together multiple disparate sources of identity data (e.g., ICE files, driver's license records and photos, tax records, utility bills, social media postings, etc.).

All this is not to suggest that law enforcement is unethical. But laws and regulations are not hard formal concepts that exist in a vacuum. Discretion, judgment, and cultural factors are part of the equation that can make laws either part of an agreed consensus-based social compact or the legal basis for a police state. AI and big data have given immigration enforcement a giant push away from the consensus “live and let live” approach to longtime US residents and toward an aggressive pursuit of undocumented people, even those who have been living peacefully in the United States for decades.

On a smaller scale, consider the doorbell camera. A neighbor of one of the authors recently heard a squealing of brakes, the sound of an impact, and then a car accelerating away. A doorbell camera recorded a video of the immediate aftermath: the car accelerating away, followed by a figure staggering into the frame from the left and then out again on the right. Amazingly, the young man who was hit was not seriously injured.

Doorbell cameras feature high-definition video recording and motion-detecting algorithms. Some record continuously, saving the video for a few seconds and then discarding it unless motion is detected. They are popular as personal security devices, enabling residents to see who's at a remote door and preserving evidence of intrusions. As in the escaping auto case, they can also provide useful information for police. Where do ethical considerations come into the picture?

In November of 2019, the online publication The Intercept claimed to have seen Amazon internal documents that discussed adding facial recognition capabilities to its entrant in the smart doorbell competition, Ring. The publication said the introduction of facial recognition would lead to the development of “neighborhood watch lists” of suspicious or undesirable people. An attorney for the American Civil Liberties Union (ACLU) raised the alarm that enabling watch list features on Ring would widen the scope of existing government watch lists and exacerbate the denial of due process associated with such lists. And, it is not hard to imagine a consumer-enabled watch list capability taking on a life of its own and leading to algorithm-enabled vigilantism. To be sure, Amazon denied that it had plans to implement facial recognition technology in Ring, though it admitted that it had thought about it. However, technology once developed is hard to keep bottled up. Like water running downhill, it will eventually find its way around obstacles. Pure intentions of the original inventor are no guarantee of long-term protection.

Technology

In 2015, it came to light that Google Photos was mistakenly labeling some dark-skinned people as gorillas. Google's AI had gone down the same path that crude racists had trodden for decades, in referring to African Americans as gorillas or other types of apes. Data scientists at Google, horrified at what was happening, tried to fix the problem. To prevent such offensive mislabeling in the future, they dropped the label “gorilla” from their algorithm's lexicon. After that, the algorithm would either mislabel gorillas as something else or simply throw up its hands and say, “I don't know.” For good measure, Google included other terms, such as chimpanzee, chimp, and monkey in the ban. The offensive mislabeling problem was solved by “disappearing” certain animals.

Google's Vision missteps are not limited to Google Photos but underlie the technology itself. The publication AlgorithmWatch conducted an experiment with the image of a hand holding a thermometer gun. When the hand was dark-skinned, Google came back with the label “gun.” When the hand was colored with a lighter overlay, Google's label changed to “monocular.”

In one sense, Google's action on Google Photos was laudable: immediate correction of grossly offensive behavior by its algorithm. In another sense, though, it revealed arrogant presumption. Its algorithm couldn't label people correctly, but instead of discontinuing the labeling (which was hardly an essential service), it elected to alter the reality with which the algorithm was working to eliminate the possibility of offense.

This was a trivial example, perhaps: the addition of more training data will no doubt improve Vision's algorithms over time. Google's behavior should be seen in a context in which managing and altering the popular understanding of reality has long been at the center of totalitarian government behavior. When Joseph Stalin died in 1953, his key lieutenant, Lavrentiy Beria, lost favor and, in the spirit of Stalinism, was executed. The keepers of the Soviet Encyclopedia, mindful of their political duties, “erased” Beria from the books, sending owners a four-page spread on the Bering Sea and Admiral Bering to replace the lavish portrait of Beria. AI can rewrite reality much more effectively, requiring neither paper nor scissors.

Ignorance Is No Defense: AI in the Context of Existing Law and Policy

Although AI technology has begun performing tasks with near-human or superhuman skill, it is not above the law where such laws may apply. If an AI or ML system takes an action that is illegal or recommends an illegal action that is subsequently taken by a human, that is a violation of the law. For example, the Federal Trade Commission (FTC) has prosecuted numerous companies for violations of the Fair Credit Reporting Act of 1970 for discrimination by automated decision-making systems.⁵ We have already highlighted some extreme examples of this earlier in the chapter within cases where the form of AI was working as intended, yet working toward a goal that was actively harmful. While much press is devoted to new frontiers of AI application that may lack laws or regulations (such as facial recognition systems), there are some domains where laws already exist and provide means of redress.

For example, some industries that have long leveraged algorithmic decision-making, such as financial services in the Equal Credit Opportunity Act and software-based medical devices, are subject to laws and regulations that extend to algorithmic/quantitative decision-making to ensure transparency and fairness in their applications. For example, in banking, it is illegal in the United States to deny credit to a customer on the basis of their sex, race, or ethnicity. This is true regardless of how the loan decision is made, and banks currently have highly developed internal frameworks for auditing their own models (AI or otherwise) for fairness to ensure compliance with existing regulations. Companies (and by extension their data scientists and modelers) who create algorithms in these industries are subject to the laws of the countries and territories where their models are applied. Similarly, data scientists working with data and models in a classified environment are still required to maintain classification of that information. So, while the technology behind AI may be new, many protections already exist against its criminal or negligent uses in the form of existing laws and regulations in industries where the misuse would follow old patterns.

What does this mean for data science practitioners? Basically, algorithmic decisions are subject to the same or greater degree of regulatory coverage as those made by humans or previously existing quantitative decision support systems. If you are working as a data scientist in a field that has existing laws and regulations that are meant to protect consumers, there is a high likelihood that your models may already be subject to regulation. This is especially so in industries that deal with sensitive personal information, such as medicine, banking, and insurance. In those cases, you might be required to show the basis for an algorithm's decisions, which means an interpretable model must be used.

A Finger in the Dam: Data Rights, Data Privacy, and Consumer Protection Regulations

Apart from those in already regulated industries, up to the beginning of the twenty-first century, there were few if any laws or rules that applied to businesses relying on access to personal information. Personal data were increasingly monetized by digital marketers, internet service providers (ISPs), and search and social media companies.

Concerns over the aggregation, use/misuse, and sale of personal data and information by corporations led in the latter part of the 2010s to significant attention from international lawmakers. This was particularly true in Europe, where citizens were guaranteed an explicit right to privacy by law (Article 8 of the Charter of Fundamental Rights of the European Union). It was clear that the impacts of the internet economy had rapidly outstripped any nascent protections that were put in place at its founding. As the costs of large-scale data breaches at companies such as Adobe, Equifax, and LinkedIn and rising identity theft became apparent, these costs were also being borne primarily by consumers.⁶ Lawmakers recognized that regulation may be required to ensure adequate protection of consumer information and sufficient punishment for a failure to safeguard sensitive data.

The most prominent and influential of the regulations to emerge from this period was the General Data Protection Regulations of the European Union (GDPR). Adopted in 2016, the GDPR went into effect in August 2018. The EU enacted GDPR in line with the Right to Privacy enshrined in the 1950 European Convention of Human Rights, one of the founding documents of the supranational union. GDPR provide sweeping, albeit generic, protections to European citizens including (but not limited to): a right to “be forgotten” in terms of online presence, as well as a requirement that any technology companies provide mechanisms by which EU citizens can have their data removed entirely from online platforms.

Because the GDPR were one of the first and most comprehensive frameworks for data regulation, these regulations have become an archetype for other consumer data protection laws and regulations at US state levels. For example, the California Consumer Protection Act (CCPA) was inspired by GDPR, especially in the degree and manner by which personal data are pseudonymized.⁷ As of September 2020, other states like Massachusetts and Oregon are considering or enacting laws similar to CCPA for their citizens.

At the national level, Canada enacted its Personal Information Protection and Electronic Documents Act (PIPEDA), which is broadly similar to CCPA in terms of who is protected, what rights they hold to their own data, and the responsibility of businesses that transact in personal information. The influence of GDPR has also spread to non-Western countries of the world that are all too often left out of the tech policy conversation. In Africa, Kenya recently passed its own data protection laws as inspired by the GDPR mold. Brazil's General Law for the Protection of Data (Lei Geral de Proteção de Dados [LGPD]) went into effect in August 2020 and is heavily influenced by GDPR. Southeast Asian nations have a patchwork of data protection regulations, including comprehensive GDPR-like regulations planned in Indonesia and Vietnam. In the last 10 years, the online world has gone from largely unregulated to some form of regulation in most of the world.

What does this mean for data science practitioners? Regulations like GDPR have raised real questions about what data can and cannot be rightfully included in a model by data scientists. For example, if a person has requested that their data be forgotten, must a model trained using that person's information now be retrained? According to GDPR, the answer is likely yes, given restrictions not just on the storage of personal information but also its processing. Other regulations like CCPA do not necessarily extend to the analysis and processing of personal information, but focus primarily on levying penalties against companies that fail to adequately safeguard consumer data. Regardless, as data residency and localization laws have begun to build upon the frameworks laid by GDPR and other data privacy regulations, data scientists must be careful whose data they are accessing, and when and how, when building training datasets.⁸ This is not just a question of ethical transparency or fairness, but also may increasingly be a question of law, depending on where those data reside.

Trends in Emerging Law and Policy Related to AI

At a recent panel on Data and Privacy at the University of Virginia, one of the authors was asked by students from the School of Engineering and Applied Sciences and the recently formed School of Data Science what he saw as being the most important development in AI in the next five years. The response was quick and, at least from the perspective of the students, unexpected: regulation. While it's important to remain abreast of the state of technical art in a field like AI or machine learning, the direction that development takes can be significantly influenced by laws or regulations that make further advancement along a particular path illegal. Facial recognition technology seemed like a solid technological bet as a business 18 months ago; now, companies that made that bet are seeing Dr. Hyde emerge from the laboratory in the cellar.

The advanced technological nature of AI means that large information asymmetries exist between developers/producers of AI and the general public. As in other professional fields, such as medicine, law, and finance, this asymmetry can create a significantly increased potential for harm. As with data privacy issues, governments around the world are moving at different rates to respond to the potential legal issues posed by AI with respect to their citizens. If regulation is to govern the data—the raw field material—by which AI algorithms are trained, so must it also govern the more powerful technologies built on that data.

What is distinct about potential AI laws from the two aforementioned categories of law in this chapter is that governments, in addition to being both the prime customers and adopters for AI, are also charged with being responsible for ensuring the appropriate and ethical use of the technology by their citizens. The end result for data scientists is that the legal and regulatory landscape around AI is likely to become an increasingly dynamic environment as governments attempt to balance their deepening investments in AI with the needs of their civil societies. Because of the highly dynamic nature of the AI legal landscape, we focus here on emerging trends, rather than specific or forecasted laws.

Perhaps no country in the world is as broadly concerned with the legal and ethical ramifications of AI as Australia. As early as 2004, Australia had set out 27 best-practice principles for the application of automated decision-making to questions of administrative law to ensure transparency and fairness in adoption. More recently, an active and public debate around automated decision-making in government has arisen as a consequence of the “robodebt” scandal.⁹ The Australian government adopted a machine learning technology to validate social welfare payments and assess individuals who had collected more than they were due, only to discover that the system reinforced deep and systemic biases, particularly against minorities. Following this embarrassing deployment of machine learning, legal scholars in Australia began calling for more comprehensive protections against the misuse of AI. The Law Council of Australia has proposed an ethical framework for the development and implementation of AI, along with processes to support implementation.

Elsewhere, there is ongoing discussion and debate about AI ethics and draft legislation. In Europe, serious questions are being raised about the proper use of algorithms in government and assuring the public a right to due process. The European Commission's (EC) recent whitepaper on AI highlighted specifically that the opaque nature by which AI algorithms arrive at decisions poses unique risks to ensuring their fair and just application. Simultaneously, the EC recognizes that any law regulating AI must be sufficiently flexible to encompass future innovation in the rapidly evolving technical domain.

In the United States, federal executive agencies are issuing guidelines for the use of AI in advance of expected laws (e.g., the DoD's Ethical AI Principles). Legislation has also been proposed on the floor of Congress (specifically, the Algorithmic Accountability Act of 2019) that would require companies as well as federal agencies to assess the potential impacts of their AI algorithms on citizens and consumers.

What does this mean for data science practitioners? Industry-specific regulations and privacy rules may not be created specifically with AI in mind but do impinge on the work of data scientists. In some industries, e.g., healthcare, employers may require data scientists to at least be familiar with these industry-specific regulations. Laws specifically aimed at governing AI or other algorithmic decision-making systems, by contrast, lie at the core of data science. Depending on where data scientists work or where their algorithms may be deployed, governments are increasingly looking at whom to hold accountable for the outcomes of AI models and how. This is a highly dynamic area with significant geographic variability. Governments, to some extent, have been prevaricating or avoiding laws and regulations that would have a hard and specific impact on the practice of AI. Governmental actions have focused more on issuing ethical principles and guidelines. With the legal landscape less than clear, sometimes the only thing data scientists can do is to minimize the harm caused by their own actions. This leaves it to data scientists and broader project teams to translate those ethical principles and guidelines into actual procedural and technical steps (which is why we need the tools presented in this book). Most recently, these efforts have centered on optimizing models to meet specific fairness metrics.

NOTE   BIAS, FAIRNESS, AND UNFAIRNESS   What are the definitions of bias and unfairness from a legal standpoint? One is tempted to say that bias and unfairness are self-evident. The problem, though, is that different people see bias and unfairness differently.

Nearly everyone would agree that a social club with an explicit prohibition against membership for certain religious groups shows bias. Now, change the situation slightly—suppose there's no explicit prohibition, but the implicit preferences of the membership committee result in the same groups being excluded? Most people would still contend the club is biased. Consider next an engineering college that selects the highest-scoring applicants for an incoming class, and the result is an overrepresentation of students from affluent families. Some might contend that the selection process itself is implicitly biased, perhaps as a result of a narrow focus on criteria (like test scores and extracurriculars) where high-income groups have more opportunities. Suppose now the selection process is a random selection instead of score-based, and owing to an imbalance in who applies, there's still an overrepresentation of high-income students being admitted. Some schools consider the lack of diversity resulting from these issues to be a bias problem, albeit of society as a whole, and undertake its correction. Others then contend that this correction itself is a form of bias.

Such questions are an unending source of debate in society and within the legal profession itself. Legal scholars Ronald Dworkin and Richard Posner have each written 500-page tomes on the subject of equality and rights, from different perspectives, with each coming to differing conclusions. We will not attempt to contribute to this debate. Each organization must at least grapple with the question and come to its own judgments, and data scientists must think for themselves on the matter. In this book, though, we bring forth tools that can be used to help control the development and deployment of AI projects, not only bringing them into compliance with legal and regulatory requirements but also aligning them with the ethical principles of the organization or the individual.

Similarly, there is little consensus on what constitutes (nonstatistical) bias and fairness within a data science context. In Chapter 7, we will go more into the specific metrics that have been proposed to quantify fairness. However, it will suffice for now to say that fairness is subjective and depends on the context of the modeling task at hand.