Questioning AI

Artificial intelligence (AI) is an umbrella term I’m using for a group of technologies being used in myriad ways today. As it continues a rapid evolution, our collective understanding, classification, and vocabulary surrounding AI will undoubtedly change.

I’m not an expert on AI. My interactions with it are pretty typical: I frequently interact with the results of AI through social media feeds, “personalized” product recommendations, and less obvious integrations of it within organizations from banks to the TSA. Like millions of others, I’ve also begun exploring the potential uses of generative AI tools like OpenAI’s ChatGPT and Google’s Bard.

But my day job involves some useful skills when it comes to understanding how we use AI and the potential impacts of that use. I analyze and interpret data to ultimately help businesses make decisions. Among the most fundamental aspects of that work are understanding where the data come from, how they’re collected, and (just as much as what they tell us) what they’re not able to tell us. While some analysts may dread someone questioning the underlying data or methodology behind what they’re sharing, I take delight in someone appreciating the importance of those things. Sure, some people plainly dismiss information that doesn’t align with their desired position, but I’ve more often encountered clients who simply have a healthy level of skepticism and appreciate the risks associated with the decisions they’re to make. With AI, we could all stand to be a bit more like those skeptics.

AI can work quickly and tirelessly and it can draw on a much larger body of references than we can alone, but it does so at our behest. It is a tool. We can consult AI to inform our decisions or outsource those decisions to it, but we (as a society, but not always personally) choose to give it power in our lives. If AI acts on our behalf, it’s because we’ve chosen to delegate that action. We place an increasing amount of faith in AI to behave accurately and safely, and do so often without the due diligence we would take with a human assistant.

AI is a human product, and therefore a biased product

While human fatigue and limitations of speed and mental capacity are designed out of AI (indeed, this is why it’s valuable to us), one very human element is still present: bias. AI is created by humans and it is trained on content generated or curated by humans. Human biases are part of the fabric of AI and the mechanisms of machine learning can not only repeat those biases, but they can amplify them. Many of us who interact with AI don’t understand its inner workings and come to view the black box as something very distant from its human creators. During recent oral arguments in the Supreme Court for Gonzalez v. Google LLC—a case disputing the protection of YouTube’s content recommendations under a 1996 law—the word “neutral” was used over 40 times in regards to algorithms used by YouTube and other online platforms. Close examination of AI implementations, however, reveals that they often stray far from behavior we would describe as neutral if carried out by a human. Recent work by researchers at the University of Southern California found that over one-third of the “facts” included in at least one AI database they examined were biased in some way.

“If you are human, you are biased.”
— Howard J. Ross

In some cases, algorithms are built with conscious biases that aren’t communicated to end users (or the non-user parties they impact). YouTube’s recommendation algorithm was designed not necessarily to show you content you will enjoy, but to serve content you will keep watching. The potential difference between those criteria may be minuscule to most of us, but it can also lead viewers to watch increasingly extreme content they aren’t likely to initially seek out. As the debate around the threat of TikTok in the U.S. swirls, one concern is that the Chinese government could covertly influence the content seen by a large swath of Americans by adjusting the platform’s algorithms.

More pervasive and potentially more impactful, however, the data used to train AI models contain bias. Amazon notably scrapped an AI project to assist with hiring after it was discovered that it interpreted gender skews in previous hiring pools to penalize female candidates. While Amazon says it never put this technology into practice, a memo leaked in 2022 suggests the company may soon try again. COMPAS, a software advertised to predict the risk of criminal recidivism, has been found to produce elevated risk scores for Black individuals. In several documented cases, judges and parole boards relied on risk scores to determine sentencing or parole decisions despite other factors that seemingly pointed in a different direction.

Moving beyond recognition

In recent years, many of us have become more aware of our implicit biases. We have worked to better understand these, identify their potential harm, and overcome them. Calls for increased transparency and regulation have aimed to prevent the effects of these biases from continuing, including in areas like hiring and criminal justice. As the previous examples demonstrate, simply replacing human decision-makers with AI doesn’t solve the problem. Just as we must investigate our own harmful biases and work to correct them, we must do the same with AI.

As users of AI, the following questions can help us promote the development of more ethical tools and ensure we use AI more responsibly. While these can’t be fully answered in many circumstances today, normalizing a more skeptical approach to AI can encourage tech developers and regulators to ask these questions as well.

• What model/technology is being used? Is the provider transparent about the source of the AI?

• Is there an incentive for the developer/provider if this AI demonstrates a certain bias? If so, would that bias potentially conflict with my own goals or ethics?

• How representative is the content used to train this AI? Does the body of content reflect existing biases I wish to avoid?

• Does the result align with other sources, similar cases, or my expectation? If not, is it likely because the AI has properly considered something else or because it’s biased against another outcome?

• What are the risks associated with using the output of this AI? Am I comfortable using it based on the information I have?