Algorithmic bias in the context of artificial intelligence, machine learning, and natural language processing could mean varying degrees of negative impacts. For Dr. Vered Shwartz, assistant professor of computer science at the University of British Columbia, building culturally aware NLP models is important to mitigate any harm like cultural and even gender discrimination in critical decision-making processes like loan approvals and filtering job resumes.
Dr. Vered Shwartz is also the CIFAR AI Chair at the Vector Institute. Her research endeavors include computational semantics, pragmatic reasoning, and building AI models that are capable of understanding language at the human level. She’s made contributions as a post-doctoral researcher at the Allen Institute for AI where she pursued that understanding of implicit meaning in human speech and developing “culturally aware” NLP models.
On The AI Purity Podcast episode 8, Dr. Shwartz shares her research insights and shines a light on the existing artificial intelligence biases and how we can mitigate them.
Dr. Vered Shwartz On Being Drawn To The Field Of Computer Science
Dr. Vered Shwartz shares on the podcast that “what drew me to computer science is that I love problem-solving”, citing that as the intrinsic interest that led to her success in her current discipline. Venturing specifically in the realm of natural language processing was more serendipitous in comparison. She found out about NLP after taking a course on it during the final year of her undergraduate studies. She became captivated, and she would go on to pursue her master’s degree and PhD in NLP.
There was a pivotal moment in her academic journey around the time deep learning was revolutionizing NLP. “It was a very interesting time”, she recalls. As a non-native English speaker working on NLP that primarily revolved around the English language, she saw early on the parallels between her improving her English while teaching software to “speak and interact in English with users.” She says most of her work today still revolves around how both humans and computers understand and misunderstand language.
The AI Purity Podcast has had many experts in computer science share their thoughts on AI. Check out a previous episode where we discuss The Negative Societal Effects & Biases in AI Systems with Dr. Ted Pedersen.
The Importance Of Implicit Meaning and Advanced Reasoning
Algorithmic bias doesn’t just happen. After all, the software is trained with thousands if not millions of data in order for it to understand human commands.
Dr. Vered Shwartz was motivated to explore the implicit meaning and advanced reasoning in AI because of her interest in the “real-life and societal aspects of computer science”. After all, it’s what led her to pursue a career in NLP. During her PhD as she was learning lexical semantics, or how individual words can mean different things once they interact or are combined with each other in phrases, the importance of being implicit.
An example she cited was “noun compounds like olive oil versus baby oil, which have the same head noun ‘oil’ but have very different meanings.” She continues, “We use the condensed form to convey the meaning of oil extracted from olives and oil used for babies. [If] you’ve encountered this term before about baby oil, you would probably know very well that it’s not the same as olive oil. It’s oil made for babies.”
What comes naturally easily to understanding humans using language might not always translate to machines. “Computers were very bad at this”, Dr. Shwartz says. What is common sense for humans didn’t always start as common for machines as it probably is today for large language models like Chat GPT for example that can now generate human-like sentences and reasoning. While Dr. Shwartz acknowledges that there has been significant progress made, there’s still space for these machine learning models to suffer from “hallucinations”. This happens when large language models generate false or misleading information. Despite the great feats these models have achieved in recent years, Dr. Shwartz points out that machines’ reasoning abilities are inconsistent, something that wouldn’t happen in regular human conversation.
There’s a reason why it’s important to develop culturally-aware models. In today’s age, AI and machine learning are more and more integrated into everyday lives. Read our previous blog on ‘Machine Learning Applied In The Real World’ to know more.
Algorithmic Bias: What It Is and How To Eliminate It
According to the Harvard Business Review, algorithmic bias can occur when certain populations and underrepresented in training data or when pre-existing societal prejudices bleed into the training data. Just as important as it is to be very implicit with meanings and explanations, machines should be trained with fair data across cultures and languages. Due to the large volume of training data, there are, it can be quite hard to weed out and use just the culturally sensitive content. Training data bias is the culprit for the algorithmic bias that manifests in AI and language models.
An example Dr. Shwartz cites is for large language models that generate text. The training bias comes from web text that could contain explicit and harmful rhetorics like racism. These large language models then could “inherit” or perpetuate those beliefs.
Algorithmic bias affects not just underrepresented cultures, training data bias could also mean favoring one sex over the other. Dr. Shwartz cites a well-known case involving the company, Amazon, which used a CV filtering system that discriminated against women. Because the model they used was training primarily on historical data that reflected bias against women, their resumes were treated as “out-of-distribution” and less likely to be selected.
With the large-scale deployment of language models these days and artificial intelligence bias being very much a thing, Dr. Vered urges users to be cautious. She also emphasizes the need for transparency and cultural awareness in NLP models in order to mitigate further risks.
AI Purity provides machine learning developers with an accurate AI detector to help create better software and models for future users.
How To Prevent Algorithmic Bias
Preventing algorithmic bias may not be as simple as it sounds. The first probable step would be to acknowledge the issue of underrepresentation or misinterpretation of cultural and linguistic groups in NLP models. The next would be to make sure to use culturally-sound training data that is both implicit and inclusive.
Dr. Shwartz mentions her own work where she and her team identified a problem within a small commonsense reasoning model called COMET, a model trained primarily on annotations from US-based English users. Just like her earlier example with the difference between olive oil and baby oil, this model failed to understand the cultural significance of a dish called a Dutch baby which is a type of German pancake. Instead, the model interpreted it as something negative and unethical because it was unfamiliar with the term. The fix? They retrained COMET using datasets that included definitions across various cultures. The result? A more culturally aware model.
It’s not just NLP models that can work with this strategy as Dr. Shwartz explains a similar approach applied also to vision models that would often only portray Western-centric imagery just based on the primary training data that was used to develop it. When the vision model was asked to generate images of breakfasts it used to only generate Western-style breakfasts until it was trained on a large-scale dataset of images from various cultures.
While the aforementioned strategies have somehow fixed those smaller models, they might not be sufficient in solving how to prevent algorithmic bias. Dr. Shwartz then discusses the two main challenges in fixing the problem for large language models like Chat GPT.
-
Scale and Dominance
Large Language Models like Chat GPT are trained on vast datasets that are usually “scraped” from the web. Algorithmic data can be inevitable because it scrapes everything from racist to sexist rhetoric. That data is also primarily taken from North America which further exacerbates the inherent bias.
Even with a conscious effort to include culturally diverse data, Western and English content would still dominate. If developers try to reduce the amount of the dominating dataset, it would produce a reduced quality of output and effectiveness of the model overall.
-
Cultural Interpretation
Unfortunately when it comes to language models, familiarity with concepts doesn’t always equate to proper interpretation and understanding. The human language can be ambiguous and sometimes context-dependent, a process that language models have yet to truly grasp.
Listen To The AI Purity Podcast
Despite the marvelous feats AI and large language models have achieved these past few years, it’s obvious that there is still room for improvement, to say the least. While large language models can predict and mimic human language and conversations, our discussion with Dr. Vered Shwartz tells us there’s still room to grow into more culturally sound and aware models free of algorithmic bias.
If you’re using large language model chatbots like Chat GPT, it’s imperative to use an AI detection tool just to balance out the scales. You never know what these models are generating, whether they are misinformation, a form of hallucination, or a result of tricky training data.
AI Purity’s AI Text Detector shows you exactly which sentences are AI-generated or human-written and premium users can enjoy comprehensive reports that include a readability analysis and a similarity score. Discover peak AI detection today with AI Purity!