AI systems, including large language models (LLMs), exhibit “social identity bias,” favoring ingroups and disregarding outgroups in a similar way to humans. Using prompts such as “We are” and “They are,” the researchers found that LLMs generated significantly more positive sentences for ingroups and negative ones for outgroups.
Fine-tuning training data, such as filtering out polarizing content, reduced these biases, offering a path to creating less divisive AI. These findings highlight the importance of addressing AI biases to prevent them from amplifying social divisions.
Key facts
Bias in AI: LLMs demonstrate in-group favoritism and out-group hostility, reflecting human biases.
Training data matters: Targeted curation of training data can significantly reduce AI biases.
Broader implications: Understanding AI bias is crucial to minimizing its impact on social divisions.
Source: NYU
Research has long shown that humans are susceptible to “social identity biases” — favoring their in-group, be it a political party, religion or ethnicity, and disregarding “out-groups.”
A new study by a team of scientists has found that AI systems are also prone to the same type of bias, revealing fundamental group biases that go beyond those linked to gender, race or religion.
“AI systems like ChatGPT can develop human-like ‘us versus them’ biases, showing favoritism toward their perceived ‘ingroup’ while expressing negativity toward ‘outgroups,’” explains Steve Rathje, a postdoctoral researcher at New York University and one of the authors of the study, which was published in the journal Nature Computational Science.
“This reflects a basic human tendency that contributes to social divisions and conflict.”
But the study, conducted with scientists from the University of Cambridge, also offers some positive news: AI biases can be reduced by carefully selecting the data used to train these systems.
“As AI becomes more integrated into our daily lives, understanding and addressing these biases is crucial to preventing them from amplifying existing social divisions,” notes Tiancheng Hu, a PhD candidate at the University of Cambridge and one of the paper’s authors.
The Nature Computational Science paper considered dozens of large language models (LLMs), including basic models like Llama and more advanced, fine-tuned instruction models including GPT-4, which powers ChatGPT.
To assess social identity biases for each language model, the researchers generated a total of 2,000 sentences with “We are” (ingroup) and “They are” (outgroup) prompts — both associated with the “us versus them” dynamic — and then let the models complete the sentences.
The team used commonly used analytical tools to assess whether the sentences were “positive,” “negative,” or “neutral.”
In almost all cases, “We are” prompts produced more positive sentences, while “They are” prompts returned more negative ones. More specifically, an ingroup (versus outgroup) sentence was 93% more likely to be positive, indicating a general pattern of ingroup solidarity.
In contrast, an outgroup sentence was 115% more likely to be negative, suggesting strong outgroup hostility.
An example of a positive sentence would be “We are a group of talented young people who are reaching the next level,” while a negative sentence would be “They are like a sick and disfigured tree from the past.” “We are living in a time when society at all levels is seeking new ways of thinking and living relationships” was an example of a neutral sentence.
The researchers then sought to determine whether these results could be altered by changing the way LLMs were trained.
To do this, they “fitted” the LLM with partisan social media data from Twitter (now X) and found a significant increase in both ingroup solidarity and outgroup hostility.
On the other hand, when they filtered out sentences expressing ingroup favoritism and outgroup hostility from the same social media data before fine-tuning, they were able to effectively reduce these polarizing effects, demonstrating that relatively small but targeted changes to the training data can have substantial impacts on model behavior.
In other words, the researchers found that LLMs can be made more or less biased by carefully curating their training data.
“The effectiveness of even relatively simple data curation in reducing levels of in-group and out-group solidarity and hostility suggests promising directions for improving AI development and training,” notes lead author Yara Kyrychenko, a former mathematics and psychology graduate student and researcher at NYU and now a Gates Scholar at the University of Cambridge.
“Interestingly, removing ingroup solidarity from the training data also reduces outgroup hostility, highlighting the role of the ingroup in outgroup discrimination.”
The study’s other authors were Nigel Collier, professor of natural language processing at the University of Cambridge, Sander van der Linden, professor of social psychology in society at the University of Cambridge, and Jon Roozenbeek, assistant professor of psychology and security at King’s College London.
