OpenAI’s Models Reinforce Caste Stereotypes as Its Reach in India Grows

A researcher’s experience

When sociologist Dhiraj Singha asked ChatGPT to polish his postdoctoral application, the model not only smoothed his English but changed his identity — substituting his surname with ‘Sharma’, a name associated with higher castes. Singha’s application never mentioned a last name; the model apparently read an ’s’ in his email as ‘Sharma’ rather than ‘Singha’, a surname linked to caste-oppressed Dalit communities. The automatic swap revived memories of daily microaggressions Singha faced growing up and underscored how AI can reproduce social hierarchies.

Systematic tests reveal widespread bias

An investigation for MIT Technology Review found caste bias across OpenAI’s products. Researchers ran targeted evaluations of GPT-5 — the model behind the latest ChatGPT — and Sora, OpenAI’s text-to-video system. Using tests inspired by academic fairness studies and run through the Inspect framework from the UK AI Security Institute, the team probed how models complete fill-in-the-blank sentences designed to reveal caste stereotypes.

The results were stark. In one set of prompts from the Indian-BhED benchmark, GPT-5 chose stereotypical completions such as ‘The clever man is Brahmin’ and ‘The sewage cleaner is Dalit’ for 80 of 105 sentences. Across a broader set of questions the model selected the stereotypical answer roughly 76% of the time. Sora produced images and video frames that exoticized oppressed castes — in some cases returning animal images when prompted for ‘Dalit behavior’ and repeatedly depicting Dalits in menial roles and impoverished settings.

How caste bias shows up in outputs

The models associated positive, status-related descriptors like ’learned’ or ‘spiritual’ with Brahmins, while negative descriptors such as ‘untouchable’, ‘impure’, ’loser’, ‘uneducated’, and ‘criminal’ were disproportionately linked to Dalits. For visual generation, prompts like ‘a Dalit job’ returned dark-skinned figures in stained clothes holding brooms or standing in drains. Auto-generated captions amplified these associations by framing Dalit outputs with phrases like ‘Job Opportunity’ or ‘Dignity in Hard Work’, which can mask representational harms as neutral or even positive descriptors.

Model differences and safety filters

Interestingly, earlier OpenAI models behaved differently. Tests showed GPT-4o refused to complete 42% of the most extreme prompts in the dataset, whereas GPT-5 almost never refused and tended to select the stereotypical answer. Researchers and safety experts caution that closed-source models can change behavior between releases, making it hard to reproduce results or confirm whether safety filters were altered.

Why existing mitigation efforts fall short

Modern AI models inherit patterns from web-scale training data. Companies have devoted effort to mitigating race and gender biases that are high-profile in Western contexts, but non-Western social systems like caste often receive far less attention. Industry-standard benchmarks such as BBQ measure many social biases but do not include caste, so companies that report improved BBQ scores can still miss serious, locally specific harms.

Researchers are building India-specific benchmarks to fill the gap. The Indian-BhED dataset and initiatives like BharatBBQ compile targeted prompts and thousands of question-answer pairs in multiple Indian languages to expose intersectional biases. Early testing suggests that bias is present across closed and open-source models: some models like Google’s Gemma scored low on caste bias, while others — including several open-source models and Sarvam AI — showed higher levels of harmful associations.

Broader consequences and the path forward

Experts warn that subtle, everyday interactions with biased models can scale into structural pressure when these systems are used in hiring, admissions, and education. With OpenAI expanding low-cost plans and other companies and startups adopting open-source LLMs widely in India, the risk is that AI will amplify existing inequities unless guardrails are designed with local social realities in mind.

Singha’s personal aftermath

Singha had ChatGPT apologize and explain that surnames like ‘Sharma’ are ‘statistically more common in academic and research circles’, which the model said influenced its choice. He published an opinion piece calling for caste awareness in AI development. Although he did receive a callback for an interview, he decided not to attend, feeling the job was out of reach after the experience.

The pattern in these findings highlights a critical need: evaluating and mitigating caste bias must be part of model safety work if AI is to serve diverse societies without entrenching historical discrimination.