Stanford Study: AI Hiring Algorithms Systematically Reject Black and Asian Candidates

Researchers from Stanford, Chapman University and Northeastern University analyzed 4 million job applications submitted by 3.4 million people and found that a widely used AI candidate-assessment tool systematically steered Black and Asian applicants toward positions where they were rejected more often than other candidates. The study, published under the title "Algorithmic Monocultures in Hiring" and presented at the ACM FAccT conference in Montreal, is the largest analysis of real-world hiring data of its kind to date.

How the Algorithm Worked

The pymetrics tool did not analyze resumes in the traditional sense. Candidates went through a series of short online games measuring, among other things, risk tolerance, information-processing speed and propensity for altruism. Based on these results, the algorithm generated a profile that employers compared against a profile deemed desirable for a given position. Results from these tests were retained for up to 330 days, meaning a single unfavorable assessment could affect the same candidate's subsequent applications for nearly a year.

The research team, which included Rishi Bommasani, Sarah H. Bana, Kathleen A. Creel, Dan Jurafsky and Percy Liang, checked whether a given group of candidates was recommended for a given position less often than 80 percent of the rate of the most frequently recommended group. This is the standard "four-fifths rule" used in US anti-discrimination law to assess whether a given employment practice has a disproportionate impact on a particular group.

The Blackball Effect

The most troubling element of the study is a phenomenon the authors call "algorithmic blackball", or systemic exclusion. Because many employers rely on the same technology vendor, a candidate rated poorly by one algorithm had a higher chance of being rejected at other companies using the same tool as well. According to the study, about 4 percent of candidates who applied to ten positions were rejected by all of them, and bringing the probability of such systemic rejection below 0.1 percent would require submitting as many as 25 separate applications.

Algorithmic monoculture, to me, is any circumstance in which similar outcomes occur because of algorithms - Sarah Bana, study co-author, Chapman University

This phenomenon differs from classic discrimination by a single recruiter. When dozens or hundreds of companies rely on an identical or very similar algorithm from the same vendor, a flaw or bias embedded in one system spills over across an entire segment of the labor market at once. Researcher Kathleen Creel described the phenomenon as a situation in which "the same algorithm dominates a given sector", which turns a single design flaw in a tool from an isolated incident into a structural barrier for entire groups of candidates.

What the Study's Authors Say

Rishi Bommasani, one of the researchers from Stanford University, cautioned that the study's findings should not lead to a wholesale rejection of artificial intelligence in hiring, but rather to greater caution in how it is deployed.

I don't think we want to discourage the application of AI in this domain, but recognize the stakes are high and be judicious in the approach - Rishi Bommasani, Stanford University

The authors stress that the problem lies not solely in the algorithm itself, but in the way companies deploy and audit it. Analyses of hiring-tool effectiveness are typically conducted at an aggregated level, for an entire company or industry, which masks disparities that only become visible when individual positions are examined. A position can look neutral in overall statistics while systemically rejecting a specific group of candidates.

What This Means for Employers

US employment lawyers commenting on the study's findings note that responsibility for an algorithm's discriminatory decisions rests with the employer, even when the tool was supplied by a third party. Recommendations for companies include conducting impact analyses at the level of individual positions, requiring transparency and validation data from technology vendors, introducing human oversight of rejected applications, and maintaining documentation on the selection and monitoring of AI tools.

For Polish companies, which increasingly rely on automated resume screening and online competency tests, the study is a warning sign even though it concerns the US labor market and its local anti-discrimination rules. Hiring tools built on similar candidate-assessment mechanisms are appearing in Poland with growing frequency, and the EU's AI Act classifies systems used in recruitment as high-risk, which brings additional audit and documentation obligations.

The study was based on pymetrics data from 2018-2022, before the company was acquired by Harver, so it does not directly reflect the tool's current version. The authors caution, however, that the mechanism of assessment homogenization, a situation in which many employers rely on the same technology vendor, remains a present-day risk regardless of the specific product or company involved.