Will AI give you the job? Automated hiring tools spark discrimination and secrecy lawsuits

A rise in lawsuits over AI use in employment decisions is raising questions about how companies hire and fire

For the last four years, Erin Kistler has applied for thousands of jobs at companies like Paypal, Microsoft and Netflix, only to find her résumé disappear into a black hole. A product manager with nearly 20 years of experience, Kistler believes she was qualified for every role, yet she never received a single interview

Now, Kistler is suing Eightfold AI, the Silicon Valley maker of hiring software used by hundreds of companies, including those where she applied, in a class-action lawsuit. The case, filed in January in California court, is one of the first to argue that automated screening functions as an undisclosed consumer report or applicant dossier, ranking job applicants on their likelihood of success without giving them the chance to see or challenge the results, according to Kistler’s legal team.

The lawsuit is one of several new legal battles over companies’ use of AI in employment decisions. Workers are also suing Meta over an internal AI system that allegedly targeted them for layoffs because they took parental or medical leave. And a recent lawsuit against IBM alleges that AI tools discriminated against older workers

Companies in the US are increasingly turning to AI to make faster and more efficient workplace decisions, often under the guise of objectivity. But experts who study this type of software say it can also introduce or exacerbate bias that could follow candidates across companies. Lawmakers are taking notice. The outcomes of the legal battles could shape future employment practices, transparency around them and the power candidates have in the process.

“There’s actually no law requiring a notice or disclosure of the use of these AI hiring systems,” said Ifeoma Ajunwa, a professor at Emory University School of Law and founding director of the AI and future of work program. “So companies are not necessarily telling workers when they’re being evaluated with AI.”

AI use in hiring is growing

Last year, 90% of employers used some form of automation in the hiring process, according to a report from the World Economic Forum. These tools range from basic filtering – like excluding candidates who don’t have a four-year degree or another prerequisite – to using AI to conduct assessments of a job applicant’s skills or even to perform the initial phone interview

In the case of Eightfold, the company bills itself as “the world’s largest, self-refreshing source of talent data”. It constantly updates an internal database with information from the résumés, LinkedIn profiles, and social media profiles of over a billion workers who have applied for jobs through its platform. Using this data, it uses AI to score applicants on a scale of 0 to 5, predicting how well they would perform in a given job. Ultimately, the score could prioritize some candidates over others for interviews.

“A large part of the problem is that job applicants don’t know … what the reports say,” said Rachel Dempsey, an attorney representing Kistler

Dempsey argues that job applicants deserve the same transparency into what goes into the hiring process that Americans already get in their credit reports. With that information, they can either dispute inaccuracies or work to improve on factors affecting their candidacy. But without any insight, it’s impossible to know if these algorithms discriminate against job candidates in explicit ways, Dempsey said.

“The concept of a black box is very scary,” she says

In a statement to the Guardian, Eightfold AI denied the allegations. “We believe the claims asserted are without merit and intend to defend ourselves vigorously,” an Eightfold AI spokesperson wrote in an email

In response to the lawsuit filed against IBM, the company denied using AI to automatically screen out candidates and in a statement said it “does not condone or tolerate discrimination of any kind.” Meta did not respond to a request for comment on its lawsuit

The fear of algorithmic blacklisting

Though employment software providers claim to free up hiring managers to spend more time with qualified candidates and remove human personal biases, experts disagree. They say AI can exacerbate biases like favoring workers of certain demographics

“As we’ve done more of the research on AI hiring systems, we actually see that they tend to replicate a lot of the same biases that human managers have,” Ajunwa said

Nearly all AI systems are trained to look for patterns, which can reinforce stereotypes or legacy biases. Amazon, for example, once used an AI tool that downranked women’s résumés because the company’s top performers were men. Amazon discontinued its use after discovering the issue. In other cases, AI hiring systems can introduce their own biases. In research for her book, The Quantified Worker, Ajunwa found that AI scored job applicants with southern accents poorly, as it struggled to understand them during voice interviews.

While human hiring managers can be unfair – choosing the applicant who was in the same fraternity, for example – “the magnitude of bias is even stronger [with AI],” says Xuechunzi Bai, assistant professor at the University of Chicago who has researched hiring bias

Bai co-authored a recent study where AI models were asked to make hiring decisions about fictional applicants assigned to made-up demographic groups called Tufa, Aima, Reku and Weki. After a few rounds of hiring decisions, the AI models made inferences about candidates that had nothing to do with their qualifications, in effect stereotyping them. If one Tufa was a good doctor, then Tufa candidates were more likely to be chosen as doctors, and Weki candidates were more likely to be hired as janitors.

The study found greater bias in AI than in human hiring decisions she examined last year. It also found that newer and more advanced models produced hiring decisions with more bias

“We were quite shocked to see this result,” Bai said. AI models are very good at solving problems where there is one objective solution, as in math or coding. “But … they’re getting worse in terms of exploring the other alternatives,” Bai said. “In the context of hiring, exploration is quite important, but these models are not trained to do it.”

The consequences of AI’s biases for an applicant can extend beyond one job opening. When a worker is rejected by a person, they can try again at a different company with a different hiring manager. But when a worker is rejected by an algorithmic system used across companies, the system “remembers the decision that was already made and makes it again for the sake of efficiency”, said Ajunwa. “In reality, you’ve been algorithmically blackballed.”

Because so many companies use the hiring software built on the same foundational AI models, the concern is that a negative flag could potentially make it much harder for a candidate to get a job at all, said Katie Creel, co-author of a recent study examining the risks of “algorithmic monoculture” in hiring systems. “People are going to be shut out of jobs more than they would have otherwise been,” she said.

Eightfold AI did not comment on the matter

Can AI be responsibly integrated into hiring?

While AI tools may save time or help test candidates’ skills, hiring managers shouldn’t allow AI to make decisions, said Iman Abuzeid, CEO of Incredible Health, an AI hiring platform used by more than 1,500 healthcare employers. Incredible Health offers an AI agent to perform phone interviews, but the AI doesn’t autoreject, score, rank or decide who moves forward

“It’s about collecting information from the candidates so a human is better armed to make the decision,” Abuzeid said

Abuzeid also said about 10% of Incredible Health’s AI interviews are audited for bias by a human, a process that’s becoming a legal requirement by a growing number of cities and states

In New York City, a law that took effect in 2023 requires employers using automated systems in hiring to conduct annual bias audits and notify candidates in advance about the tech’s usage. The law, however, only applies to software that “substantially assists” or replaces decision-making, creating a loophole for cases where humans are part of the process. Recent laws from Illinois and Colorado also prohibit employers’ use of AI tools that result in unlawful discrimination.

Increasingly, city and state regulations aim to make notifying job applicants that AI is being used a baseline requirement. But ideally, candidates would have transparency into what their algorithmic dossiers say about them, said Jenny Yang, a partner at Outten & Golden, the law firm representing the plaintiffs in the lawsuit against Eightfold AI

“This initial transparency is helpful in better identifying where there may be problems,” Yang said, referring to notifications that AI was used. “There’s a lot of growing concern among workers that they may be denied opportunity for reasons that no one understands.”

Explore more on these topicsShare
Reuse this content

Source: www.theguardian.com