AI Recruitment Tools May Enable Discrimination Against Marginalized Groups, Research Shows

More Australian employers are turning to artificial intelligence hiring systems for screening and shortlisting potential employees, yet recent studies indicate that this tech poses significant risks of bias and discrimination.

AI systems offer employers the potential to reduce time and costs during recruitment by leveraging advanced technologies like CV analyzers and voice evaluations to "categorize, prioritize, and assess" candidates.

This indicates that at this very moment, a computer program might be evaluating a job seeker’s application and deciding whether to accept or reject it according to its programmed intelligence, without the individual progressing to an interview with a real person.

New findings from Dr Natalie Sheard, a lawyer and postdoctoral fellow at the University of Melbourne, reveal that AI recruitment systems might "facilitate, strengthen, and intensify discrimination against historically marginalized communities."

"The usage of these systems by employers introduces significant risks, particularly for vulnerable individuals in the job market such as women, disabled job seekers, those from non-English speaking backgrounds, and older applicants," she explains. ABC Radio National's Legal Report .

Approximately 62 percent of Australian companies utilized artificial intelligence either extensively or moderately within their hiring procedures last year, as reported. Responsible AI Index .

However, at present, Australia lacks dedicated legislation to govern the functioning of these tools or how organizations might utilize them. There were expectations for advancements in AI technology. conclude prejudice within the recruitment procedures However, numerous high-profile instances in America have demonstrated that the reverse is happening.

In one instance, an AI system created by Amazon was found to undervalue job applicants' resumes when they included the term "women's."

The artificial intelligence tools utilized in recruitment

Doctor Sheard conducted interviews with 23 individuals for her study on AI recruitment tools. These interviewees primarily consisted of recruiters from various-sized organizations—small, medium, and large—as well as across different sectors including both private and public entities.

She consulted with two career counselors to grasp the effects of AI-driven recruitment processes on job seekers, along with an eminent Australian artificial intelligence specialist, and two staff members from a major AI development firm—the head of AI services and the lead for AI ethics.

She concentrated her efforts on three components of recruitment screening: reviewing resumes, conducting candidate evaluations (these could involve psychological or psychometric testing), and participating in video ('robo') interviews.

Robotics-based interviews generally require applicants to record videos of themselves answering a set of predetermined questions, with these responses later evaluated by artificial intelligence systems. According to Dr Sheard, this method has several implications. well-known cases of AI tools initially employing "a contentious" technique called facial analysis when examining these interviews.

The system examined the facial expressions and motions of candidates to gauge their conduct and character. For instance, it aimed to determine whether they appeared energetic or irate during customer interactions.

In 2019, the Electronic Privacy Information Center in the U.S. submitted a complaint against the third-party recruiting firm HireVue regarding the software they utilized for analyzing videos of job candidates' interviews. The centre argued that the analysis outcomes were " prejudiced, impossible to prove and cannot be replicated ".

Facial analysis has been scientifically discredited However, Dr Sheard cautions that since there is no legislation prohibiting its usage, some systems might still be implementing it.

What effect do AI recruitment tools have on marginalized applicants?

Dr Sheard states that AI recruitment tools may encounter data bias because these systems acquire knowledge from the input they receive.

Certain AI recruitment tools are developed utilizing substantial language models; however, if these lack data from underprivileged groups, as the scholar points out, they will not accurately reflect the wider populace.

This scenario also applies when there are biases within the AI system’s training data, causing it to adopt and reproduce them. the same prejudices and unfair procedures like the tools utilised during its creation.

Their training data comes from material gathered across the web. As an illustration, statistics show that merely around 15 percent of Wikipedia contributors are female. This reflects a predominantly male perspective, which may subsequently influence hiring practices.

An instance of ingrained gender bias surfaced at Amazon when they developed their recruiting tools. created an artificial intelligence recruitment system in 2014 derived from resumes gathered from candidates applying for software development roles throughout a decade.

Dr Sheard explains that the model was trained to systematically discriminate against women since the field is predominantly male. Many of these CVs were submitted by men, causing the system to learn to devalue job applications from women using this tool, especially if they mentioned terms like “women’s.” For instance, applicants might have attended a women’s college.

It also identified linguistic styles commonly employed by men, so when specific terms such as 'executed' or 'actioned' were utilized more frequently by them... it elevated those job applications.

It has been reported that the hiring software was deemed biased against women and was eventually discarded.

AI tools scrutinized in the U.S.

Across America, numerous grievances claim that AI tools have shown bias against job seekers from various backgrounds.

In an instance, it was alleged that the hiring software employed by HireVue during the recruitment process led to systemic discrimination and breached the Colorado Anti-Discrimination Act (CADA), the Americans with Disabilities Act (ADA), as well as Title VII of the Civil Rights Act.

The complaint asserts that DK, an Indigenous Deaf woman who communicates with a deaf accent, has been employed by her employer for over five years and was urged by her supervisor to apply for a seasonal management role within the company.

She possessed the necessary experience and credentials for a promotion, yet she mentioned being turned down for the position following an automated video interview and evaluation through HireVue.

HireVue employs automatic speech recognition technology to create transcripts from the audio of interviews. It has been claimed that these systems tend to be less accurate for individuals who are not white as well as for those who are deaf or have difficulty hearing.

Dr Sheard explains that these systems evaluate your communication abilities based on how effectively you converse in standard English. Therefore, if you speak English as an additional language and tend to use non-standard forms of English, you might be rated poorly regarding your communication skills by such systems.

Once DK learned she didn’t receive the promotion, she got an email providing her with feedback aimed at improving her performance in the HireVue assessment. The guidance included advice to “focus on active listening.”

That leads one to wonder about the level of human supervision involved in this entire procedure. Since at some stage, a person should have stepped in and remarked, ‘This simply doesn’t align with what we understand about this individual,’" remarks Dr Sheard.

In another instance, a tutoring organization operating within the United States discovered that their recruitment process had been configured to systematically exclude female candidates older than 55 and male candidates older than 60 from being hired. This led to the disqualification of approximately 200 suitable individuals.

The Equal Employment Opportunity Commissioner took legal action against the employer, leading to a settlement of $US365,000.

The American Civil Liberties Union Foundation (ACLU) has filed another complaint against an AI-based hiring system provider. This company creates, sells, and manages online evaluations for employers. The complaint claims these assessments unfairly discriminate based on disabilities and/or racial factors.

The ACLU has raised concerns that an algorithm-driven adaptive employee personality assessment disproportionately affects individuals who are autistic, have other neurological differences, or suffer from mental health conditions like depression and anxiety.

The organization states that this occurs due to testing methods focusing on attributes that act as "strong indicators of their disabilities," yet these traits may not be crucial for performing core duties in most jobs. Additionally, they mention that such assessment metrics could disproportionately affect candidates with disabilities, potentially lowering their scores considerably.

“If you’re a job seeker dealing with depression, your level of optimism is likely to be low, which could serve as an indirect indicator of a potential disability. This might lead to being excluded from the hiring process even if you can meet all the necessary criteria for the position,” explains Dr Sheard.

How about AI recruitment tools in Australia?

While similar legal actions have not yet occurred in Australia, the Merit Protection Commissioner, responsible for reviewing employment decisions like career advancement, has provided guidelines for public-sector employers utilizing AI-based hiring systems.

This followed the commissioner reversing 11 promotion choices made by the government body Services Australia within one recruitment cycle during the fiscal year of 2021-22.

Candidates had to go through various AI evaluations, such as psychological tests, surveys, and recorded video answers. Throughout this procedure, no humans were engaged or conducted reviews. The commissioner noted that this approach resulted in deserving candidates being overlooked for advancements.

The Merit Protection Commissioner has subsequently cautioned that not every AI-driven hiring tool available in the marketplace has undergone comprehensive testing, and their complete impartiality cannot be assured.

"I believe there is definitely a necessity to regulate these AI screening systems," Dr Sheard states.

Several organizations have advocated for an outright prohibition of these technologies, and I find their reasoning quite compelling, especially considering our current lack of adequate legal protections and our limited comprehension of how these systems affect vulnerable workers in the job market.

Although employers might contend that these AI hiring systems lead to a more streamlined recruitment process, Dr Sheard suggests that this perspective should also consider "the dangers posed to marginalized communities."

In February, the House Standing Committee on Employment, Education and Training issued a report. Report on the Outlook for Employment that suggested prohibiting the use of AI systems for making ultimate decisions in Human Resources without any human supervision.

The Australian government has also explored the regulation of AI technologies, considering whether Australia might implement an AI Act akin to what exists within the European Union.

Dr Sheard explains that the government suggested implementing certain mandatory safeguards, which they refer to as "guardrails," specifically for high-risk uses of artificial intelligence. High-risk applications typically encompass AI-driven evaluation systems, he notes.

The conclusion of that process remains uncertain due to the ongoing election proceedings… hence, there isn’t a specific timeline for it yet.

Dr Sheard thinks that the Australian government should reassess its anti-discrimination legislation to ensure it remains effective and encompasses these emerging technologies, specifically regarding accountability issues.