Three AI tools, three different winners: Welcome to the recruitment casino
Yesterday's top candidate becomes tomorrow's reject. The same CV ranked #1 by ChatGPT gets #8 from Gemini and is completely lost by Claude. And 55% of applicants? They end up invisible to the recruiter at the other end of the sorting system.
In new research by Eunomia HR that tested how off-the-shelf AI tools screen CVs in real-world hiring scenarios. The results should worry anyone on either side of the hiring process.
The Reality Check Of The Modern Job Market
As someone currently navigating the job market myself, I've seen firsthand how every job they see on LinkedIn gets about 100 applicants in a day, with graduate jobs easily getting 1,000+ in a week. It is no surprise really when we are living through what the Financial Times describe as "a terrible year to graduate and find a job" with graduate hiring down 30%+ on the same time in 2024. This places a massive burden on recruiters to sort through poor applicants, AI generated slop CVs, blatant lies in job history and overseas applicants without the right to work in the role.
We've created an absurd arms race where candidates use ChatGPT to write applications that get screened by ChatGPT on the other side. it's AI talking to AI while humans sit on the sidelines hoping for the best. The bitter irony is that genuinely qualified candidates who write authentic applications are getting filtered out because they didn't use the right AI-optimised keywords, while the 'slop CVs' that hit all the algorithmic triggers sail through.
This has led to the proliferation of AI powered sorting tools to help these recruiters sort through their CVs and find the best applicants faster than ever. Yet recent research has exposed some issues in these new tools that could cause some of the best applicants to be lost in the sea of applications.
When Martyn Redstone's team at Eunomia HR ran 300 head-to-head resume screens using ChatGPT-4o, Gemini 2.0 Flash, and Grok 3, they discovered something alarming. The three AI models agreed on just 14% of their daily top-ten shortlists. Put three AI recruiters in a room with the same CVs, and they'll disagree four times out of five. These are the same backend models used to power the majority of HR hiring tools.
But it gets worse. The individual models reshuffled identical resumes by an average of ±2.5 rank places day-to-day, with one resume jumping from #10 to #1 in 48 hours using Gemini. The AI isn't just inconsistent between platforms, it's inconsistent with itself.
This volatility isn't happening in isolation. Recent research from the University of Washington found LLMs favoured white-associated names 85% of the time and never favoured Black male-associated names over white male-associated names. The ongoing Workday lawsuit, backed by the EEOC, suggests AI vendors may be held accountable for discriminatory screening practices. AI is only as accurate or as fair as the data it has been trained on. If a company has been discriminatory in the past, its AI tools will reflect that in their decision making on candidates.
Why Is This Happening?
The core problem isn't the technology itself, it's how it's being implemented into the hiring process. LLMs are pattern-matching machines trained on historical data fed into it during training. When that data reflects decades of hiring biases, the AI doesn't eliminate discrimination; it enforces it.
When Eunomia examined the reasoning behind AI selections, they found models recycled the same three phrases 96% of the time, signalling minimal deep resume comprehension. These tools aren't evaluating candidates thoughtfully. In reality they're performing sophisticated keyword matching dressed up as intelligence. The backend prompts that channel or direct how it makes a decision are not doing enough to correct this variance.
Most recruiters don't realise that AI models have randomness built in by default. The same prompt with the same CV can generate different results each time unless you specifically set the variance to zero, something most off-the-shelf implementations don't do. AI is new technology and there is little accurate and accessible information for recruiters to help them tackle these issues, let alone be aware of this happening.
The Human Cost For Job Seekers and Hiring Managers
For job seekers, this creates a nightmare scenario. You're not just optimising for one ATS anymore or set of keywords, you're trying to game multiple AI systems that can't even agree with themselves. Your application might be perfect for the role but invisible to the algorithm because you used "managed" instead of "led" or went to the wrong university.
Fifty-five percent of resumes in Eunomia's talent pool were never shortlisted by any model on any day. These weren't unqualified candidates, they were HR Business Partners applying for an HR role. They simply fell into an algorithmic blind spot. By no fault of their own and with no feedback ever being given for failed applicants, set to wonder forever where they went wrong.
For companies, the risks are equally severe. Beyond the legal implications of discriminatory hiring practices, you're potentially filtering out your best candidates. The person who could transform your business might never make it past the first screen because an AI model had a different random seed that morning. Here's the uncomfortable truth: We might be better off with human bias than algorithmic bias. At least humans can be held accountable.
The Path Forward: Human-in-the-Loop
The solution isn't to abandon AI screening, it's to stop treating it as a standalone gatekeeper. Amazon learned this lesson the hard way when their AI recruiting tool penalised resumes containing words like "women's basketball" after being trained on predominantly male applications. Research the tools you are using and do not trust them blindly. Try and get a better understanding of how it is filtering your candidates.
Successful implementation requires a human in the loop when the AI is making its decisions. An easy way to implement this is to work with the AI on the first 10 or 20 applications so it can learn from you on what you are looking for in a role. This approach combines automation with human oversight to ensure accuracy and compliance, with humans providing critical feedback, refining outputs, and correcting biases.
Here's what I recommend:
For Recruiters:
- Set temperature or variance to zero if the functionality exists on your tool for consistency
- Use structured prompts with clear scoring rubrics
- Treat AI as a first-pass filter, not final arbiter
- Regularly audit outputs for bias patterns
- Maintain diverse training data
For Job Seekers:
- Include multiple keyword variations for key skills
- Focus on quantifiable achievements but avoid the meaningless %s
- Keep formatting simple and parse-friendly, most ATS prefer Docx over PDF
- Apply directly when possible to bypass initial screens
- Don't assume rejection means you're unqualified
- Get a referral or message the hiring manager
The Bottom Line
These tools are too useful to ignore but too unstable to trust blindly. We're at a turning point. We can either sleepwalk into a future where algorithms perpetuate discrimination at scale, or we can build systems that augment human judgment rather than replace it. The technology isn't inherently good or bad but how we deploy it will shape who gets opportunities and who gets left behind.
Written by Naoise Law - LSE MSc Graduate specialising in AI.
See my website Naoiselaw.com for all my blogposts and portfolio or chat with me via AI chat about my experience or how I could help your business today.
You can email me at Lawnaoise@gmail.com or message me here on linkedin.