Any recruiting software vendor can claim they leverage AI. It’s on you to figure out if they’re the real deal or not.
Once confined to the realms of science fiction, artificial intelligence (AI) has not only become a reality for recruiters, but a crucial tool in sourcing diverse candidates, making hiring decisions, and improving the candidate experience. Sixty-three percent of talent acquisition professionals say AI has already changed the way recruiting is done in their organization.
So if you’re in the market for AI recruiting software, you have many great options to choose from. But you need to be careful. Some vendors are taking advantage of the AI hype, preying on ill-informed recruiters with a product that’s nothing more than smoke and mirrors.
To help you avoid getting scammed or winding up with a system that doesn’t fulfill your needs, here are five tips for evaluating AI recruiting software vendors.
Tip #1: Always ask 'how'
AI vendors aren't too keen on peeling back the curtain on how their product actually works, whether out of fear that their secrets will be given away to competitors, or that they’ll be found out as frauds. All they want you to know is that data goes in one way, and all the amazing things AI does, such as identifying trends or mirroring human behavior, comes out the other.
This phenomenon is so common, there’s actually a name for it: “the black box problem.” It’s next to impossible for users to know what’s going on in the black box that is AI. And that presents real problems when you’re trying to figure out how an AI tool came to a certain result or what its limitations are.
You don’t need to be an AI expert that understands all the intricacies, but wherever possible, you should probe vendors on how their product works.
Here are some suggested questions to ask:
What data is used to produce these results? If they can’t tell you, that’s a red flag.
Where does this data come from? If the AI needs data you don’t have, or can’t produce, avoid it.
How often is this data processed and analyzed? The closer to real time that data is analyzed, the better.
How do you deal with false positives or false negatives? False positives/negatives are expected. It’s how the vendor adjusts the AI to produce better results in subsequent analyses that matters.
Are your algorithms open source or proprietary? If they’re open source, what is the source? There is no right answer here, but if they use an open source algorithm, you can do some research to see what AI experts have to say about it.
It’s unlikely that vendors will tell you all of the ingredients in their AI secret sauce, but you should hopefully get some sense as to whether they’re actually leveraging AI in their product, or if they’re just saying they are.
Tip #2: Be careful with benchmarking
While it’s true that AI requires a lot of data to work, that doesn’t mean all of that data has to come exclusively from your company. Through what’s called “benchmarking,” recruiting AI vendors can anonymize and aggregate all of their clients’ data to train their AI and drive better, data-driven results.
Most often you’ll see this in compensation planning. When deciding how much to pay a certain role, AI can quickly analyze all of the relevant salary data in the vendor’s database to recommend what you should offer in compensation, based on what other companies are paying for that role in your area.
Benchmarking employee salaries in Zenefits (Source)
AI vendors love to tout how many clients they have informing their benchmarking capabilities. But the number of clients is less important than who those clients are.
For example, if you’re a midsize retailer, but an AI vendor’s clients are mostly enterprise-level manufacturers, a lot of that benchmarking data isn’t going to be relevant to your needs. Ask vendors about their clientele make-up to determine if their benchmarking capabilities will really be useful or not.
It’s also important to remember that it goes both ways—your data will be used by the vendor’s clientele for benchmarking. Though it’s anonymized, it’s still a good idea to ask what data is used and how it’s used to ensure you’re not breaking any employee privacy laws, or revealing enough to give away your identity to competitors.
Tip #3: Look for proven results
Vagueness is the ultimate red flag when it comes to evaluating AI recruiting software vendors. Beware of vendors that say they work with Fortune 500 companies or multinational clientele without naming actual names. Even recognizable logos could be a red herring, as vendors may have partnered with these companies for a pilot or a co-branding opportunity, without a full-fledged AI implementation.
What should you look for instead? Case studies. A published case study, when done right, can tell you a lot of information about the legitimacy of an AI vendor.
A good case study will tell you:
The actual name of the client and what type of business it is
A description of the problems the client was having and their desired use case for AI
The Xs and Os of how the AI tool was implemented and trained
The results of the AI implementation (with hard data)
The more case studies a vendor can share with you, and the more those clients in the case studies mirror your own business and recruiting AI needs, the better.
Tip #4: Know what a realistic timeline looks like
The instinct with any new technology purchase is to get up and running quickly. Time is money, right? And every second you have to spend setting up software is another second it’s not providing a return on your investment.
So when a recruiting AI vendor tells you their system is ready to go out of the box, or can be set up in minutes, it can be tempting to sign a contract right then and there.
Don’t fall for it.
The reality is that every AI implementation requires at least some customization—and a lot of data structuring and testing—before it can start working as intended.
A relatively simple AI application, such as an AI sourcing tool, might take three months of testing and tinkering to source candidates you’d actually want to hire. A more complicated implementation, such as a recruiting chatbot that interacts with applicants using natural language processing (NLP), could take up to a year before you’ll be comfortable going live.
Talk to vendors and go over a feasible implementation timeline with them to not only manage your expectations, but to also get a sense of how much support you’ll get from each vendor once implementation is over. If you don't have AI experts in-house, you don’t want to be left high and dry by the vendor once the initial setup phase is complete.
Tip #5: Do a trial run
Just like you should test drive a car before you drive it, so too should you test drive AI recruiting software. A free trial period can give you time to determine if the AI solution can really do what it claims.
A trial run can take many forms, but should be based around your desired use case and outcomes. For example, if your goal with AI recruiting software is to quickly find the best candidates amongst a pile of resumes, give the AI and a human recruiter the same dummy resume data to determine if they arrive at the same candidates, and if the AI does it faster.
A trial run with an AI recruiting chatbot designed to talk to and assess potential job applicants would look completely different. Either have a test applicant interact with the chatbot, or come up with a list of common questions or scenarios to pose with the chatbot to try and trip it up.
The hope with these trial runs is that you can discover major pain points to help you decide if you want to continue with that vendor or not.
Do your due diligence
The potential with AI for recruiting is only matched by its peril. Even well-intentioned uses of perfectly capable AI products can and will go awry, and introducing a bad product to the process will only exacerbate your problems.
Perhaps the biggest tip is to take your time. Talk to multiple vendors, do multiple demos, and ensure you have as much information as possible to make an informed decision. The effort will be worth it in the end.
From here, head to our recruiting software page to learn more about potential software options and read product reviews from real users.