Vetting Your AI Recruiting Tools
After I pointed out the problems and regulatory problems with many of the AI tools that recruiters use, I got a few DMs. I decided to simply write a blog about what needs to be done so that I can provide a link to this blog instead of typing the same thing over and over.
The main problems with the AI tools are the same problems that many other selection tools have had over the years. Adverse impact is when a protected group is accidentally being discriminated against (e.g. IQ test scores are lower for males of African descent). Bias is baked into the tool because of the people used to test its effectiveness (e.g. incumbents used in the reliability and validity testing were all males under 30 years of age). Correlation is not the same as causation, so the tool is not accurately predicting future performance. Over-reliance on the tools results in a very homogenous (no creative tension or unique points of view) team. Vendors rely on meta-analyses that really only expose how bad the assessments are in an effort to sell you the “least worst” tool. (One of the most-quoted studies admits that the best tool, General Mental Ability tests, have a validity of .51, which is only .01 better than flipping a coin.)
Let’s go all the way back to 1978 and look at the absolute minimum acceptable standards for recruiting/talent acquisition tools - The Uniform Guidelines on Selection Procedures. These were designed to ensure equitable employment decisions.
Key Aspects of the Uniform Guidelines (UGESP):
- Scope: They apply to all selection procedures used to make employment decisions, including hiring, promotion, demotion, referral, and licensing.
- Adverse Impact: If a selection procedure has a disproportionately negative impact on a protected group, it is considered discriminatory unless validated. The "4/5ths Rule" (or 80% rule) is used as a guideline to identify potential, but not definitive, adverse impact.
- Validation Requirements: If adverse impact is found, the employer must prove the selection procedure is "job-related and consistent with business necessity". Technical standards for validation (content, construct, or criterion-related) are outlined.
- Record-Keeping: Employers must maintain records of the impact of their selection procedures on race, sex, and ethnic groups.
- Alternatives: Employers should search for alternative procedures with less adverse impact, if available.
The guidelines were adopted by the EEOC, Department of Labor, Department of Justice, and the Civil Service Commission. This is your bare minimum to avoid federal legal issues.
The goal is not to barely comply with laws and regulations. The goal is to accurately identify candidates that will succeed in the job (predictive). No outside vendor is going to have a tool that automatically works straight off of the shelf for the exact roles inside your organization. You need to validate that the AI vendor’s tool(s) comply with UGESP and effectively predict performance on the job.
The first CYA move is to require the vendor to provide you with all of the vendor’s own reliability and validity studies. An assessment is “reliable” if it is actually testing what you want to test. For example, are you testing ‘bias for action’ and ‘analysis paralysis’ or are you really testing ‘patience’ or ‘test-taking abilities’? An assessment is “valid” if it ties to actual job performance. For example, you might think your top salespeople are more assertive than the others - and you use an assessment for that - but, the real differentiator is perceived credibility in the eyes of the prospects. Vendors that dismiss legal scrutiny or who cannot produce the white papers and data need to be removed from consideration. If the vendor can explain model inputs, training methodology, data interpretation, and the tools’ limitations, they are still in the running.
Though the vendor has done their own research, you need criterion-related validity evidence tied directly to job performance for the role you are trying to fill. Use your current employees, but do NOT tell the vendor which employees are your top performers. Give the assessment to people who are currently in the job. Include a mix of top, average, and bottom performers so that you are testing for what separates the best from the rest. Have the vendor present the results of assessing each employee and have them tell you who is in what performance group. If they got it wrong, that tool will not help you identify potential high performers from the pool of job applicants. (Vendors hate this type of testing because it often exposes their ineffectiveness.)
Conduct your own longitudinal research. Map out the touchpoints in the sourcing, attracting, selecting, and onboarding/training processes. Look at the candidates’/new hires’ performance at each touchpoint. It should be possible to ‘grade’ candidates at each touchpoint and then validate which grades were most accurate in predicting performance. (Once the new hires have been on the job for 6, 12, and 18 months, how does their performance compare to the veterans?) Things you might discover include behavioral interview questions that are better at predicting future performance, managers that consistently want to hire poor performers (so that no one outshines them, perhaps), key moments in the job simulation assessment that are strongly tied to future performance, and/or trick questions that are meant to test creativity or performance under pressure that have absolutely no correlation to job performance - even though they greatly increase candidate anxiety and hurt your online reputation on Glassdoor and Indeed. (Google famously ran the numbers for years and found out that the brainteaser questions were worthless.) GPA, academic achievements, and the prestige of a university will all fail to predict future performance. Questions about how you ‘might’ handle a future situation are also not correlated to future performance, but questions about how you ‘actually did’ handle a past situation will probably correlate to future performance. (Past performance is a solid predictor of future performance.) When I have done longitudinal studies on future leader performance, learning agility and tolerance for ambiguity have both been excellent predictors of leadership success. Unfortunately, what you won’t find is whether you missed out on great candidates because you relied on keyword parsing scores to move your initial applicants into the next step in the selection process… You lost track of them when they were kicked out of the funnel.
You can include some AI (or other vendor-provided) assessments in the selection process as long as you create and monitor how those scores predict performance. Early on, it is best to keep those scores secret and don’t use them in the selection decision until you have enough data that shows that they are valid. There are no perfect assessments or processes because humans are extremely complex and cannot be boiled down to a few numbers. Just collect reams of data and look for adverse impact, bias, and other risk factors. That protects you and your employer. Also look for criterion-related validity evidence of actual future job performance. That makes you super valuable because you are constantly raising the bar and bringing in future stars.
The one AI tool that I suggest you try out is a chatbot. You have very little time, lots of jobs to fill, and even more candidates who all want to connect with you to make a positive impression and to get the status of their candidacy. Allow those candidates to self-serve via a chat bot that is tied to your FAQs - and that provides accurate answers (hallucinations in recruiting are VERY expensive). If the machine learning is awesome and the data security is legendary, then you might want to tie your chatbot to your funnel so that candidates can get their personal status. It will save you a lot of time and candidates will be much happier knowing that they are getting current, accurate information. Plus, it is available 24/7.