Leaky Hiring Tests

Is your hiring test leaky? I mean, does it pass too many unqualified candidates? I recently did a search for “hiring tests.” Google turned up 84 million listings, Yahoo about 70 million, and Ask … well, I stopped counting after 106 pages. By any standards, selling “hiring” tests is a big business. But, there is a big difference between a good hiring test and a leaky one.

photo by Harry Wood

Leaky tests pass-through marginal performers and, depending on the type of job (unskilled, semi-skilled, professional, managerial) they can cost organizations between 10% and 50% of annual payroll. In other words leaky hiring tests can be the single most expensive mistake organizations can make.

Here are some common-sense guidelines to dry-up leaky tests.

Self-Reported Data Leaks

A leaky hiring test often begins by asking employees to answer items describing him or herself. It might be given to your own employees or to people around the country with the same job title. Scores are collected, averaged, and used to screen job candidates. Sounds good, right? Wrong.

A couple things happen when we are asked to describe ourselves. In the best case, scores are idealized self-presentations … how we want people to see us. In other cases responders might be completely out of touch with reality or just faking it. Even when tests include an internal truthfulness scale to flag inconsistent answers, self-reported information exists purely in the mind of the candidate.

Averaging Leaks

Averaging scores is a bad thing. Averages can describe groups, but they cannot describe individuals. For example, you might believe Californians are flaky and southerners are rednecks. But when you get to know an individual Californian or Georgian as a human being, you usually learn he or she does not match the average stereotype. Accordingly, when people are assigned to performance groups, it’s rare for any individual to match their group average. Group test scores and individual scores are two entirely different things.

Score-Setting Leaks

Passing scores are supposed to predict good performance. Failing scores are supposed to predict bad performance. If you set cut-scores based on averages of high-performing groups, you learn nothing about the low group. In fact, both groups may have a lot in common! Hiring accuracy depends on knowing what makes them different.

Cause or Pause Leaks

Not all test factors cause performance. For one thing, many tests like the MBTI, the DISC, and others were all developed to measure on aspects of normal personality. They might work in a communication workshop, but not all normal personality factors apply to jobs. Research shows only three factors correlate with job performance and six to job fit. The rest are either irrelevant or overlapping.

Be wary of tests that ask a few questions and use the answers to comprehensively describe behavior. Watch out for tests that try to describe every aspect of human personality. Finally, avoid like the plague any test that comes without objective supporting data showing factor scores directly lead to (or somehow affects) job performance. If it does not cause, you need to pause.

Article Continues Below

Assumption Leaks

How many people do you know claim they are intelligent, but aren’t? Claim they are good with people, but aren’t? Claim they are organized, but aren’t? Research shows there is almost no correlation between scores on a personality test and skills such as building interpersonal relationships, solving problems, or ability to learn. Organizations that trust personality scores to predict actual skills are pre-destined to make mistakes.

Purple Dinosaur Leaks

I love you/ you love me/we’re a happy family/ with a great big hug and a kiss from me to you /won’t you say you love me too? … Ratings are like Barney the Dinosaur relationships: grouping is often based on who the manager likes best. Whenever you ask managers to rate employees’ performance, the scores will probably underemphasize actual job skills and overemphasize sociability.

Performance Confusion Leaks

Even assuming your managers are blunt-force honest, and ratings are made on jobs where numbers can be tracked (e.g., customer service, production, sales, and so forth), what happens when an employee such as a Customer Service Representative is rated both on quality and number of customers served (i.e., the two are usually inversely related)? If management cannot decide what’s most important for an employee to accomplish, then what exactly are you supposed to measure?

Employee Similarity Leaks

All those high and low performers you are studying belong to a special group: sufficiently skilled to remain employed. In other words, there really may not be enough difference between one employee and the next to measure a useful difference in scores. If you have no identifiable and trustworthy standards for comparison, then how can you set scores?

Same but Different Leaks

Some vendors offer norms for drivers, salespeople, emergency medical technicians, and so forth, implying they can be used for selection. But look at your own workforce. Are all your employees with the same title high performers, do their jobs require identical tasks, do individuals match the group average, and do high and low performers have score differences? Playing the job-norm card is an effective way to market a leaky test but it does not help the test user make better hiring decisions.


Leaky tests are great examples of junk science. I advise my clients to take them with a grain of salt. Leaks come from many sources: restricted score range, conflicting metrics, useless test factors, self-report errors, overemphasizing manager bias, underestimating job skills, trusting personality to predict actual skills, comparing individuals to group averages, and assuming job titles all involve the same skills. Considering the cost of water these days, don’t you think it’s a good idea to tighten-up the faucets?


14 Comments on “Leaky Hiring Tests

  1. Great summary of the pitfalls to be avoided. I think the one that sticks out in my mind are tests that use minimal info to provide a dossier about every aspect of a person’s potential. I have seen adjective rating checklists with 20 some items that the test company claims can tell you all you need to know about a candidate both pre and post hire. These tests are very popular because they are easy to use make sense. An influential high schoolteacher of mine once told us that if we could remember 1 thing from his class for the rest of our lives it should be: “there is no free lunch” I still fall back on this almost daily. Want to really predict who will excel at a job? You’d better do more heavy lifting than a 20 item checklist.

    At the end of the day, those considering using assessments need to do their homework and decide on some clear parameters around their investment and commitment to making it work for them. Its OK to use a profiling assessment as long as you dont expect it to deliver more then minimal accuracy.

    Buyer Beware- another ancient saying that still applies today

  2. Great article. I often tell our customers that a personality profile will tell you that your worst salesperson is an optimist. “We’re going to close that deal next month for SURE!”

    At Hireology, we’ve built our assessment model around a series of behavior-based interviews that truly do predict the candidate’s chance of success in a role. The key? Each job profile is unique, based on input from the manager leading the team. This way, two similar jobs (Customer Service Manager) with the exact same job responsibility (customer service) can have widely different success drivers due to the differences between team culture and management style.

    Would enjoy hearing your thoughts on our approach.

  3. Dr. Williams,
    I amdealing with the other side of the coin. Assessments that are filtering out qualified people. A major challenge. I have to educate managers that it is a tool not a veto.

    Rich Goldberg

  4. @Carol – We’ve built the model to minimize the chance that a manager can “get it wrong.” There are certainly some managers who fail to understand certain aspects of a role, but we can’t do much to change that. (As Wendell rightly argues, when you dumb down an assessment to be 100% “manager-proof,” it ends up being meaningless.)

  5. @Rich…that can be a problem…If you have done everything right, this indicates your candidates don’t have a full set of job skills..In that case, I suggest you revisit and re-verify the job requirements and business necessity, re-check your validity numbers, adjust cut-points, prioritize skills, decide what you must have, and what you can live without.

  6. A tough part of doing what I do is explaining the fact that predictive hiring science is never even close to 100% accurate. We are using data to make predictions that are tied to probability and in which there are many many sources of noise and error. I remind myself that the beautiful thing about Psychology is that humans are all unique and are not even close to consistent in behavior across similar situations.

    It is hard to both sell the value of a test and explain away the fact that there is some error because the errors have names and families. At the end of the day, we have to accept that sometimes we have false positives and false negatives. The goal is to minimize these instances by careful work and checking via the methods WW mentions.

  7. “…marginal performers and, depending on the type of job (unskilled, semi-skilled, professional, managerial) they can cost organizations between 10% and 50% of annual payroll.”
    Where does this figure come from? How is it defined and calculated: “marginal” compared to whom? Is this range valid for all types of firms at all times?



  8. @Keith…man you have questions!! I’ll send you some separate papers by email…but,for the purposes of this forum, the original reference was cited in an old ERE article I wrote a few years back (maybe you can find it)…”Marginal” means anyone performing below the group average, but not so low he or she gets fired. People often call this, “terminate and stay”. Wayne Cascio is one author who writes extensively on this subject…Of course, as a job becomes more critical, the costs go up. I’m sure the percentages will change from organization to organization (it is most obvious with salespeople)…but the bottom line is leaky hiring is EXPENSIVE!!

  9. Great analysis and summary as always. Regrettably, your article is so thorough it becomes a checklist for the naysayers for not using pre-employment tests for hiring and screening candidates. Employee selection is not a task, it’s a responsibility. It’s not simple, it’s complex. And unfortunately too many business owners and managers don’t want to tax themselves with doing what they need to do to hire and retain the right people. But as your title suggests, these are leaks and leaks can be plugged. I hope most people get the message and don’t use your sound advice as an excuse.

  10. Great points, Wendell. Your section on Self-Reported Data leaks is especially on-the-money. Just because a candidate describes himself as intelligent, outgoing, and highly objective, for example, doesn’t mean he actually IS. We can’t make hiring decisions based on who a candidate tells us he THINKS he is versus who he ACTUALLY is.


Leave a Comment

Your email address will not be published. Required fields are marked *