The Black Hearts and White Knights of Pre-Employment Testing

You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time. — Abraham Lincoln

Honest Abe must have known about hiring tests. You see, the results of a foolish test will always show-up as soon as someone takes a closer look. Foolish tests make bad candidates look good on paper, but crash on the job; and, they make good candidates look bad on paper … but never get a job offer.

Like Abe said, you can’t fool everyone all the time.

I call the difference between good and bad hiring vendors “Black-Hearted” and “White-Knights.” A White-Knight vendor presents thorough research showing its test predicts job performance with a very high degree of accuracy. White-Knight vendors always report jobs, demographics, charts, tables, subjects, correlations, and probability of being wrong. They are very dull. A Black-Hearted vendor presents happy-user stories that are very light on real data and heavy on nonsense numbers.

You might ask why they do this. Well, I guess they might be reluctant to say anything negative; they don’t know how to develop a professional quality test; they don’t know any better; they don’t care; or, a little of all the above.

Right or Wrong, Never in Doubt

I’ve talked to dozens of junk-test vendors. They are all totally convinced their test is the best thing to happen since tax deductions. Their flashy websites often contain pseudo-scientific brain wave charts, claims their training tests are also useful for hiring, and ridiculous stories of client effectiveness. But — and, here is the hitch — a closer look at the miracle test only they were smart enough to develop shows it doesn’t begin to meet professional test standards. Heck. After a few minutes discussion, you find out they never even heard of them! One proud vendor, for example, told me they couldn’t go back and fix their test because the shareholders would never stand for it! I’ll guarantee any client using that test ends up with clones — and not necessarily skilled ones.

Cloning high producers is one thing. Cloning employees with the same personality and motivation can be a disaster.

What’s the Big Deal?

The big deal is no test is 100% perfect, but White-Knight tests are very good at separating good employee/managers from bad ones. In fact, their developers go to great lengths to prove that very thing. Black-Hearted vendors don’t really care about all that messy science stuff. They just want to make money, and client-users, not test vendors, are always responsible for test use. A Black-Heart test vendor can sell its garbage all the way to the bank. And, they have become more prevalent since the Web opened cheap doorways into every office computer.

At its best, a Black-Hearted test will just waste your money. At worst, it will bloat your payroll with unproductive clones, create unnecessary legal challenges, impair product quality, produce obnoxious service people, hire/promote unqualified managers, irritate clients, and make you work harder. And that’s for starters.

Why Follow Professional ‘Standards?

It’s simple, really. When vendors follow the 1999 Standards for Educational and Psychological Testing, users are assured the test has stable items, measures something productive, is consistent over time, and predicts job performance. Isn’t that a nice thing to know when using scores to decide whether to spend 70 thousand bucks on a professional knowledge worker? Just try to get approval to drop that kind of money on a piece of equipment without doing some exhaustive leg-work.

Always remember there is a big difference between a test that shows differences between people and one that shows differences in job performance.

How to Spot Black-Hearted Vendors

Here are some tell-tale characteristics of Black-Hearted Vendors.

They divide workers into two groups and calculate average profiles. This is a really nasty technique because it looks legit, but nothing could be further from the truth. For example, averages hide individual differences; the groups might be divided based on opinion instead of objective standards; difference in scores might have nothing to do with job skills; differences might have occurred by chance; and you have no assurance the test factors actually cause high or low performance.

Article Continues Below

They promote the “benefits” of comparing scores to national profiles. This is another example of junk science at its finest. As mentioned before, averages hide differences. Ask yourself, for example: do all people in XYZ job have the same profile? Are all people in the job-match profile fully skilled, highly productive producers? Do your own workers with the same profile perform differently? What is the individual range of scores in each job-profile? Do the profile factors actually cause job performance?

Their technical manual is filled with happy customer stories. Every test should come with a technical manual that contains an honest explanation of how it was developed, why its theory is important to job performance, results obtained from internal reliability data, demographic impact, and its ability to predict specific performance in various jobs. You need to know that stuff. A Black-Hearted test skimps on data, never reports the size of the studies, throws numbers around that look important (but aren’t), and fills in the blanks with stories from happy clients praising its virtues.

Another clue to a Black-Hearted test is an easy-to-read manual, few details, heavy self-praise, and referencing theories of long-dead people like Maslow, Jung, Freud, Aristotle, and Groucho Marx. These are all signs a Black-Hearted vendor has no idea what he/she is talking about. As Monty Python said in the search for the Holy Grail, “Run away!”

They proudly claim their test contains dozens and dozens and dozens of factors. This is truly amazing. You see, over the years, researchers have shown only about six factors are related to job fit and about five to seven are related to job performance. So a list of 30, 40, or 50 factors is utter nonsense. It’s a clear indication you are dealing with a Black-Hearted huckster. Furthermore, extensive research shows even the best list of personality factors is poorly correlated with job performance (i.e., only a few percentage points). Personality only adds useful accuracy when based on a thorough job analysis, and when used in combination with other validated tools.

Need Help?

Black-Hearted salespeople are very good. They know that clients are seldom test experts and won’t systematically compare their hiring test scores to job success. They base their decisions on personal trust. If every client was an expert, there would be few, if any Black-Hearted test vendors in the market.

What you can do to protect yourself, though, is to recognize the warning signs outlined above. Then, find someone who is an expert in psychometrics (i.e., the identification and measurement of human skills and factors) to review your tests and processes. There are not many of them around, but you can poke-around their professional website to find a few names. The right expert can help navigate the deep waters of testing. When you are done, they will have saved you a small fortune avoiding bad hires.

You might have been fooled once, but you don’t have to be fooled all the time.


8 Comments on “The Black Hearts and White Knights of Pre-Employment Testing

  1. Nice job Wendell. I have spent 12 years battling sub par pre-hire assessments and am continually frustrated by the fact that most of my clients have no ability to discern the good from the bad. I am very open to multiple methodologies and as you know remain very open minded to new ideas and technology. But none of this can hide the fact that at the core, good measurement science must be adhered to. It is truly sad when experience with a sub par vendor then sours a company on the whole concept of employment testing. part of the blame is on the end users because they often do not see the value in the proper set up and evaluation required to make pre-hire testing work for them. My mantra is and will remain- No free lunch!

  2. Wendell,
    Thank you for this article. I deal with this issue every single day. The tests and methodology we use are highly validated and patented. This article should be posted on our home page! Please keep spreading the word and warn people that it may not only be bad practice it may also get their company into a lawsuit.

  3. Good stuff! Something every recruiter, hiring leader and HR manager should read and own (3-5 times).

  4. As always, love your work, Wendell. Just disappointed that you couldn’t find more for one of my favourite Wendellism’s about getting a chicken to climb a tree.

  5. Great post Wendell. I’ve always said if these tests were all so great there wouldn’t be something like 6000 of them on the market. There are this many because of the nation of sheep attitude we have. Everyone is looking for a quick fix…

  6. Thanks Dr. Williams. ISTM that while many recruiting assessment test vendors are borderline charlatans, so are many of the people who sell other kinds of recruiting paraphernalia, and as for those of us who make recruiting prognostications: well, we were created to make astrologers look good…


    Keith “No Snake Oil for Me Today” Halperin

Leave a Comment

Your email address will not be published. Required fields are marked *