Promises, Promises: How to Identify a Bad Hiring Test (Part I of II)

testYou probably already know there are hundreds of self-report tests promising great hires. What you might not know is most of them are poorly designed junk. Why is this important to know? First, the test user — not the test vendor — is primarily responsible for test use. Second, junk tests hire too many wrong people and turn away too many right ones. Finally, if the first two do not bother your conscience, consider the cost of poor hiring practices is estimated between 20% and 50% of yearly payroll. So how do you identify the junk tests? Let’s start with a few basics.

What’s a Test?

A test is anything used to distinguish between a qualified and unqualified job candidate. It has questions. It has answers. And, it is scored. The most popular test (and one of the least accurate) is called an interview. Tests also come in other formats including application forms, resume reviews, candidate sourcing, web applications, pencil and paper questionnaires, and so forth. Everyone uses tests. Get used to the idea.

Better, Not More

In your heart of hearts, you probably already know that not all test factors are important to performance in all jobs. People who do the research consistently identify six job-fit factors, three job-performance factors, and three job-skill factors that make the performance short list. What about the rest? Overlapping or irrelevant to job performance. In fact, thanks to computers, we know that approximately 28,000 personality related descriptors collapse into only about 5-7 general factors. These are often loosely referred to as the Big Five: In other words, you can ask someone to complete a 28,000-item questionnaire, score their answers, and get only about five-seven common themes. More is not better; more is just confusing.

On the other hand, there are just three critical skill factors that consistently relate to job performance. These are mental horsepower, organizational ability, and interpersonal skills. It’s common sense. High performers are smart enough to do the job, self-organized, and have people skills. What about tests that ask a few questions and then produce long narrative reports? Well, I think you already know the answer to that question: it sells more tests.

How do you know which factors should be measured for your job? Isolate only the ones that directly affect high and low performance. Unsure about which factors or whose test to use? Stick with unstructured interviews. At least your mistakes will be randomized.

Hide and Seek

Do tests that rely on self-descriptive questions uncover hidden secrets? Consider this. You answer a few questions (i.e., Do you like being around people? Are you generally the life of the party? Are you outgoing? Do you easily talk with strangers?) and then someone adds up your scores. Miracle of miracles! Your personal report says you like being around people, being the life of the party, being outgoing, and communicate easily with strangers! Tests that just total up your scores and report back either your own answers or words that could describe most anyone should stay in the training room. Only a well-designed hiring test can identify patterns that might hinder or facilitate work.

Performance or Preferences

Self-reported tests come in a variety of applications. Clinical tests evaluate dysfunctional behavior so professionals can effectively treat patients (e.g., MMPI). They often contain personally invasive questions about sexuality, violence, psychopathic thoughts, and anti-social behavior (e.g., bad mojo unless you are a licensed professional treating a mental patient).

Workshop tests evaluate differences between people (e.g. DISC, MBTI, ACL, and so forth). They often contain factors the author thinks are interesting. Many of the most popular workshops tests are based on old, obsolete research. Their authors are long gone. One test I recently reviewed even quoted Hippocrates as a source. I’m surprised they did not reference the four-humors medical theory as well.

Article Continues Below

Hiring tests are usually deadly dull. Casual readers should immediately be suspicious of any vendor who claims their test invokes lost mysteries, the author was the founder of a mystery school, or alien astronauts hand-delivered this wisdom to earth 1,000 years ago. Also beware if the vendor is well-known for its training workshops, is not a trained selection test expert (there are only a few thousand in the world), or will sell the test to anyone without examining their educational pre-qualifications. That should be your second warning not to use it for selection.

I find it’s fun to ask vendors, “Was your test specifically developed to predict job performance?” Legitimate ones are quick to produce pages and pages of studies and data. Wanna-bes either get surly or argue, “No, but the scores can be useful when making a hiring decision!” Huh? On one hand, they say their test does not predict performance, but on the other hand, they say it can be “useful” making a hiring decision. Whoa! Did someone put funny stuff in the brownies? If a specific test cannot tell me about a candidate’s potential job performance, am I not wasting my organization’s money and taking considerable risk with someone’s welfare?

Legitimate hiring tests are not only designed to predict job performance, they include boring job-related factors and studies that vendors are only too happy to share. By the way, if you don’t understand the mumbo-jumbo, then hire a selection test expert for a few hours to review it. It’s cheaper than a bad hire. I just finished reviewing a vendors’ report that contained so many errors, wrong terms, bad science, and misinformation that I considered submitting it to Saturday Night Live.

Lies, Damn Lies, and Statistics

Reasonable recruiters and hiring managers want to reduce turnover, improve productivity, and minimize training. They do this by testing applicants before putting them on the job. In other words, they want test-score assurance their decision will be the right one. Here is the bad news. Although you might have hoped you would never again encounter statistics, it is the language of test validation. And, brothers and sisters, it’s an area where many vendors use smoke and mirrors. Here are some examples:

  • Small or Unequal Samples: small or unequal sample sizes often produce squirrely data that cannot be trusted or generalized.
  • Group Clustering: dividing people into two groups and comparing their scores only tells you the groups are different. It does not tell you anything about individual skills.
  • Shot-Gunning: giving a broad multi-factor test to a group and looking for correlations is no assurance the factor causes performance (i.e., both ice cream sales and shark attacks increase in the summer).
  • Unclear Targets: productivity has many definitions; you need production data that is trustworthy and free from personal opinion.

I’ll continue this article in part two.


36 Comments on “Promises, Promises: How to Identify a Bad Hiring Test (Part I of II)

  1. Well put Wendell, for the most part. Still, you know better than to paint all interviews with the ‘bad test’ brush when patterned behavioral interviews do about as well as cognitive ability across all jobs (Schmidt and Hunter, 1998, Psychological Bulletin). I would be most interested to know where you came up with the statement: “poor hiring practices is estimated between 20% and 50% of yearly payroll”. I would agree based on simulations I have run, so would like to know your source.

    A couple minor points worth noting include: [1] Small Ns result in poor power to detect valid predictors. They become a problem when the vendor conducts many small N studies and then cherry picks the large correlations, conveniently forgetting the small or negative ones. This happens a lot, and is caught by the ‘cone of validity’ analysis arising from biomedical research. [2] The use of incumbent samples in validation research is another ‘smoke and mirrors’ problem when applied to personality-keyed tests. Incumbents will reveal their personality and that does moderately predict job performance when done well. However, candidates are much less willing to do so, and the validities in candidate samples often shrink from the .30s to the single digits and teens. Problem: You can’t hire incumbents (they already work for you). [3] The problem with ‘shot-gunning’ is not a causal arrow problem. It is a problem of finding 5 correlations out of a hundred that are ‘statistically significant’ when that is about how many would occur by chance alone. While pure researchers are interested in what causes what, applied science is only interested in what works— which predictors correlate with sound measures of value delivered on the job.

    Overall though, another sound post that contributes to pointing readers in the right direction.

  2. Companies, too often, put so much stress on test results and establishing an “ideal” profile, and they end up allowing the test to make the hiring decision. So, to me, that says the organization almost passes the hiring “buck” to the test and releases accountability of those that should be performing a strong interview that digs for a Situation, Action, and a Result to each question answered. I worked for a company that actually screened out candidates with a test. I battled tooth and nail pushing the point that we need to rely more on ourselves than the test, and we are missing some great candidates.

    I took it upon myself to do a validity study about the correlation between test patterns (identified as KPI’s) and true performance measurements (annual review data, revenue dollars, turnover, etc..), and the raw data proved that there was absolutely ZERO correlation between the “ideal” personality profile that the vendor put together for the organization and quantifiable performance measurements of top performers. In my opinion, the best way to use those tools is to see what management methods work best with different personalities. What do you think?

  3. Hi Tom…I hope you give me license to reduce all the mumbo-jumbo we psychometricians use every day into over-generalized aphorisms for the average reader. Yes, I have read all of Campion’s work on SBE and was a master TS trainer while I was at DDI. Although I oversimplified interviewing in the article, my experience with SBEs is most users think SBE’s are just better questions…They seldom, if ever, do a JA to identify the key competencies; and they seldom, if ever use job-standardized behavioral anchors to evaluate the candidate’s responses. On a scale of 1 to 5, with unstructured interviews being a 1, and 5 being a Campion-quality SBE , I’d rate the average SBE user at a 2. I don’t remember who published the 10%-50% annual payroll data…I think it was Wayne Cascio in either a TIP or Personnel Psyc article a few years back. Anyway, I find annual payroll is easier to explain than utility analysis. Right on with your psychometrics comments. Thanks for the additions.

  4. Hi Mike, I agree with you. We face several problems doing validation studies…One is the problem identifying a really-trustworthy definition of “productivity” (e.g., in my experience, a manager’s rating of productivity is often half-fact and half-opinion). Another, as Tom pointed out, is having enough people in the study. And the last is testing something that actually affects job performance. Add all three together and you usually get garbage data.

    I’m afraid your vendor is a classic example of what not to do. Again, in my experience, it is more important to isolate ONLY the critical factors that affect job performance… I find personality “profiles” are either insufficient, misleading, or just plain junk. Isolated personality factors are much more important. A few years back, I worked with a company that had 130,000 on-line applications a year. After we isolated one or two critical personality factors, we were able to reduce turnover in certain positions by about 30%.

    Finally, as you pointed out, it is a mistake to use group averages to make assumptions about individual skills. As to other personality traits? They may not affect productivity, but they will affect how the employee feels about the organizaiton, department, and manager.

  5. Wendell – let me start off by saying I totally agree with your fundamentally points and theme. I do have a question regarding your top three, however – IQ, EQ and organizational skills. My experience seems to indicate there are a few others more important – intrinsic motivation to do the work required, the situational fit with the hiring manager leadership style (the Blanchard stuff) and culture/environmental fit (pace, decision-making, resources). Gallup’s First Break All of the Rules, seems to indicate that these are more important in predicting job performance and OTJ satisfaction. What’s your take?

  6. Dear Lou
    I will leave it to Wendell’s capable fingers to respond directly to your question on his take. I can say that the first rule that Gallup’s ghost writer broke when writing the book you cite is to read, understand, and incorporate what we KNOW vs. THINK, as so nicely pointed out be Keith Halperin in your record-breaking ERE discussion of last week. I say this as someone who was offered the job of Chief Selection Scientist at Gallup– so I did a little homework.

    PS: I didn’t take the job in the end.

  7. So Tom – you’re saying the Gallup survey of 80K is flawed and the Q12 are not correct? IF so, this seems to contradict Pink’s work in Drive – he cites dozens of research papers – which all support the Gallup study, especially with regard to intrinsic motivation being the primary driver of OTJ performance and success. His work also supports Leadership IQs study of the primary causes of OTJ under-performance.

  8. Details, Details Lou. Those little things that are really hard to see when you are flying fast at high altitude.

    The Q12 (as far as I know) is a measure of incumbent engagement. It was never meant and has not been used for selection by Gallup or anyone else. I have every reason to believe it does a fine job for that purpose. Google research by Schmidt and Harter on that front.

    I have to confess to being unfamiliar with ‘Pink’s work in Drive’– being more familiar with Pink’s work at the Grammy awards. Different Pink, I assume. But I would be most interested to hear about any correlations based on Ns greater than 200 between measures of ‘drive’ and measures of job performance for candidates. Point me to them and I will check them out. So far, ‘drive’ as a component of conscienciousness had predicted job performance, but certainly not with anywhere near the strength of GMA (general mental ability).

    Now perhaps we should let Wendell in on his own post for at least a couple of days. 🙂

  9. Wendell has a way of putting fundamentals out in a provocative and challenging manner and he sure hits the mark oftentimes here. The issue of selection is often a train wreck because the essence of talent is often missed in interviews, assessments and all the other divining rods currently used. In work with JetBlue, we discovered that a combination of personality type and temperament and a scrubbing of the transferable skills critical to success in a job category yielded a potent profile of the kinds of people most likely to succeed in their customer service category. The product developed, TalentDNA, established the skills criteria and was then compared to the transferable skills each incumbent (or potentially, each candidate) was most impassioned about using based on his or her analysis of his own life and work history. Skills overlaps could be tracked and training assets could then be assessed and selected based on the real needs of those in the job category, even accommodating differences in learning styles. Moving away from the concept of “test” and to the a more user-friendly, collaborative and interactive process using both high tech and hi touch can be a great way to define talent in a three dimensional way i.e. type and temperament, personality and character, most impassioned transferable skills, values and content interests (self-esteem is another key factor as people rise to the level of success that their self-esteem can absorb) and, coupled with competencies can yield powerful individual profiles as a building block to talent management from sourcing through onboarding and beyond.

    The work of Daniel Pink is well-researched. The issue of “fit” or “alignment” is critical in success on the job as it is the fundamental basis for the Holy Grail of engagement.

  10. Hi Brian…see my earlier comments about having the complete package to do the job…fit and alignment is only one part. Although he is an engaging speaker, I’d be interested knowing when Dan Pink’s “research” was reviewed by people who are actually trained to identify and measure job performance…Being published in Borders is substantially different from being published in a scientific journal.

    I advise anyone to be EXCEPTIONALY cautious of mistaking entertainment for scholarship.

  11. Thanks Tom…it was nice to take a short break…Feel free to chime in anytime.

    Ok Lou…I may be wrong, but whenever I get a question like that, the person usually has a hidden agenda and wants me to confirm their beliefs.

    Sorry, in my experience, there is more to performance than we can measure in the hiring phase. First, we have the four basic factors that it takes to do the job (i.e., brains, interpersonal skills, organizational ability, and motivation). These are not to be confused with IQ, EQ, or any other Q. They address whether the person CAN DO and WILL DO what the job requires. This is independent of the working environment, and for the most part, it’s all we can measure.
    Once on the job, the employee encounters the H.B. job-maturity leadership style stuff. The H&B theory is common sense (i.e., manage the “green” employee more closely than the “seasoned’ one), but I don’t recall H or B addressing the issue of whether the employee is even capable of doing the job. In my experience, you can coach, train and develop until the end of time, but you cannot make a silk purse out of a sow’s ear…of course, you could always lower your standards and make a nice sow’s-ear purse.

    Manager and organizational fit will certainly mediate employee performance, but I think of it this way…
    From the first day, the employee either does or does not have the core skills to do the job. If he or she is blatantly incapable, nothing the manager or organization does will make a difference. If the employee is fully skilled, the manager and organization can supply the resources for him or her to perform…or they can extinguish motivation.

    The issue I have with leadership theories starts with the basic assumption that leaders think they can mold anyone into anything…if they stopped to think about their own experience, they would see this concept is ridiculous. As for the Q12, 11 of them represent how an employee is treated…they hold nothing new. It’s just good performance management (assuming, of course, the employee starts with the core ability to perform).

  12. Wendell – Of course, I have an agenda, but it’s far from hidden. What I’m surprised is that there is a lot of science available, that those who profess using the latest thinking, seem to ignore. They in fact, seem to revert insular thinking – if it’s not invented here it can’t be any good, so they berate it. To minimize the Gallup stuff as trivial, makes no sense – what’s the point?

    If intrinsic motivation,organization fit (team, culture,manager) and clarifying expectations are shown to be drivers of on-the-job performance why hasn’t the profession found ways to measure these pre-hire. In Campion’s (among others) work, they seem to proscribe to this – e.g., structured interview, job analysis, multi-raters, same questions, detailed rating guidance, etc.), yet it seems like many folks still assess indirect measures – competencies, behaviors, skills. Why not tie these directly to job needs and conduct a Campion-like process. (Note: The not-so-hidden agenda, is that this is what Performance-based Hiring is all about.)

  13. Wendell – I loved the article. Very whitty and right on. I am taking a statistics class and I am very interested in the application of statistics to talent acquisition. Do you have any recommendations with regard to some reading material? I also have recently looked into DDI’s TS Trainer Cert to provide hiring manager and additional recruiter training for my organization as we grow, and to improve our hiring processes. Is there a test that you can recommend as well?

  14. Mon cher Lou
    For someone who gets the practice side of things so right (your Performance Interview Training and integrated hiring solution is loaded with powerful best practices), you manage to get the theory so wrong. Of course, your training in this space occured in the school of hard knocks vs. the graduate school. It makes sense.

    Stop whining about the Gallup stuff. It is a non-issue. The Q12 has no bearing to selection or hiring whatsoever. Just ask Jim Harter (their engagement Chief Scientist) and he will tell you that. Gallup does have selection offerings that include the Strengths Finder and their ‘Perciever’ interviews. There are problems worth debating with that Gallup ‘stuff’, but the Q12 is not even in this universe.

    Your last paragraph begins with ‘IF’. It should end there also. That is what needs to be established through credible evidence— the IF part. It has not, to my knowledge. No doubt motivation and fit are important, but not of paramount importance, as Wendell so nicely explains.

  15. Thanks for the clarification comments Tom…

    Lou, have you ever considered that people like Tom and I might have actually read, and even studied, BEI, the G12, EQ, IQ, BBQ, and just about every pop-psyc theory that enters the marketplace before we comment on it?

    In addition, your expressed belief that personal competencies, skills, and behaviors are “indirect” measures is absolutely mind-boggling. You might want to re-consider studying what JA, MTMM, MR, and BARS are in an scientific sense before using them to promote your product.

  16. Jami…DDI’s TS is a good program, just be sure you do a JA before you use it and build a standardized rating score to score the questions.

    You will have to tell me more about how you want to use stats talent acquisition before I can comment.

  17. Tom, Wendell – I actually love these conversations. However, I’m glad that you finally agree with me that a job analysis is an essential component of assessing competency. Without tying behaviors, competencies and skills to a specific job and then assessing a person’s intrinsic motivation to use these capabilities doing comparable work, you’re measuring their indirect use. Situational fit is the key to making BEI work. Thanks for the nod of agreement. The indirect method unfortunately is what most companies do – being superfical about real job needs – with the result they hire qualified people unmotivated to do the work required and/or don’t fit well in the situation. This is exactly what Gallup’s Q12 would predict, even though Tom wants to dispute this rather obvious benefit. Sometimes statistics don’t tell the whole story. But thanks again with reluctantly recognizing the critical importance of the job analysis.

  18. ROTFL …your last comment about “finally” agreeing with you about JA is hilarious. Never have I seen such skilled word-smithing to promote one’s own opinions. Do you also argue with your physician about the benefits of using crystal meth as part of a healthy weight-control program?

    Your concept of situational fit as the “key” to making BEI work is utter nonsense…The key to making BEI work is: 1) identifying critical job-skill elements, 2) multiple interviewers using BEI interview techniques seeking data from the candidate about demonstrating these (specific) elements; 3) integrating interviewer data, and 4) comparing the quality of response to step 1.

    A few years back, you argued that BEI was not as effective as your product; then, you shared the epiphany that you actually were doing BEI all along and never knew it! Now you claim to be a BEI expert. Give me a break. Job-fit is only one situational component that affects job performance…and it tends to change from one manager (or corporate initiative) to the next.

    Since you profess having more psychometrics’ expertise than anyone else on the planet, why don’t you attend the upcoming professional convention in Atlanta? There you will find two thousand international professional practioners anxious to be enlightened.

  19. Wendall – I thought you’d like that. What you’re saying is what I’m saying – all interviewers, agreeing to the JA, the use of a structured BEI with focus on the performance objectives of the job, measuring intrinsic drive to do this work, and 8-9 other key drivers to predict OTJ performance using the same questions and a rating form with specific guidance regarding the scoring. This plus not making a Yes/No decision unless this hiring team is in essential agreement on each factor. Voting Yes/No is verboten, however, getting managers to do the above is the real challenge – that’s why I’ve changed the name to a Performance-based Structured interview. This also requires managers to define exceptional performance.

    (If you can get me to be a keynote speaker at the Atlanta event, I’ll be there in a NYC-second).

    What I’ve described has been validated (by a PhD even)over multiple companies in a variety of positions and used with reductions in turnover of 50% or more and an equal increase in OTJ performance and satisfaction. Interesting data point: in over 500 situations, the degree of intrinsic motivation to do the work in the specific situation seems to be the primary variable that correlates most with OTJ performance. This is what we refer to as situational fit along multiple dimensions – job fit, managerial fit, cultural fit, resource fit, decision-making fit, organization fit, team fit (working with comparable teams on a 360 basis – inside and outside the dept/company). A highly motivated person working for a demotivating manager is one key problem we try to prevent.

    When you have a one-year guarantee on making placements you tend to use all resources available to make sure you get it right. Recruiters who use this process have success rates far higher than what’s predicted in the literature. Why do you think this is?

    Tom actually supports what we do, however, I do like to bug him,too.

  20. Now Gentlemen. (I choose to be optimistic here).
    I fully support’s Lou’s practices around clearly tying questions that focus on past performance to the VALUE (deliverables) created by that performance. Quantifying what new hires are expected to get done, by when, goes a long way towards drilling down to the gold around how a candidate has created that type of value in the past.

    I also clearly see Wendell’s point that trying to match each candidate to the culture de jour and today’s manager is much like chasing the will of the wisp. Great talent adapts within limits to strategic shifts and learns to work with a range of competent managers. Finding that great talent is the challenge facing us all.

  21. Folks,
    This is a great discussion.

    We tests thousands of candidates during the pre-employment screening phase as well as during the TNA phase using our testing tools which test for behavior as well as skill.

    I would like to share with you the conclusion of our finding (detailed reports cant be provided if requested):

    1. Confidence level rated as “High” (as opposed to Medium or Low)of employers who use testing for pre-employment(35 out of 75)

    2. Confidence level rated as “High” (as opposed to Medium or Low)of employers who use testing for TNA (52 out of 75)

    3. Confidence level rated as “High” (as opposed to Medium or Low)of employers who use testing for Performance (52 out of 75)

    4. 55 out of 75 rated Knowledge Skill & Abilities tests to be a more accurate measure of a testing component

    20 out of 75 rated Behavior tests to be a more accurate measure of a testing compoment

    When HR managers were asked “why do you use Behavior testing services during pre-employment testing?”
    26 out of 75 replied “Serves as key deciding factor during selection”
    49 out of 75 replied ” Serves as Good HR policy”

    Sample came from companies in the Middle East, South Asia and Asia-Pac. The survey had 50 questions. The N was small but it gave us a very good understanding how the market thinks.

    As a researcher turned entrepreneur, I would say that Lou might be slightly weak on his understanding of testing methodologies, but Lou knows what the employer wants, and he can serve the employer’s needs better.

  22. I’m not sure what you mean…In my experience every employer wants a high-performing employee…and, whether they realize it or not, every employer uses testing (e.g., an interview)…However, if you also ask an employer about interview effectiveness, virtually all will say they are not accurate…so please explain to me how someone with a weak understanding of testing methodology can serve the employer’s needs better?

  23. There actually is an answer to your question Wendell…not sure I am the person to try – especially after reading this entire thread – I am probably the least capable – but here goes anyway…

    The person that has a Gladwellian number of hours experiencing the hiring process from end to end – has seen just about every manifestation of the good and bad in getting a high performing new hire positioned – and my guess is that if asked to answer your question who is better suited a testing/assessment expert or a recruiting expert they would probably say the recruitment expert. The reason, as some have mentioned, is that assessment is the “guts” of a great hire – but it is not what actually gets the high performer into an organization.

    So science becomes a part – a very key part – but a part of the process. (I know Lou’s system quite well and agree with a lot of it) and would add what you and Tom advise at every step!.

  24. There is any interesting chart made public by the Recruiting Roundtable which shows nine variables and their affect on improving quality of hire. The top three – hiring manager and recruiter partnership, understanding real job needs and recruiter conversion rates. Assessment accuracy had a neutral position on quality of hire. I put the chart on the Recruiter’s Wall network –

    The point is that good assessment skills while important are less important than strong recruiting skills. This is a point that every recruiter who has closed a deal with a top person knows, but somehow the point seems to be lost on the academics involved in the process. When supply exceeds demand assessment tests strongly correlate with on the job success since it doesn’t matter who gets hired, but when demand exceeds supply it’s a whole new ball game. Is there any science around closing and quality of hire to demonstrate this obvious and commonsense point?

  25. The point I have been trying to make in the past, the present, and the future is if an organization wants a highly skilled employee, they need to evaluate him or her using trustworthy, reliable methods. I really am not talking about sourcing, on-boarding, recruiting, or anything else.

    If you want a good athlete, he or she has to be able to play the sport. If you want a good soldier, he or she has to be able to get the job done. If you want a good employee, he or she has to have the right skills for the job… Read the Guidelines. Read the Standards. They resprsent best practices.

    What part of this is so hard to accept?

  26. Wendell – none of it is hard to accept – you’re preaching the Gospel of Assessment – it is the basis of our work…

    The reason in my opinion that we have 30% turnover or more each normal economic year is due to the huge number of people that get into the wrong job because of interviews and resumes used as the basis for hiring…

    If all jobs were filled with methods that you, Tom and Saleem speak of – we’d have a much happier and productive nation!

    (By the way – I have been a member of ERE since the early days ’98-’99, and have followed many posts and threads and have not seen one as lively or informative as this post…I’ll have to stop by more frequently…)

  27. KC – let me offer a different Point of View – I became a very good interviewer because I had to guarantee my candidates for a year. And the guarantee wasn’t based on skills, competencies or knowledge. It was about achieving consistent results in the new environment using these skills and abilities, etc. Assessment were only a part of this. A structured interview was part of this. Ref checks were part of this. In fact, I learned more about my candidates talking with their spouses at home or when scheduling follow-up meetings. A lot of insight is developed when someone is “off camera.” The assessment is a threshold variable only – once you’re above it, there are more things that matter.

    I’m not surprised that academic-types don’t get the idea that placing a top person in the right situation within a tight budget is not about assessments, because they’ve never orchestrated the whole process, they’ve only played the fiddle. And they do seem to think that fiddling is all that counts.

  28. OK, Lou. You know more than anyone else. No educated person has ever actually run a business, hired an employee or made a payroll. I’ve got to go now… Besides, I’m late for my violin lesson.

  29. I would like to add in one last word as the sweet chords of the Wendell’s violin pleasure us:
    Having been in research (academia), then applying the knowledge into business, I realized that its very easy to play with numbers on models and spreadsheets, but when you try to apply these models or validation studies in the real word, academia is seldom correct.

    So to battle this, we at HireLabs have create a partnership between our client’s line managers and our internal test creators. As a result we have realized that better tests are created, and the risk of making the wrong hire is reduced (statistical data can be provided).

    I would sincerely advise my peers from academia to try to understand the realities on the ground. This way we will be able to put egos at bay, and improve our financial productivity collectively.

    I look forward to an invitation to Wendell violin recital 🙂

  30. Very outstanding article…the test should not consist the descriptive questions, as it will not identify the right personality of an applicant…the goal of the test maker should be to ease up the candidate while doing the test, not to stress him out. The well designed test consists the questions that help the candidate to clearly understand and easily solve the questions…

  31. As Wendell suggests I know little about music, however, I went to a play last night about Gershwin. It turns out he did something unheard of in the musical world at the time – he changed the key mid-song. As a result Al Jolson recorded Swanee and it became a big hit. The only point of this is that there might be new or better ways to achieve a goal, using different approaches and concepts than have been used in the past. However, progress is never made if new ideas are are pooh-paahed because they threaten the status quo.

    Saleem – I endorse assessments – see page 185 of Hire With Your Head, however, I suggest using them as late as possible in the hiring process to minimize cost and prevent top people who aren’t looking or quietly looking from opting-out too soon. Of course, test vendors don’t particularly like this recommendation. If supply exceeds demand and a company is convinced the pool of remaining candidates is talent rich, then it doesn’t matter when the test is administered. However, the talent rich part is typically not the case. Also, there is some error induced due to range restriction if the best people aren’t taking the assessment.

  32. @Lou, I love your book, and believe it or not, I have two copies: I bought one, the my mom gave me one on my birthday 🙂

    Unfortunately I have to humbly disagree with you on the ‘timing’ of when the assessment test should be taken. Our clients in the Middle East have started doing quite a bit of hiring, and they like to test all the candidates before they call them in for an interview. Provided that our tests yield good results (which they have done so far), our clients are able to save a lot of time in the resume screening process, because they only invite those who have scored either an A or a B in the test overall… sometimes they invite the Cs, and rarely focus on those candidate who have scored Ds and Fs. So this way if the client receives 150 resumes, only 20-25 will score As and Bs (depending on how difficult we customize our tests).

    You can do the math on the cost savings with respect to time (resume review, phone interview etc).

    Finally, in order to counter the issue of “an expensive assessment service”, HireLabs has priced each assessment at $150 (regardless of how many candidates you test for that position)… so I hope you see how assessments are no longer expensive

  33. Saleem – don’t be humble! Of course, when supply exceeds demand we’re in total agreement. I’m probably in total agreement with Wendell then, as well. Everything I suggest is under the condition of scarcity of enough good people. My whole experience is based on producing talent without enough supply, but plenty of demand. This is the only reason a company would pay a fee for an outside recruiter. Your rates seem eminently reasonable, too.

  34. Lou – the talented top 10-20% are always scarce, regardless of economic conditions – thats why there are only 20% of them…right?

    Having built Talent Communities for the last 10 years for a host of billion dollar sized companies – there is no question that to cultivate the best you have to know who they are and challenge them to show their value. Using products like Saleem’s we can provide the candidate with the means to “show off” their abilities. Its not the only means of determining a candidate’s worth – but it sure adds a variable that TPR’s haven’t had previously…

  35. A healthy debate, to be sure, but seems like a revisiting of an age-old argument: researchers don’t know what it’s like in “the real world”, and consultants (internal and external) don’t appreciate the value of research.

    There is probably some truth to both of these statements. But what exactly are we arguing over? That good (creative, effective, etc.) recruiting is important? I don’t think Wendell would disagree with that. That high quality assessments are important? Lou acknowledges that.

    The debate seems to be around which is more important, as if that will determine where we spend our thought and energy. This is not a zero-sum game. We should pay attention to accurately describing job requirements, organizational brand, attracting qualified applicants, using high quality assessments, ensuring top notch leadership, etc. etc.–all of the HR aspects of a high performing organization.

    One thing we should all agree on is that we will never be able to predict with 100% accuracy how well someone will do on the job. Unless your measure of performance is their test score.

Leave a Comment

Your email address will not be published. Required fields are marked *