If It Does Not Cause, You Need to Pause

Do the vast majority of people who pass your personality tests turn out to be exceptional performers? If you answered “no,” then your tests aren’t testing. Recruiters and hiring managers are led to believe people who pass their personality tests will be successful. Unfortunately, practical experience shows that about 50% of employees and 70-80% of managers still fail to meet expectations. It’s a hard concept to grasp, but don’t be fooled by statements like: “The XYZ is not a hiring test … but it can be used to help make hiring decisions.” That’s like saying, “Ignore the rattle … the snake’s harmless.”

Cause? What Cause?

Here is an example of traits often found in personality tests: dominance, compliance, extraversion, judgment, sensitivity, curiosity, conscientiousness, humility, and determination. First, we’ll show you a silly-science example: 1) divide producers into groups (e.g., high and low performers); 2) give both groups the same personality test; 3) see which scores differ; and finally, 4) use candidate scores to predict group membership.

After impressive number-crunching, suppose the A-list group had higher average dominance, compliance, and extraversion scores; the B-list group had higher average curiosity, conscientiousness, and determination; and, both had the same average judgment, humility, and sensitivity scores. Is this enough evidence to use the results for selection or promotion? Noooo.

Lies, Damn Lies, and Statistics

Anyone can compare two sets of numbers and tell you whether they correlate; but, it takes careful study to know whether A actually leads to B.

For example, skirts and stock markets tend to move up and down together, beach ice cream sales and shark attacks tend to move together, and watermelon sales and temperature move together. But, skirts do not cause the market to change, sharks do not buy ice cream, and selling watermelon does not cause it to be hot. You can probably think of many others, but the most important statistical concept is, “If it does not cause, you need to pause!”

True professionals know beforehand the factors they want to measure. Then, they use stats to compare scores with performance to try to prove themselves wrong! I know it does not make sense, but remember that since the future is murky and uncertain, it’s better to reduce mistakes than seek perfection. Explaining things after the fact is creative story telling. Professionals make an informed prediction, collect data, and try to disprove it.

Screwy Thinking

Returning now to our example: We already discussed why throwing things against the wall to see what sticks is unprofessional. Now let’s consider the Lake Woebegon effect; that is, the men are all strong, the women are all pretty, and the children are above average.

Let’s suppose in our previous example that shoe size was one of our factors. We know individual shoe sizes in both Group A and Group B ranged from size 6 to 12. However, Group A folks averaged size 8 and Group B averaged size 10. Does that mean an applicant wearing a size 9 will become a member of Group A? A size 12 a member of Group B? Nope. And, Nope. Group-level data tells us about groups, not about individuals! Bad analyst! Bad!

How about this? There are four people in Group A and 10 people in Group B. Aside from the problems we already discussed, can we compare the two groups? ‘Nope again. One person in Group A has a 25% impact on the group’s overall score, while one person in Group B has only a 10% impact. Furthermore, the group sizes are so small it would be silly to think scores would generalize to all candidates. It takes at least 25 (preferably, hundreds) of subjects to draw reasonable conclusions. No soup for you, analyst!

Article Continues Below

Oh, yes, one more thing. Can we trust someone with a high score in the judgment trait to be smart? Get real! Most studies show less than a 1% relationship between personality scores and cognitive skills and about a 10% relationship with interpersonal behaviors. Why? When people take a self-descriptive test you never know if they are honest, trying to make a good impression, delusional, clueless, and so forth. If you need someone who is smart, give them a problem to solve … not tell you about it!

Importance of Being Wrong Less Often

It may sound counterintuitive, but it is easier to reduce the number of bad hires than it is to find superstars. The future is murky, filled with unpredictable events that elude even Karnak the fortune teller. Folks who expect 100% hiring and promotion accuracy are going to be frustrated. No system known to humankind can perfectly predict the future. There are just too many uncontrollable variables.

The present, however, is more tangible. So, instead of trying to ensure perfect success, it’s actually easier to reduce test error by screening out unskilled people. It goes without saying that manager-employee compatibility is very important; but, in addition to personality factors, organizations expect employees to have cognitive abilities, motivations, and so forth. This is the 20% that delivers 80% of job results.

Summary

Selecting or promoting people based on silly science is more than just bad practice. It’s unethical, irresponsible, and unprofessional. Qualified people are rejected, unqualified ones are hired or promoted, and the inevitable potential of legal action increases. And, it could get worse. What do you think will happen when all those incompetent employees think they should be promoted to management?

Reducing the odds of making a wrong decision requires tests, interview questions, application blanks, and so forth that are grounded in a solid theory of job performance; that is, they measure things that cause high or low performance. If you cannot, for certain, prove you are measuring factors that cause performance, you will never graduate from the half-wrong club.

“If it does not cause, you need to pause!”

Topics

21 Comments on “If It Does Not Cause, You Need to Pause

  1. Hmmm… where do I start? Much to say about this subject!

    First of all… with regard to the statement, “The XYZ is not a hiring test, but it can be used to help make hiring decisions”… I agree. (Well, change the word ‘test’ to ‘assessment,’ and then I agree!) Hiring managers (or recruiters) should never be advised to use an assessment as their sole decision-making criterion. We advise that they take a “3-legged stool” approach. They need to consider the candidate’s resume (background, skills and experience), their performance in the interview (we’re considering both the information shared as well as the interpersonal dynamics ), in addition to what the assessment says.

    Second…All assessments are not created equal. I won’t get overly detailed on this point, but… assessment users need to be concerned about things like reliability and validity studies and adherence to standards set forth by organizations like the Dept of Labor.

    Third…along the same lines of all assessments not being created equal, sadly, NO — they don’t all measure what they claim to measure. As you indicated, often those who “pass” the assessment don’t always perform. And here is where we start getting into trouble with what assessments can tell us.

    Does the assessment tell us who the candidates really ARE, as opposed to who they THINK they are? This may sound confusing, so let me use an example. If an assessment asks me to rate myself on traits like assertiveness, energy, sociability, and intelligence… I may give myself high ratings on ALL of these traits. But there’s nothing to compare it to…. I might give myself a 10 on all of these traits, but in reality…. compared to others I may realistically only be a 5. Some assessments that hiring managers are using are missing the “normative” function that allows the candidate to be compared to others in different areas.

    Another dimension that is missing in many personality-based assessments is a cognitive section. Personality or behavioral characteristics can tell us HOW a person is likely to do the job, but they don’t tell us if a candidate CAN do the job. Some positions require higher cognitive abilities —- higher than average verbal and numberical abilities and reasoning skills. Just because someone has the “right” personality for the job, it doesn’t mean that they will be successful.

    A third dimension that adds in predicting success is basic interest in the job. For example — when assessed in my numerical abilities and reasoning styles, I score in the top 5%. It’s fair to say… I’m pretty good at math! But… I would wither away in a job that required me to do mathematical calculations all day. I’d be good at it… but I would hate it and eventually leave. So, the third dimension that needs to be considered, in addition to the first two (CAN they do the job and HOW will they do the job) is… do they WANT to do the job?

    The final point I’ll make with regard to identifying potential top performers has to do with the “model” that you are comparing the test results to. You mention that those who “pass” (not a fan of that word!) the assessment are not always successful. But the question here is… what model are hiring managers using to determine who fits the model.. or, stated less correctly — who “passes” the test? The better your performance model is that you’re comparing the assessments to, the better predictive information the assessment will give to you. The secret here lies in providing solid data about your current top performers. This provides a proven benchmark (specific to your company, industry, geography, and job position) that you can compare the assessment results to. Creating reliable benchmarks is a whole other fun topic… that I won’t go into today!

    To tie it all up, the bottom line is that assessments can prove not only valuable, but also critical, to companies in helping them avoid placement mistakes, but… it’s fair to say that some assessments are infinitely better than others!

    Lisa Sperow
    http://www.QPMGroup.com
    http://www.CloneMyTopSalesReps.com

  2. Lisa, I certainly hope I misinterpreted what you said following “First of all…”. You clearly state that you do not believe the assessment should be used as the only factor in making a decision. Then you say that you advise your Hiring Managers and Recruiters to use a “3-legged stool” concept. If I am in your group, you have just told me that the assessment actually is a deal-breaker. If a candidate passes the resume leg and interview leg, but fails the assessment, they have no chance because a 2-legged stool cannot stand. I realize your metaphor may not be meant so literally, but I wonder if your Managers realize that. We, out here in email land, can go only on what you wrote, and what you wrote defies what you intended. I do not bring this up as literary critique, but as a way of pointing out how people often offer what they understand to be good advice, but which is not received by others as such. Several decades ago, I was directed to use an assessment tool as the tipping point in the event a candidate passed in every way but the assesment, and told to always get a partner if I wanted to override the assessment score. Weeks went by, and after the fifth unsuccessful attempt with my “partner” to override the assessment score for a good candidate, I started giving the assessment first…i.e., it became the single most critical component. Now, decades later and much wiser, I am absolutely convinced that, overwhelmingly and in spite of assurances to the contrary, a non-passing assessment score will bounce every candidate, with very, very rare exceptions.

  3. Wendell, I failed to mention…excellent article, wonderfully entertaining as well as informing. Thank you.

  4. Lisa: Spot on!
    Use an assessment not a test-a test implies yes/no, right/wrong, pass/fail. An assessment provides additional information upon which to base a decision.

    The more job related dimensions of a person that an assessment measures, the more valuable the assessment information can be. The results provide avenues for further exploration during the interview process.

    The closer the model which is used to produce the assessment evaluation is to the actual hiring position vice the general working population, the more accurate the results will be.

    Final comment: A hiring decision is always the employer’s responsibility; don’t pass hiring decision responsibility off to any one type of input-appearance, interview, references, assessment results etc. (exceptions might be credential requirements, background check results) Consider all available information and then use your brain!

  5. Thanks, Robert. Good point, too — the results do provide great avenues for further explanation during the interview process. And yes— the purpose of using assessments is to make the best possible decision based on the best possible information. At the end of the day, a real person makes the decision whether or not to hire another real person!

    Lisa

  6. Hi Wendell,

    I think you make some good points, but I’m not sure about the conclusion. Your basic point seems to be that companies need to use assessments which are highly grounded in the job, finds true causality rather than simple correlation, and probably cost a bundle to build and validate.

    Personally, I agree with that. In a perfect world, that’s just what they should do.

    But I think I disagree with your point about correlation and causation. I understand the scientific issue you raise. But your examples are all straw men. I don’t think anyone would conclude that sharks buy ice cream or that watermelons make it hot. Is anyone anywhere saying a sample size of 4 people is useful for anything?

    But how about this? There’s never been a controlled scientific study that shows that smoking causes cancer. Perhaps it’s just a correlation. Having a Harvard MBA doesn’t cause you to be rich, but I would suspect they earn above the median.

    If you do a study on 10,000 call center agents and find that you can test for a trait that leads to higher customer satisfaction, do you still need to find cause? Or is correlation good enough? On the assessment level, someone with a high degree of resilience will probably last longer as a collections agent than most. Someone who is more detail oriented will probably do better as a technical support rep than someone who isn’t. Not in every case, but often enough to have a significant impact overall.

    If you’re a high volume hiring employer, you may be pretty happy if someone can come along, determine the traits that your best agents have, and test for them. If the result is a workforce that performs better, stays on the job longer, and produces better outcomes for your customers, who wouldn’t want that?

    Also, remember that we aren’t in a perfect world. The alternative screening methods I’m seeing are pretty random. A bad result is bad for the employer and the employee. Who wants to get hired into a job they are going to hate and fail at?

    So yes, if you don’t see cause, you must pause. But don’t pause for long if your current process isn’t working. There are solutions out there that will give you great results without needing to scientifically prove causation.

    Dan
    denthoven@evolvondemand.com

  7. Hi Jim— Thanks for the comment. Let me clarify two things:

    First of all … as I mentioned earlier — I don’t believe in referring to a candidate’s assessment performance in a pass-fail manner. An assessment “is what it is.” They’re not “tests” (at least the ones I work with aren’t) — they’re simply designed to measure who a person is.

    When people use the words pass and fail, they’re typically referring to how closely the person’s assessment “matches” the performance model (ie, combination of attributes, cognitive abilities, etc) that they think they want for a given position.

    That’s one of the first and most important steps in using assessments… knowing what you’re looking for in hiring and why. For simplicity’s sake, I’ll continue with my example from before. You think you want to hire people who score 9’s on assertiveness. (and let me clarify — this is a HUGE oversimplication — you’re looking for a pattern / combination of traits, skills, and interests… not just one. I’m just using one for example purposes). WHY do you want to hire someone who is highly assertive? Here is where we go to benchmarking. A great way to determine what you’re looking for in potential hires is to assess those who are currently successful at your company in the given position. Important: You must use quality metrics so that you can objectively define success. You may find that top performers typically score 7,8’s and 9’s on assertiveness (another point to note — you’re always looking for a range, never an exact number) You may also assess top performers and find that, on a cognitive scale, your top performers rate very highly on numerical abilities, suggesting that those who do not score at this level may not perform as well as those who score higher.

    All of this to lay the foundation for my point: You’ve indicated that I am suggesting that the assessment is a deal-breaker. Rather than saying the assessment IS a deal-breaker… it’s better to say that yes, the assessment CAN be a deal-breaker. IF your comparison model is valid, the assessment really can help you predict success in a position. Using the numerical abilities example — if your candidate scored a 2 in this category and your benchmark of top performers scored 7’s, 8’s, and 9’s — then YES, this should be cause for concern. It COULD be deal-breaker. (Some companies even benchmark their bottom performers… what if the bottom performers all scored 2’s and 3’s here? This could be a good indicator that this is a really important characteristic for job success.)

    Again… I’m oversimplifying by breaking it down to just one category. In reality… you’d look at a combination of scores, and if your candidate fell outside the benchmark in many of the categories, then YES, the assessment CAN be a deal breaker, because it would suggest an overall poor match to the job (which is exactly what we’re trying to determine!) If the candidate falls slightly out of the range in key areas, you will know up front the areas that he/ she will likley struggle with if given the job.

    You mentioned that you were advised to consult a partner if you wanted to override the assessment. I think that is a great idea. At some point, you’ve got to have faith in your instrument. That’s the beauty of a GOOD assessment… they can help you see beyond what you’re able to tell in a few brief interviews. I know that personally, I’ve erred on the side of hiring people who were just super-likeable in the interview process! We hit it off, we laughed, I WANTED to hire them… as the school of hard knocks has taught me well, likeability doesn’t always ensure a great job match or top performance!

    As you can tell… I’m a huge fan of assessments used correctly and for their intended purposes. Managers need to understand how to properly establish their performance models and how to interpret an assessment to understand the gold mine of data that they provide. I’ve heard the horror stories from people who ignored the assessment data, hired anyway, had miserable experiences, and learned to place a high value on what the assessments told them. (And again.. I’ll refer back to my Not-all-assessments-are-created-equal statement.)

    I hope that clarifies things!

    Lisa
    http://www.QPMGroup.com

  8. Articles on assessments always seem to draw fire. Wendell, yours is entertaining and, with charity around your amusing and not-meant-to-be-insulting examples, pretty much what other experts say. Correlation, however tells a lot and you should admit it. How often do medical doctors not really know why a drug heals, but it (usually) does? Nearly all the time.

    Bad assessments and bad use of assessments are, yes, bad. But good assessment do a lot – they generate tremendous, stunningly better results than non-assessments. Both individual examples and meta-studies show that.

    So, use assessments as the primary means of selection and you will do better than any other method. You won’t get it right every time – just more times than with any other approach. I am always puzzled how companies can show the increased performance of those who scored well (I hope they mean scored consistently with the specific requirements of the job rather than just, you know, well) compared to those who scored poorly. How do you know the performance of those who scored poorly unless you hired them and WHY DID YOU HIRE THEM?

  9. Does hiring based on well-written and administered assessment tests without regard to work history (resumes) produce, better, the same, or worse *hires than hiring using resumes without assessment tests? What results come from using both?

    Keith

    * Could you interview based on test results instead of resumes? If so, how would you do it?

  10. Hi Keith,

    It can be tedious to go through resumes, but they do provide us with a starting block to determine WHO to interview and assess. Since most jobs require some level of skill or experience, the resume tells us where to start and who to consider assessing.

    Here’s what an assessment can tell us: WHO is this person? What characteristics does he possess? We like to say that characteristics are the building blocks to competencies.(the skills may not be there yet, but… we have reason to believe that the person is well-suited to develop and master them.)

    For example… someone may have the combined set of characteristics necessary to be a good engineer. He may have excellent mathematical skills, he may be well-suited for a sedentary position with little interaction with others, and he may prove to be more objective (vs subjective) in his judgment, etc… all potentially good traits for an engineer. However, if you’re hiring for a senior level position, you may want a minimum of 10 years’ experience. The profile may suggest that this would be a great person to develop into a fine engineer, but… he doesn’t have the necessary level of expertise just yet that is required for this particular job.

    So, the background serves to help us identify assessment-worthy candidates. Although assessments can provide value far greater than their cost, most companies still wouldn’t want to spend money assessing candidates that don’t meet their initial criteria for a given job.

    That said…. using our previous scenario, if a company did choose to assess our engineering candidate and found that he matched the company’s performance model for its top engineers… they might consider bringing him on into an entry-level position with the goal of developing him into a senior level engineer someday!

    The above answers your second question (hopefully!. I don’t know of any situations in which a hiring manager has hired based on assessment results only without regard for work experience, so I can’t answer the first.

    I hope this helps!

    Lisa
    http://www.QPMGroup.com

  11. Maybe I can cut to the chase…A resume, application, interview, and so forth should be considered a sales effort. The data may or may not be true or the person may or may not be in touch with reality. The data from a validated test (exercise, simulation, case study, problem-solving exercise, and so forth) is more trustworthy because it is hard to fake.

  12. Hi Keith,

    I think there are actually a very large number of jobs where the resume doesn’t matter at all. Think about what used to be considered “entry level” jobs. These would include food service, call center, and other low skilled non-physical hourly positions. Now these jobs are being filled by everyone from college kids on summer break to moms reentering the workforce and (un)retired people backfilling their retirement accounts.

    There are far more important factors that predict success (and prevent early attrition) than what you find on the resume. Assessments can answer questions like if they have the fundamental traits that will make them successful and be happy with the position.

    Assessments can also find other issues that indicate the applicant will be really unhappy with the position, or that they’ll fail at it. (This is the part that Wendell refers to when he says it’s easier to eliminate bad hires than it is to predict super stars.)

    Wendell is totally right that a resume is a sales tool. People sell themselves into jobs they don’t actually want all the time. Then they quit. The right assessment process can stop this in a way that no resume review or interview can.

    Dan
    denthoven@evolvondemand.comn

  13. @Lisa. Dr. Williams, Daniel

    Thanks for your explanations.
    So, assessments can usually serve as useful supplements to resumes, and in some cases (like entry level positions) you don’t need resumes. When would it not make sense to use good assessment tools in conjunction with resume-based interviews? (Bob Gately showed me that some assessment tests are quite affordable.)

    Happy 4th,

    KH

  14. Right on both counts. As a general rule, the bigger the potential OJT mistakes, the more tools you should use. However, since most jobs cost thousands of bucks every month(right or wrong) you still need to do a good job. Whatever assessment tool you use, make sure it is validated (i.e., scores=STRONG evidence of job performance)for the job…especially one that includes number crunching or logic-type questions.

  15. What a great article! Something so many people fail to consider is this issue of “what does that have to do with job performance?” I’ve heard people in interviews ask questions like “what do you do in your spare time? Oh you snow board? So you’ve got bad knees”…umm no, snow boarding has no direct connection to bad knees and nor does it have any relationships to how good you’ll be as a sales person so get off the BS questions.

    I think it starts with (after a general attitude towards the process) poor process for defining jobs in terms of measurable objectives, concrete behaviors, and related disqualifiers. Establishing this process actually makes screening much faster, easier, and more accurate…but it also does more, such as help to build effective training programs (how many companies today will pay for personal or professional development for their staff that don’t cause better performance? When you clearly define the day to day behaviors you can analyze breakdowns in productivity and train to specifically address those issues).

    It’s amazing how much value could be added to companies by effectively defining the skills necessary to drive the productivity/performance of the employee and hasn’t been done (often even simple things like typing rate, use of hot keys, etc.)

  16. Wendell,

    Good article. The most significant point is in your summary that it takes a number of factors and criteria in the “interview” process to increase the probability of making the best decision on a hire. This is something I’ve been saying for years and do for my clients. No one tool is going to be a panacea.

  17. Thanks, Dr. Williams. Are there any neutral reviews of assessment tests (perhaps through SHRM or some other organization), something like a Consumer Reports? If not perhaps something like Yelp?

    -kh

  18. Academic research is the most objective and neutral source…It’s controlled, as objective as one can be in the real world, and reviewed by peers with equal qualifications. Although it grows and changes with the body of knowledge, it’s the best you can get.

  19. @ Keith

    Hi Keith (again),

    I’m not sure of the position in the US as regards tests reviews. Buros (http://www.unl.edu/buros/) is one source which I know is highly quoted but I don’t know about their test review process. Another is the British Psychological Society’s (BPS) test review process, which is independent and highly regarded in Europe and has been influential in defining the process now adopted by the European Federation of Psychologists’ Associations (http://www.efpa.eu/).

    Even though it is UK-based, take a look at the BPS website (http://www.psychtesting.org.uk/test-registration-and-test-reviews/test-reviews.cfm), as reviewed there are tests from publishers such as SHLPreVisor which are used internationally. We (www.matchpointcareers.com) use SHL tests as part of our approach to candidate profile and job matching, and we are confident that they are as good as anything on the market.

    I have also authored tests which have been reviewed by the BPS and have no issues with the outcomes whatsoever!

    Best wishes

    Angus

Leave a Comment

Your email address will not be published. Required fields are marked *