What’s Wrong With Reference Checking? Pretty Much Everything (Part 2)

This article addresses five questions raised in response to Part 1 of this series published last week. It addresses the best ways to assess candidate performance pre-hire and when to use references.

Question 1 — What are the most accurate indicators of past, current and future performance?

Finding accurate real world predictors of future performance is difficult but not impossible. Professional sports teams find that the best predictor of a new hire’s potential performance is their performance on the field in practice and preseason games — i.e., a work sample. Google looks at a multitude of factors that can be combined by an algorithm to successfully predict both future on-the-job performance and retention risk. The U.S. military and numerous firms in industries with extreme operational risk like airlines and chemical production facilities rely on sophisticated simulations to assess how a candidate would react in various situations.

There are literally hundreds of potential tools and approaches that can be used. Unfortunately, the vast majority of research on the subject is questionable at best.  

Rarely are academic researchers trusted to manipulate the assessment processes of organizations hiring in significant volume such that the process could provide valid data, and few if any organizations execute consistently without extreme oversight  Commercially sponsored studies are flawed in that they are executed to prove a tool or approach works, not to test the extent to which it works.

That said, most studies generally conclude, and I tend to agree, that work samples are often the best predictors. Based on my observations and learnings, I routinely recommend the following.

Most Accurate Tools for Assessing Current Performance

  1. Temporary hiring — The best way to identify top performers is to hire them into the job as temporary workers, contractors, consultants, or interns. Interns often have the highest success rate among college hires because you can rely on their internship track record as a predictor of their ability to perform in your environment.
  2. Actual work cases — next to “playing in the game” is assigning them a case challenge based on the real problems that the candidate would face shortly upon accepting an offer. The problem should be a real current issue, so that even if you opt not to hire the candidate, you benefit from their insight into the issue. An alternative approach that has been used by firms like Raytheon is to give them a “broken” or flawed process and ask them to find the errors and source of problems.
  3. A verbal simulation — this approach takes advantage of the traditional interview time period and asks candidates to walk you through how they would tackle an actual work problem. Firms using this approach generally use a case that has already been solved (so that the interviewer/assessor is familiar with all of the relevant information the candidate may need to inquire about and knows the likely outcome of each solution step introduced).
  4. Samples of current work — in roles that produce physical products, one of the best ways to assess a candidate is to assess samples of their current work. If you want to find out if a chef is any good, you taste their food. In some cases candidates may not be able to provide existing work samples from their current employer due to confidentiality issues.

Most Accurate Tools for Assessing Future Performance

  1. Ask them to forecast — if you need to assess candidates’ ability to perform in the future, consider asking them to predict future opportunities and problems your industry or firm will likely face and the skills and competencies that will be required to address them successfully. Consider whether their vision aligns with or is more/less robust than yours.
  2. Future work case — give the candidate a problem or opportunity that is projected to occur two years out. Ask them to “walk you through the steps” that they would take and the skills that they would need to resolve it.
  3. Demonstrate leading-edge learning — the most important competency in a fast-changing world is the ability to continuously learn and stay on the leading edge of knowledge. Even if candidates and employees are successful performers today, it is unlikely they will remain top performers unless they are continuous self-directed learners. You can assess whether they have that competency by asking them specific questions about what are the leading-edge “next practices” in their area, who are the benchmark firms, the key thought leaders, and what specifically do they do every day to remain on the leading edge of knowledge.

Question 2 — When do you recommend using reference and background checks?

As I said in the first part of this article, it’s okay to use one or more reference and background checking approaches provided that you understand why you are doing so and that you have designed a process that can be consistently executed to limit exposure to the vast majority of limitations found in the typical traditional process. If you have read my stuff for a while, you will know that I look at references as future candidates, so getting current candidates to identify their network through any process is a win, even if it wasn’t the intended focus of the process.

Two good reasons that would make doing references essential:

  1. A correlation with quality of hire — any time you have supporting data showing that at your company there is a positive correlation between a candidate’s on-the-job performance and retention after one year (quality of hire) and a scale-based recommendation output by your reference/background checking process.
  2. When there is a legal requirement — when you are hiring for roles that require due diligence to confirm proper licensure, certification, education, etc.

Note that 99.9% of all murderers, bank robbers, and major evildoers are first-time offenders, so the fact that they haven’t done anything in their past doesn’t prevent them from future missteps. If you really believe that criminal, driving, credit, licenses, etc. are valid indicators, you need to check these elements periodically post hire to ensure that your employees maintain their initial pristine status.

Four not-so-good but still-common reasons to do them:

Article Continues Below
  1. Covering your butt — the most common reason to use reference checks are to shield yourself from potential verbal lashings (after a new hire turns out to be a real turkey) by being able to say “but I checked their references.” In addition, in the almost miniscule chance that you’ll be called in a negligent hiring suit, having checked the references will get you a handful of minor points from the judge.
  2. When you want a particular candidate no matter what — if the recruiter or the manager is dead-set on a particular candidate, I understand the common practice of assigning an inexperienced and positive-thinking person (overly positive people will seek out positive attributes until they find them) to do the reference checking. You can almost guarantee that they will come back with the desired “I found no problems” answer.
  3. When you need another excuse to fire a bad performer — if you are reluctant to fire a bad-performing new hire, I have seen many firms purposely redo reference and background checks in the hopes that they can find a lie or omission that could be used as an excuse to terminate.
  4. You yourself need job security and are a bit selfish — if your job involves doing reference checks and recruiting in general, it’s in your best interest to continue executing a time-consuming, flawed, yet accepted process. Avoiding more effective candidate assessment methodologies will ensure that a greater percentage of new hires will fail and therefore require replacement recruiting; in other words, provide job security.

Question 3 — Are reference/background checks accurate assessments of past performance?

First of all we must establish that there is little to no consistency in what is considered a reference check. I cannot say that no reference check focuses on past performance, but based on my observation of practices in place at hundreds of organizations, the vast majority come nowhere close to remotely gathering intelligence on past performance. For example, many of the following elements often focused on during a reference/background check are not indicative of performance:

  • Job title
  • Dates of employment
  • Credit scores
  • Criminal history
  • Personality traits
  • Attainment of a degree
  • Knowledge of buzzwords
  • Information relating to job performance in another era (skills and knowledge grow obsolete too!)

All performance can be quantified, but rarely would a reference checker be granted access to real records of output/productivity from another employer. While candidates could request copies of past performance appraisals, we all know that most view that process as being worse than reference checking with regard to validity!

Question 4 — Are reference checks predictors of present or future performance?

If traditional reference checks are weak ways of assessing performance and especially past performance, do they provide any insight into present or future performance potential? Again, the answer is a sad no. As typically executed the traditional, haphazardly executed reference check is not a valid indicator for the following reasons:

  • The past isn’t today or tomorrow — the world has and continues to change, rendering information and skills obsolete at a fastening pace. Behaviors and actions that produced spectacular results yesterday may fail miserably if the environment changes, which we know it does.
  • Your culture is different — what makes one successful in one environment will not necessarily do so in another. Top performers can easily become average performers under a new manager, and vice versa. A lot of factors influence a candidate’s ability to perform.
  • They may have been bridled — a candidate with a poor “performance related reference” may have everything needed to perform in your environment, but been unable to do so in a previous environment due to lack of professional freedom, proper tools, poor management, etc.
  • The individual has changed — skill and motivation levels change over the years, so how someone previously performed in a different state of mind may not be indicative of how they would perform in a different state.

Question 5 — Are the current vendor offerings in the areas of reference and background checking worth examining?

Technologies and service offerings in the HR/talent management space are evolving at a rapid pace. Many emerging offerings are not bound by the traditions and ingrained practices of the corporate world, and bring valid arguments to the table that all professionals should be aware of. The hottest tools under development today, and others that have been around for a short time, focus on providing organizations with a more comprehensive look into a candidate’s professional network. Tools leveraging a 360-type approach increase the size of the assessor pool, and thereby increase the degree of structure afforded the process and the probability that more qualitative data will emerge. I can’t say without firm specific evaluation that any tool would produce valid results in your organization, but some criteria that I would recommend using when assessing top vendors include:

  • Proof — do they have proof demonstrating the correlation between reference process outcome and on-the-job performance in real client organizations that can be referenced themselves?
  • Integration — do they integrate the many different checks (i.e. credit, work history, personal references, criminal, educational, license, driving, Internet etc.) into a single system?
  • Metrics — do they have access to/knowledge of professional intelligence about the validity of component processes so that they can advise you using data on what practice improve/harm your candidate assessment efforts?
  • Global — do they present a solution that can be used around the globe and that would not be limited in adoption by variances in culture or language? (The labor market is quickly becoming more global, so it is increasingly unrealistic to assume than a candidate’s references will come from the same region or cultural background as the candidate.)

Final thoughts

This series has introduced 30+ reasons to be wary of traditional reference checking processes, as well as a host of alternative approaches that could lead to much more accurate assessment of a candidate’s past, current, and future performance. As the EEOC increases its scrutiny and focus on this subject, everyone in the field of recruiting should learn to be cynical and test their efforts and solutions being adopted.

Dr. John Sullivan, professor, author, corporate speaker, and advisor, is an internationally known HR thought-leader from the Silicon Valley who specializes in providing bold and high-business-impact talent management solutions.

He’s a prolific author with over 900 articles and 10 books covering all areas of talent management. He has written over a dozen white papers, conducted over 50 webinars, dozens of workshops, and he has been featured in over 35 videos. He is an engaging corporate speaker who has excited audiences at over 300 corporations/ organizations in 30 countries on all six continents. His ideas have appeared in every major business source including the Wall Street Journal, Fortune, BusinessWeek, Fast Company, CFO, Inc., NY Times, SmartMoney, USA Today, HBR, and the Financial Times. In addition, he writes for the WSJ Experts column. He has been interviewed on CNN and the CBS and ABC nightly news, NPR, as well many local TV and radio outlets. Fast Company called him the "Michael Jordan of Hiring," Staffing.org called him “the father of HR metrics,” and SHRM called him “One of the industry's most respected strategists." He was selected among HR’s “Top 10 Leading Thinkers” and he was ranked No. 8 among the top 25 online influencers in talent management. He served as the Chief Talent Officer of Agilent Technologies, the HP spinoff with 43,000 employees, and he was the CEO of the Business Development Center, a minority business consulting firm in Bakersfield, California. He is currently a Professor of Management at San Francisco State (1982 – present). His articles can be found all over the Internet and on his popular website www.drjohnsullivan.com and on staging.ere.net. He lives in Pacifica, California.



8 Comments on “What’s Wrong With Reference Checking? Pretty Much Everything (Part 2)

  1. Long_____Sigh!! What a difference a week makes!

    Last week, I complimented John on citing the evidence. This week I have more questions than compliments.

    John’s first section today focuses on the question, “What are the most accurate indicators of past, current and future performance?”

    Here are my questions derived from John’s first section.

    John says: “There are literally hundreds of potential tools and approaches that can be used. Unfortunately, the vast majority of research on the subject is questionable at best. Rarely are academic researchers trusted to manipulate the assessment processes of organizations hiring in significant volume such that the process could provide valid data, and few if any organizations execute consistently without extreme oversight.”

    My question: How truthful is this pile of assertions? No citation here. This is pure John. A supplemental— If partly true, does that mean we should ignore the hundreds of studies that have appeared in referreed journals and trust John’s intuition?

    Johns says:”Commercially sponsored studies are flawed in that they are executed to prove a tool or approach works, not to test the extent to which it works.”

    My question: Is this even slightly true in a validation study that follows the Dept of Justice and Labor guidelines?

    Johns says:”That said, most studies generally conclude, and I tend to agree, that work samples are often the best predictors.”

    My question:”Is this statement supported by the research– even the study that John cited last week?”

    Now I will shut up and let others answer. Where are Charles and Wendell when you really need them? I expect Keith will have something to say. Others on the Institute of Human Resources–Quality of Hire board may want to chime in.

    But, to paraphrase Robert Frost, I will return if dissatisfied at what I learn from having sighed.”

  2. John,

    Can you comment in the next or in response to this about when someone in a specific discipline, usually IT or other Engineering related profession, takes it upon themselves to call the CURRENT manager or co-worker without approval from HR or Talent Acquisiton. Usually I have heard: “I know someone who works there.” And more often than not that is followed by the fact that he or she has already called the manager of the person being interviewed. Of the few times that this has happened, only one person has not called to inform the HR department that someone called his company. Now, most applications, you know, that legal document, has a disclaimer that the current company will not be called until an offer has been accepted. Yet, IT feels that it is beyond the scope of all this. Frankly, with all the laws they are responsible for regarding privacy, that they do this consistently is rather reprehensible. What do you recommend that smaller HR departments can have more teeth in these type of situations?

  3. @ Dr. Janz: I’m getting so used to many (but not all) of the major ERE article contributors stating their assertions/opinions/anecdotes as facts, that it’s not producing much of a reaction in me anymore.

    I’ll come up with some acronyms for this sort of thing to save us time in the future:
    ONF: “opinion, not fact”, also, TWYS: “that’s what you say”
    AND: “anecdote, not data”
    TIIT: “totally impractical, ivory-tower”.
    SPSO; “self-promoting snake oil” (Not so much of that right here.)

    I welcome additions and revisions to this list.


    Keith “FUBAR” Halperin

  4. Is the subtext clear enough? Pay him some $$$ for his consulting work!

    my comment on Part 1 is below – it applies here, too:

    Dr. Sullivan, let’s assume you have kids and you’ve been in a situation where you needed a babysitter. Do you give your babysitter candidates the MMPI and a cognitive ability test and base your hiring decision solely on their scores? Oh wait…you interview, ask for references and try to learn more about their background? Why do you do that, when tests alone should tell you what you need to know?

    Let’s consider that meta-analytic correlation of .29: First of all, it’s not such a bad value — personality tests tend to predict less, and they are often used as well (will your next article be about how bad personality tests are?). Second of all, this number is an average (meta-analysis is a fancy average) across all types of reference checking systems and all types of criteria — there surely is variation in there, where some reference checks are good (reliable, relevant) and some bad, and some criteria are good and some bad. This means that some studies are likely showing even higher validities (though again, .29 isn’t so bad). Third , to the extent the reference check is predicting a relatively infrequent (low base-rate) behavior (theft, absenteeism), then even putting aside the previous points, .29 is looking even better. Fourth, this validity might be incremental, meaning that it meaningfully predicts criterion variance above and beyond the prediction offered by other measures.

    You point out practical problems with the reference checking process, but why not frame this as an opportunity to improve the process rather than slamming reference checks wholesale? Why can’t a good reference checking process supplement other methods for recruiting, selecting and training talent?

  5. The controversy about this article is because, there is nothing wrong about Reference Checking, but there is much to be improved on how it is done.
    In short, it should have been called, “What is wrong with the practice of Reference Checking?”.
    Because we all know the intimate truth that personal recommendations about anything are VERY powerful if not the best way to get results.

    When you spend the time to improve the practice of reference checking, leverage the latest technologies and make sure it is well done, like we did at Checkster for many years, then you will get great results and we’ll happy to show you. If you do it poorly for sure you will get nothing back.

    So this series of article is confusing the practice with the science. It is like saying, most drawing of a person is only a poor representation of that person, unless of course you know how to draw. Thankfully today we have camera, the same is true for Reference Check 1.0, the phone way, it is old fashion and often inaccurate. Try the Reference Check 2.0 and you’ll see the difference.

  6. Yves,

    I’m not sure I agree with you that the original article and most of the postings that followed confuse the practice of reference checking with the science. I think the original article suggests that science is important in all things used in the hiring process, including reference checking. This is why we, SkillSurvey, the inventor of Reference 2.0, with a patent pending on the technology, make science one of the most important aspects of our business. We’ve ensured that all of our surveys contain work behaviors scientifically validated against on-the-job success. We’ve scientifically validated our solution to be reliable as well as compliant – no adverse impact, no bias. We’ve also scientifically validated that our service improves retention and predicts performance. So I’m not certain that Dr. Sullivan is suggesting science doesn’t matter, I believe he’s suggesting it matters a lot, and we at SkillSurvey agree with him.

    Ray Bixler

  7. The value of this article is that it certainly stimulates discussions. However, there are some things worth noting:

    1. The statistic given without any authority that that “99.9% of all murderers, bank robbers, and major evildoers are first-time offenders,” does not resonate as accurate. As a former prosecutor and criminal defense lawyer, as well as an expert witness that has testified in negligent hiring cases, that figure does not sound correct and it would be helpful if the source was given. The Bureau of Justice Statistics tracks all sorts of data about criminal activity and although I have not had the opportunity to find the number of serious offenses committed by first time offenders, it is actually much more common in my experience for serious crimes to be committed by repeat offenders. Certainly there are offenses committed by first time offenders. As the old English adage of law goes, “Every dog has its first bite.” However, the sweeping statement that suggests nearly all serous crimes are committed by first time offenders requires some serious examination.
    2. I think anyone who has made hiring decision agrees that the calculus of hiring is a very complex process and depends upon a multitude of factors. I am not sure I have heard anyone make a legitimate claim that they can predict with 100% certainty and accuracy if they are making a good hire ( based upon whatever criteria is being used.) The natural conclusion is that the best we can hope to do is to lessen the odds of a bad hire. The fact remains that fundamentally, at the end of day, even when the most scientifically validated tools are used, there is still an element of chance in each hiring decision. On some level, every hiring decision has an element of a “leap of faith”.
    3. I agree with Yves that a service like Checkster improves the odds of a good hire during the selection stage. Certainly, just because no one technique is foolproof does not mean that an employer should not use available resources to try to make the best possible decision.
    4. However, at the end of the day, it is reasonable to conclude as with anything involving human behavior and human beings, hiring will remain an inexact science. Humans are not like molecules that behave according immutable laws of nature, so that once the science is understood, actions and reactions can be predicted with accuracy. Humans are subject to all sorts of influences and behavioral uncertainties, not to mention that fact the people and circumstances change based upon all sorts of factors. Unless we assume human beings are essentially robots, or believe we have totally unlocked the secrets of the human mind and emotions, that element of chance will always be there.

Leave a Comment

Your email address will not be published. Required fields are marked *