What’s Wrong With Reference Checks (Part 1)

Employment reference checking and background screening should win recognition as the weakest of all corporate HR processes. A validity meta-analysis study conducted by Aamodt & Williams in 2005 found that the corrected validity coefficient for reference recommendations and actual job performance was a staggeringly low .29. Despite the facts, 96% of organizations use reference checks as a screening and selection tool, according to a recent survey by SHRM.

Anyone who’s been in the profession for more than a minute or two knows deep down that references suck as indicators.

That’s why it is a task often assigned to an unknowledgeable coordinator, outsourced, or even turned into a multiple choice online form. While there is a handful of people who swear that they do it right, the vast majority do it simply because it has always been done. Maybe reference checking is/was more accurate in smaller communities where everyone knows everyone, but for many organizations today, new hires are strangers. As we enter an era where organizations have years of data and access to sophisticated tools to better determine what does and doesn’t work as an indicator, recruiting leaders should expect more scrutiny of this process by everyone.

Article Continues Below

Government agencies and commissions that oversee equal opportunity and business dealings are paying more attention and increasingly holding organizations accountable for the actions of third-party services commissioned to carry out screening activities. If you are in a leadership role or influence someone who is, it is your professional obligation to ensure that what your organization is currently doing and plans to continue doing isn’t stupid!

The following is a long list of things that routinely plague reference checking processes broken into four categories. If you look for these weaknesses in your current approach and address them, just maybe your validity coefficient will improve.

The Top Five Most Serious Reference-checking Problems

  • Few provide detailed reference information — because former employers face legal action relating to defamation, many firms have a policy restricting reference conversations to factual information such as job title, dates of employment, and possibly salary history. Questions aimed at identifying if the applicant could be rehired by a former employer often go unanswered. The end result: some candidates may benefit from robust reference profiles, while others may not depending on the stance of their former employers.
  • Reference information provided on the wrong person — in larger organizations it is not uncommon for supervisors and coworkers to confuse past employees with the same/similar names, so it is possible the individual being described isn’t the individual in question.
  • Historical situations may not align with present conditions — references often focus on generic aspects of job performance specific to a different environment at a different time. Even if you are able to robustly identify how someone previously acted, you can only speculate that they will act the same way today in a completely different environment. Most firms have no hard data proving that in a fast-changing world where skills and knowledge become obsolete much faster, that past performance at another firm is an accurate predictor of future performance at your firm.
  • Knowledge of the applicant is limited — for a variety of reasons, the individual providing reference information may have extremely limited knowledge about the individual being referenced. For example, they may be able to confirm job responsibilities, but know nothing of actual performance if they were not the person’s actual supervisor. In large organizations, few in HR would have personal knowledge of the individual’s work history or traits.
  • Records are not reviewed — unfortunately, many references (especially telephone references) are provided by former employers who are relying 100% on their memory. You can be almost 100% certain that performance appraisals or attendance records will not be reviewed prior to the reference. Not relying on records and data almost guarantees an inaccurate reference.

Problems Related to a Changing Business Environment

  • Bankruptcies, mergers, and layoffs make getting references difficult — because some employers simply no longer exist, it is often impossible to contact previous employers. Large-scale layoffs might also mean that even if you find the firm, the applicant’s supervisor may no longer be there. Layoffs and lean staffs in HR may also slow responsiveness to reference requests.
  • Culture and language issues complicate — because different cultures and regions have unique ways of handling references, global consistency is difficult. In addition, language issues can make some reference checking extremely problematic.
  • Many have bad credit — firms that used to supplement employment reference checking with credit checks now find that so many people have credit issues stemming from bank actions that relying on past standards will cause you to lose too many candidates.
  • Identity theft issues — identity theft has grown beyond credit card fraud and has entered the employment arena. Unless you take and verify fingerprints, you may be checking the references of a real person who unfortunately is not actually the person you interviewed.
  • Illegal or private information is viewed — supervisors and managers all too often provide illegal or personal information that may unfortunately end up being used in the hiring decision. Reference checking that includes searching Internet or social network sites also increases the likelihood of you obtaining information that should not be part of an employment reference check.
  • Resume lies are more common — in a weakened economy where individuals are desperate to get a job, it is much more likely that they will stretch the truth on their resume, application, or during the reference process. This makes it even more important to get accurate reference information beyond the resume.

Problems Related to the Person Providing The Reference Information

  • Bias produces slanted references — a supervisor or coworker with a vendetta may slant the reference so that it comes out overly negative. Supervisors may provide a negative reference on a current employee in order to improve their chances of keeping them.
  • A positive slant — the majority of references are positive because people try not to say bad things. But when the reference is or was a close friend or colleague who wants to help you, the odds of finding negatives drops dramatically. Personal references are even more likely to be 100% positive because the individual selects only those individuals who they know will provide a positive reference. Managers may give a glowing reference in order to get rid of a problem employee.
  • Self-provided references may mislead or be fraudulent — many firms allow the applicant to provide the names, e-mail addresses, and phone numbers of their references, and it should be obvious that self-selected references are more likely to give positive answers. In addition, unfortunately the names that applicants provide may not have been the person’s actual supervisor. In some cases, the number or e-mail address provided might not even belong to a real firm.
  • No one is fired anymore — if you are using references to find out if someone has been previously fired from a job, you need to realize that because of legal issues, these days most individuals are allowed to resign. As a result, their employment record will not show a termination or the person will not divulge that fact.
  • Leniency effect — when corporate employees do reference checking, there is a measurable tendency to be “lenient” and to downplay negative information. This leniency is caused in part because finding too much negative information will mean that the reference checker will be asked to delve deeper (more work) or that they must reopen the position search. People providing references can also be overly positive because they know that a bad reference may cost the applicant (who may also be a friend) their next job.
  • Competitor firms will not cooperate — in a highly competitive business world, it can be increasingly difficult to get any form of information from a major product or talent competitor.

Administrative Problems With the Reference Checking Process

  • No company data to prove they work — because many firms execute their own reference-checking process, it would be a mistake to rely on external research to prove the accuracy of the process. Unfortunately, I have never come across an organization that routinely gathers and analyzes data on the predictive accuracy of their reference process.
  • Reliability is low — most corporate reference-checking processes (unlike vendor processes) are loosely designed and poorly controlled. Moreover, when reference checking is decentralized and many untrained individuals are conducting them, the execution (consistency and reliability of the process) slips even further.
  • The person doing the checks are weak — many times the employees who are given the assignment are not trained in reference checking or knowledgeable about the organization or job specifics, making it difficult for them to dive deeply into relevant matters.
  • References are often not completed until after the time of hire — due to the cost and time involved, some firms now complete reference checks after the individual is already on the job.
  • Lawsuits are real — final hiring decisions are made based on reference-checking results, and in a down economy, more rejected candidates resort to legal remedies. Unfortunately, during legal discovery you may find that your references have a sudden memory loss and they will not repeat what they initially said, leaving you in legal jeopardy (which is why written references are superior, even though they are slower). In direct contrast, failing to check references may result in the possibility of negligent hiring claims.
  • Reference checking is slow — in order to get documentation, it is important to use written references. However, written references take much longer, and a higher percentage of individuals are reluctant to participate in references when they must provide their signature. Reference checking is often not a high priority among recruiters or support staff even though slow reference checking may cause you to lose top candidates.
  • The use of leading questions — a significant percentage of the reference questions that I encounter are binary questions pertaining to positive traits that almost always elicit positive answers. Requiring the individual to force rank or pick from a list of both positive and negative answers is a superior approach. Open-ended questions result in different information, making comparisons between candidates difficult.
  • Poor documentation — telephone references often suffer from poor documentation and capture no signatures, which can be a problem should legal issues arise.
  • Automated reference-checking limits depth — web-based reference-gathering systems shift the burden of getting references to the candidate, added work that may cause some candidates to drop out. In addition, to make such tools possible, the scope of the reference is limited and the ability to dive deeper into specific responses is virtually non-existent.
  • The same process for all jobs — even though some jobs demand much more thorough reference checking (i.e. those who handle money, those with childcare responsibilities, etc.) most corporations use the same exact process for all jobs.
  • A checklist is not used — rather than using a checklist of standard questions, many supervisors who check references “wing it” or make up their reference questions as they go, resulting in no consistency. If standard questions are used, they are often not weighted based on their relative importance in predicting future job performance.
  • Mixed results make comparisons and conclusions difficult — the more references you check and the further back you go, you increase the likelihood that you will be faced with mixed references. Also, because an individual may receive several glowing and one extremely negative reference, making a decision based on consistent results may be difficult. When candidate slates include those recently out of school with little or no experience, managers are forced to make difficult comparisons between candidates with business experience and those who only have school related references.
  • All references shouldn’t receive equal weight — older references should be given a lower weight because the further back you go in a candidate’s work history, the less likely that the results will be valid predictors.
  • No permission signature is obtained — if you are using resumes instead of application forms, you may be checking references without written authorization. If you inadvertently call their current employer, you may get them fired, which could lead to them taking action against you.
  • It is not a continuous process — most references are done exclusively during the hiring process, but criminal, motor vehicle, or credit issues could occur well after the individual has been hired.
  • Inclusion requires flexibility — if your recruiting or diversity goals include attracting individuals from lower-income groups, it is important to realize that some of these individuals may be more likely to have criminal, driving, credit, or job-jumping issues that can result in weaker overall reference scores.
  • Contingent workers are treated differently — some organizations fail to require reference checks or allow vendors to use less stringent processes for temporary, part-time, and contract labor. As contingent workers become a larger percentage of a firm’s total labor, this inconsistency will need to end.

Dr. John Sullivan, professor, author, corporate speaker, and advisor, is an internationally known HR thought-leader from the Silicon Valley who specializes in providing bold and high-business-impact talent management solutions.

He’s a prolific author with over 900 articles and 10 books covering all areas of talent management. He has written over a dozen white papers, conducted over 50 webinars, dozens of workshops, and he has been featured in over 35 videos. He is an engaging corporate speaker who has excited audiences at over 300 corporations/ organizations in 30 countries on all six continents. His ideas have appeared in every major business source including the Wall Street Journal, Fortune, BusinessWeek, Fast Company, CFO, Inc., NY Times, SmartMoney, USA Today, HBR, and the Financial Times. In addition, he writes for the WSJ Experts column. He has been interviewed on CNN and the CBS and ABC nightly news, NPR, as well many local TV and radio outlets. Fast Company called him the "Michael Jordan of Hiring," Staffing.org called him “the father of HR metrics,” and SHRM called him “One of the industry's most respected strategists." He was selected among HR’s “Top 10 Leading Thinkers” and he was ranked No. 8 among the top 25 online influencers in talent management. He served as the Chief Talent Officer of Agilent Technologies, the HP spinoff with 43,000 employees, and he was the CEO of the Business Development Center, a minority business consulting firm in Bakersfield, California. He is currently a Professor of Management at San Francisco State (1982 – present). His articles can be found all over the Internet and on his popular website www.drjohnsullivan.com and on staging.ere.net. He lives in Pacifica, California.

 

Topics

22 Comments on “What’s Wrong With Reference Checks (Part 1)

  1. Thank you, Dr. Sullivan. Reference-taking is a joke and has been for a very long time. From the Recruiter/HR point of view, it’s merely a due-diligence/CYA operation, because any given refernces except from a fool are likely to be all positive. Why would you give as a reference anyone who might not describe you in glowing terms?

    Cheers,
    Keith

  2. With all that in mind do you think it’s even worth doing reference checks at all? Under what circumstances with what processes in place would it be?

  3. I would agree that traditional reference checks (we call them Reference Check 1.0) are often not too reliable. BUT true good reference checks done like a peer review is one of the most if not the most reliable assessment method. That is why all serious search firms or recruiting methodology include them, and that is what I experience every time I hire.

    The problem today is more around 2 core issues:
    1. the legal issues and lack of candid feedback.
    2. the lack of time, recruiters or hiring managers don’t take the time to do it well.

    If you can address those 2, you will have the solution.
    I have described in my book Reference Check 2.0 how the social web can help you to do it.

    Yves Lermusi
    CEO Checkster

  4. Congrats to John on citing the science. For those interested in such things, the full SIOP session, including the slides for the cited article, can be found at this link. http://maamodt.asp.radford.edu/Research%20-%20IO/SIOP%202005%20-%20Reference%20Symposium.pdf Here we find ratings of qualitative letters of reference do much worse than structured reference ratings (.16 vs. ,29). We also learn that assessments of cognitive ability, work samples, and structured interviews almost double the accuracy of reference ratings, when considering all the published objective studies.

    <bThus my question for Yves. What is the average corrected validity for reference ratings collected by Checkster (or other ‘social media’ based assessments)? I sympathize with the challenges in obtaining this value, but how can we know how much better your reference ratings are compared to those reported by Aamodt and Williams until we have that number?

  5. I would like to add Qualitative Reference Checking to a list of *Common Dysfunctional Hiring Practices.

    Some others:
    1) Paying 8-10x the amount for onsite scheduler/coordinators, sourcers, and low-level recruiters/ job posters/board scrapers when virtual ones would do as well or better.
    2) Having large numbers of interviewers (5+) and/or substantial rounds of interviews (3+) to hire mid-level employees.

    3) Being unwilling to pay $2.00/hr for Candidate Care Specialists to make certain each applicant has a professional (if not actually pleasant) application/interview/hiring process.

    4) Creating hiring processes/purchasing relevant technology without substantial user input and feedback.

    5) Not making on-time, on-budget, quality hires a deliverable for hiring managers.

    What else can you add to this list, Folks?

    Cheers,

    Keith
    keithsrj@sbcglobal.net

    *If you can show me validated studies that indicate these may not be as dysfunctional as they seem, please let me know. -kh

  6. Now Keith (Show me the Research) Halperin. A corrected validity of .29 is not as terrible as John might think. It is at least 15 points better than subjective resume sorting and adds 10 points to the unstructured 1-on-1 interview. I was hoping that Yves had research that points to even higher validity for his flavor or structured reference checks.

  7. Thanks, Dr. Janz. It occurs to me that the purpose of references may not in fact be to get a better hire, but to avoid a poor one. It would be analogous to me locking the door to my store at night to avoid theft, as opposed to locking the door to improve sales. If that were the case, we might need to set up a study to show if there were a negative correlation between good references and poor hires, as opposed to a positive correlation between good references and good hires.

    Cheers,

    Keith “Taking Off My Thinking Cap for Now” Halperin

  8. We made the shift in early 2010 to an automated reference system that has helped to eliminate checker self-bias, and to decrease the time it takes to get the references back. It provides great detail and allows some anonymity for the reference provider to protect from a legal standpoint.

    I’m happy to discuss the product with you if you shoot me an email. It’s not my product, I’m just a happy customer.

  9. Reference Checking….What’s Science Got to do With It? Kudos to all for recognizing the importance of science in job candidate selection. In fact, there is a growing body of scientific evidence supporting the value of reference feedback. Peer-reviewed studies presented from 2008 and beyond at the annual meetings of the Society for Industrial and Organizational Psychology (SIOP, div. 14 of the American Psychological Assoc.) show 360o reference feedback to be a reliable and predictive measure of job performance such as retention and supervisor evaluations (SIOP, 2009). In addition, our research on the world’s largest database of reference feedback shows no sub-group differences (SIOP, 2008), thus supporting employers in their efforts to be compliant with EEOC and OFCCP guidelines. Specific information on the psychometric integrity and compliance of reference assessments can be found on our website. Or, feel free to contact me directly or visit with us in person at the 2011 SIOP meeting (April, Chicago) to learn the significant results from our newest predictive validity study.

    Cynthia A. Hedricks, Ph.D.
    Chief Analytics Officer, SkillSurvey, Inc.

  10. Excellent article. However, it is worth noting there is a substantial difference between a “reference check” that seek to determine if person is a good fit for a job, and “employment verification,” which essentially verifies factual claims by an applicants as part of the due diligence process. In other words, contacting past employers must be clearly analyzed from two different prospective –qualitative checks to determine fit, and quantitative checks for the purpose of establishing facts such as start date, end date, job title and if available.

    It should also be kept in mind that “reference checks” and “employment verifications” are conducted at different points in the hiring process. Reference checks are an attempt to decide if a person should be made an offer. An employment verification is utilized to establish due diligence in hiring, and to determine if the person engaged in any fabrication so that the offer should NOT be made. Members of the background industry have long advocated that reference checks be conducted in-house because by definition it requires judgment calls to be made by someone knowledgeable about the firm and the position. Employment verification is something employers can do in-house but often outsource. It should also be noted that screening firms engage in substantial fraud prevention techniques in the process, such as independent verification that a past employer actually exists and that telephone numbers are valid, completely independent from any claims made by an applicant.

    I have had the opportunity to testify as an expert witness in court cases involving negligent hiring where the most horrible things have occurred, including cold blooded and brutal murders, rapes and child molestation, all as a result of unfortunate hiring decisions. What many of these cases had in common was a failure to conduct even the most rudimentary employment verification. For a number of technical reason, some of these cases involved situations where a criminal checks would not have revealed a “Red Flag,” but a simple past employment check would have alerted the employer there was an issue. In some of these cases, a five minute phone call would have revealed that the person either fabricated their employment or made up the employer. There is also an interesting emerging body of science that suggests that if a person is dishonest in the way they got thier job, there is a much higher probability of dishonesty once in the job. (Richard L. Griffith, et. al “A Closer Examination of Applicant Faking Behavior,”

    Any rate, since recruiting and hiring the right people is such a complex task, it is always critical to determine what stage you are in during the hiring process, and how each tool applies to that stage. Arguably there are seven stages in the hiring process and each stage has different tools and considerations. See: http://www.esrcheck.com/wordpress/2010/09/21/using-online-referencing-tools-and-background-checks-to-optimize-employee-selection/

    I agree with the article that reference checks can lead an employer down the wrong road. But it is also important to understand that a failure to conduct an employment verification aimed at avoiding fraudulent claims before the person is onboarded, creates a substantial risk to an organization.

  11. What a load of donuts!

    Dr. Sullivan, let’s assume you have kids and you’ve been in a situation where you needed a babysitter. Do you give your babysitter candidates the MMPI and a cognitive ability test and base your hiring decision solely on their scores? Oh wait…you interview, ask for references and try to learn more about their background? Why do you do that, when tests alone should tell you what you need to know?

    Let’s consider that meta-analytic correlation of .29: First of all, it’s not such a bad value — personality tests tend to predict less, and they are often used as well (will your next article be about how bad personality tests are?). Second of all, this number is an average (meta-analysis is a fancy average) across all types of reference checking systems and all types of criteria — there surely is variation in there, where some reference checks are good (reliable, relevant) and some bad, and some criteria are good and some bad. This means that some studies are likely showing even higher validities (though again, .29 isn’t so bad). Third , to the extent the reference check is predicting a relatively infrequent (low base-rate) behavior (theft, absenteeism), then even putting aside the previous points, .29 is looking even better. Fourth, this validity might be incremental, meaning that it meaningfully predicts criterion variance above and beyond the prediction offered by other measures.

    You point out practical problems with the reference checking process, but why not frame this as an opportunity to improve the process rather than slamming reference checks wholesale? Why can’t a good reference checking process supplement other methods for recruiting, selecting and training talent?

Leave a Comment

Your email address will not be published. Required fields are marked *