Employment Tests Are Becoming Irrelevant for Predicting Job Success

Screen Shot 2013-05-06 at 1.42.47 PMLet’s talk about the future of predicting job success and why the world’s biggest evangelist for pre-hire assessments thinks tests are in danger of becoming extinct (and is OK with it).

There are a number of emerging trends in hiring right now that center around the currency of the new millennium: data. The impact of our ability to collect, organize, and interpret data is rapidly changing all areas of the economy. Should employment be any different? There are three ways in which data is slowly killing the employment test as we know it.

The Impact of Publicly Available “Free-range” Data

People born in the past decade or so, along with all persons to come, will begin accumulating a personal digital fingerprint that will be associated with them from cradle to the grave. While there may still be some things that can be kept private, most everyone’s every move, preference, and connection will become publicly available to anyone who is interested.

The ideas behind using publicly available data to predict job performance are gaining big time attention these days as evidenced by these recent Business Insider and New York Times articles.

We are even starting to see research that suggests we can gauge an individual’s job success from social media data such as one’s Facebook usage.

Theoretically any data that exists out there can be thrown into the hopper, and its impact on job performance related variables examined. We I/O psychologists refer to this strategy as “dustbowl empiricism” because it lacks any real guiding theory, instead relying on cold, hard numbers as the only truth. This purely empirical strategy was popular in the 1950s and 1960s when various life history factors were used to build predictive employment tests or “weighted application blanks,” and is in fact still used today to some extent.

While this strategy does work, there is something unsettling about leaving out some underlying theory or framework and failing to rely on predictive factors that make rational sense. For instance, if we could show shoe size correlated with job success but had no reason why, should we feel comfortable hiring based on this factor? We are now revisiting this decades-old debate but within an entirely new framework that involves literally billions of new possibilities.

The Rise of Structured, Sanctioned, Verifiable, Shareable Personal Data

Also gaining in popularity at present are many types of data that people directly create and groom for specific purposes related to employment. Consider if you will the LinkedIn profile, which is now becoming the defacto resume. It is fluid and fully deconstructible for use in hiring situations. It has all the info needed by an employer to gauge your ability to contribute to their cause.

My friends at DDI just shared some really cool research with me in which they were able to find a relationship between LinkedIn profile elements and job success. In this purely exploratory study of 587 people across 11 organizations, the researchers examined the impact of LinkedIn profile elements such as number of connections, position progressions, and employment gaps on performance. They were able to show that a variety of rationally hypothesized factors (i.e., # of positions held) impacted key variables such as turnover.  They also found evidence of less-rational relationships such as the fact that those who gave more recommendations had higher job success levels than those who received them. This study showed a direct connection between profile elements and performance and as such I consider it to be a groundbreaking study.

Beyond LinkedIn, badging/credentialing represents an emerging concept in which persons can essentially gain digital merit badges that carry with them an assumption of competence or skill in a certain area. Currently, a great example is the Klout score, which can be used to help evaluate someone’s level of engagement in social media and is often cited as a key piece of data for those being evaluated for marketing and advertising jobs.

Article Continues Below

We are also seeing virtual badges that are awarded based on the completion of coursework or mastery of a subject that is verified or validated by a third party that is able to officially sanction it. Imagine if you will that these verified pieces of evidence can then be attached to a public data profile and that they can be easily reviewed via a predictive algorithm used by an employer without the individual even applying for a given job.

Combine this with the emerging trend of free online learning and coursework and the connectivity of social media and you have just hit on the future of sourcing and screening tools. It will not be long until anyone, active job seeker or not, is being notified of jobs for which they realistically are a good fit, without lifting a finger. In a few years a recruiter could be wearing the Google Glass with an app that will allow them to compare people they meet in person throughout their day to open requisitions in real time.

An Increase in the Quality and Quantity of Job Performance Data and Company Information

One of the most significant barriers to proving the value of hiring practices is slowly and steadily being torn down. I’m talking about the difficulty of collecting quality job performance data required to support the proper evaluation of pre-hire data sources such as tests.

Ask any I/O psychologist and they will tell you that getting companies to provide access to the data needed to do predictive analytics is like pulling teeth. In many cases it is just not available, and in others the company simply does not care about collecting or maintaining it. We are often forced to use low-quality job performance data as part of our evaluation program. You can’t predict things accurately if you don’t have good data on both sides of the equation. We I/O psychologists refer to this difficulty as “the criterion problem,” and it has held us back for decades.

Change is coming when it comes to performance data. Workforce analytics is here to stay and at its core resides data of all shapes and sizes. We are now in love with data and most major companies are tracking everything.  This allows us new proxies for the data we have not been able to collect and also provides a quantum leap in the ability to explore data relationships to look for meaningful patterns. The shift in the perceived value of data is perceptible and it is going to have a huge impact on the ability of analysts like myself to begin showing companies the direct value of their employment practices. In some cases these practices may be tests, but we should not constrain ourselves to the notion that tests are the only predictors of value. Once the post-hire data stream becomes more available, we should be prepared to find all kinds of new predictors of value.

I am not quitting my day job as a testing guy just yet. But things are changing fast and those of us in the testing business are going to be tested ourselves. The data we begin to see may make it hard for us to argue with the impact of digitized personal data on job success. The truth lies somewhere in the middle, and the best formula will include both unstructured data and digitally shareable credentials that are verified via accepted sanctioning bodies. This digital fingerprint will be used (along with other personal data) to help match people with jobs and to quantify the direct impact of hiring on an organization’s strategic objectives. Yes, tests may be involved somewhere in the data stream, but they will cease to be the only focus when it comes to predicting job success.

Dr. Charles Handler is a thought leader, analyst, and practitioner in the talent assessment and human capital space. Throughout his career Dr. Handler has specialized in developing effective, legally defensible employee selection systems. 

Since 2001 Dr. Handler has served as the president and founder of Rocket-Hire, a vendor neutral consultancy dedicated to creating and driving innovation in talent assessment.  Dr. Handler has helped companies such as Intuit, Wells Fargo, KPMG, Scotia Bank, Hilton Worldwide, and Humana to design, implement, and measure impactful employee selection processes.

Through his prolific writing for media outlets such as ERE.net, his work as a pre-hire assessment analyst for Bersin by Deloitte, and worldwide public speaking, Dr. Handler is a highly visible futurist and evangelist for the talent assessment space. Throughout his career, Dr. Handler has been on the forefront of innovation in the talent assessment space, applying his sound foundation in psychometrics to helping drive innovation in assessments through the use of gaming, social media, big data, and other advanced technologies.

Dr. Handler holds a M.S. and Ph.D. in Industrial/Organizational Psychology from Louisiana State University.

LinkedIn: https://www.linkedin.com/in/drcharleshandler






14 Comments on “Employment Tests Are Becoming Irrelevant for Predicting Job Success

  1. Yes I did not even get started with where this is leading in terms of what is viewed as acceptable by the EEOC and OFCCP. This will be a whole new battlefront. How it goes down remains to be seen. But the government simply does not have the resources to police this at the level they would need to. This is a continuation of status quo in this area.

  2. Interesting stuff. I think one of the main difficulties going forward as this develops is to put a brake on people who try to use the data incorrectly. Many of the hiring managers I’ve worked with would be more than happy to default to not hiring people with small feet if they thought it could reduce their risk in making a hiring decision. Many people make correlation/causation fallacy every day on a serial basis, it’s going to be hell to try and get them to approach the subject rationally.

  3. Richard, I couldn’t agree with you more. Based on my experience in AA compliance, in a majority of recruitment and selection processes, including the current use of testing, many HR processes are designed with the intent of ruling people “out of” rather than “into” appropriate channels that would allow for full and accurate assessment of one’s talent/qualifications.

  4. @ Richard, @ Teresa: Well said. I also think that the more these tools are (mis)used, the more people will learn how to game them, e.g., people who read this article may start giving out more recommendations on LI to improve their prospects.



  5. As an I-O practitioner I have seen hiring managers at all levels take great leaps of faith in predicting job success from a candidate’s background (e.g., they won’t fit our culture if they came from “x company”, they’ll fit the job because they were successful as an “x”, they won’t do well here because they went to “x college”, etc). With even more candidate background data readily available, the challenge of identifying the truly predictive data is a greater challenge than ever.

    On the bright side the availability of these data help immensely in creating a candidate pools more quickly and with a more diverse set of candidates who are more likely to fill the need. However, the challenge of accurately predicting fit and future performance still remains.

    What Charles says in his article makes me think that the use of good pre-employment assessment (tests and beyond) will be more important than ever. Innovating and expanding the practice of matching the right person to the right job in the ever changing digital world is an exciting challenge that all of us “testing guys” can be very excited about.

  6. Great point Charles.
    I definitely agree that tests have value and will continue to do so. But we can’t focus myopically on them as the sole data of value in predicting performance.

  7. I think there are a couple of important points to consider:
    1) I would consider it extremely unlikely that there are likely to be substantial numbers of non-trivial factors which are equally valid for all individuals for all jobs across all periods of times, i.e., “I doubt they’ll work for everything.”
    2) Humans have inherent cognitive biases- we can’t help it. Unless test creators and employers take these into account, the results will be somewhat faulty/limited.

    As an aside, it would be interesting to have very successful individuals in various jobs within a company take these tests and see if they have the backgrounds which would indicate their likely success. e.g., would these tests have indicated that young Steve Jobs and Bill Gates would be dynamic and effective CEO? Maybe, maybe not.



  8. Charles: Very interesting article. Of course, none of us knows what exactly will happen in the future. I believe one thing will remain constant, however. If companies continue to use this type of “tool” to determine fit without having gone through the process to effectively determine what’s necessary for a candidate to do the job successfully, the hiring process will never improve.

  9. @ Carol: “the hiring process will never improve.”
    As our friends at Despair.com say:
    “If you’re not a part of the solution, there’s good money to be made in prolonging the problem.”



  10. I completely agree with this article. I’ve been turned down for jobs because of these “tests.” It’s very frustrating as a jobseeker (and slightly infuriating) because they aren’t even willing to look at my resume based on the results. I was in retail/sales for over 9 years and went to apply for a different job in retail. They didn’t even look at my resume or experience, just dismissed me out of hand because of my test results. Now I’ve gone back to college at age 30 to change careers to Medical Billing but it seems no one wants me because of these tests!

Leave a Comment

Your email address will not be published. Required fields are marked *