Validating a Personality Test

Sure, it’s easy to say engineering, legal, IT, or actuarial jobs require technical degrees. People in these professions need a substantial amount of education to practice their trade. But we all know from watching folks in these professions that it takes more than a sheepskin to be successful. Sometimes, it takes certain personality factors to make a good job fit.

Job performance is a two-sided coin. On one side, all the hard skills required for the job, and on the other, all the social factors. For the purpose of this article, we’ll assume personality is on the social factor side.

The important things to ask are, “How do we decide which personality factors are important?” and “How do we measure them?”

Wrong Way

A pre-screen test (i.e., interviews, tests, resume review, application form) is supposed to determine whether the applicant is qualified. In a perfect world, good scores predict good performance, and bad scores predict bad performance (assuming we hired applicants with bad scores).

Well, there’s a right way and a wrong way to link scores to future job performance. Unfortunately, the wrong way is the norm.

The wrong way comes in two varieties:

  1. Someone likes XYZ test, gives it to every applicant, and plays amateur shrink.
  2. Someone gives a one-size-fits-all test to current employees and correlates scores with job performance.

Both varieties are filled with mistakes. I call these WW1 and WW2.

WW1 is common among most organizations I know. WW2 is common among folks who might have taken a class in statistics, but skipped the class in measuring human performance. Either way, both WW1 and WW2 turns away good employees and hires bad employees. Of course, you don’t have to take my word for it. Just do a controlled study.

What is This Thing Called Performance?

If we want to use a pre-hire test, common sense says to first determine what factor we want to measure. Second, choose an accurate and trustworthy test that measures this factor. Third, make sure the test predicts job performance. Sound simple? Read on.

Some people think they can measure “performance” by looking at performance appraisals or supervisor ratings. However, we all know most performance ratings are primarily personal opinion shaped by friendships, power, ambition, or sucking- up (sucking-up is a technical term for getting other people to think you are actually competent at what you do).

As a result, most performance review data means:

  1. The employee was unskilled at shifting blame for his or her mistakes and now must wait for the rater to get Alzheimer’s.
  2. The employee is highly skilled at shifting blame and charismatic?and thus, is well-liked by the boss.
  3. Employees who are skilled at corporate camouflage.

In general, any data that can only be evaluated at the end of a long performance period is error-prone. Sales volume, for example, is fuzzy because it’s a function of persistence, fact-finding, learning, strategizing, presentations, adapting to buyer personalities, economic conditions, market conditions, and so forth.

To use an analogy, if you want to measure the quality of grapes in a fruit salad, you cannot put all the fruits into a blender, press mix, and come back six months later with your grape-o-meter.

Performance data should be easy to see and easy to measure.

Big Nets Often Contain Big Holes

There are some folks who think they can give a multi-factor personality test to employees, and then examine the results to identify correlations. Sorry. No cigar for them, either.

Let’s assume for a moment these folks got the performance thing right. Giving one great-big test to everyone and looking for results is a major mistake, primarily because a correlation is not always a causation. In other words, just because two things are statistically associated does not mean one causes the other.

Consider the famous correlation of rising skirt hemlines with rises in the stock market. If one actually caused the other, today’s hemlines would probably be located somewhere above the waistline, causing most male traders to become so distracted a market crash would be inevitable.

Statistics is deaf, dumb, and blind. It can show how two factors move with relationship to each other. Only a human being can decide whether one factor causes the other. For example, global organization or teamwork factors may correlate with performance, but if organization and teamwork don’t cause performance, then hiring people based on these scores will reject qualified candidates and hire unqualified ones.

Article Continues Below

How Many Factors?

The DISC is a widely used tool. Just check off a few dozen adjectives and you can get a mini-novel. It looks good, but is the DISC really as comprehensive or accurate as it looks? You decide.

The DISC is based on a two-factor theory that is almost 80 years old. It states that personality is a function of being either active or passive in a friendly or unfriendly environment. Does this sound sufficiently comprehensive to define your job?

I suppose hiring everyone who described themselves as mini-Napoleons would produce an employee base of toy soldiers who continually fought for power. However, would that be productive?

I once visited a company where the HR guru only hired people who excelled at teamwork. After a few years, it had 300 employees who would not schedule a meeting unless everyone could attend, make a decision unless everyone agreed, and not confront a production problem because it might hurt someone’s feelings.

Did they get the results they expected? No. They got the results they measured.

University research shows it takes about nine to 10 factors to predict job performance. Six or seven are based on job fit and three are based on job attitude. It is probably a good idea to check your test vendor to see if the test you are using was developed based on this research. If not, it probably won’t either predict the performance or attitudes you need.


As I said before, statistics is deaf, dumb, and blind. It measures associations, but only under the right conditions. For example, data sets smaller than 25 people are filled with too much individual information and not enough group information.

That is, comparing the characteristics of 15 high producers with 15 low producers is probably going to contain a substantial amount of error.

The bottom line? Trustworthy statistics needs large numbers of people to be accurate.

What about the size of correlation? Should we shout and celebrate a correlation of .30? Does it mean we have a 30% relationship? No. It means we have a 9% relationship. The technical details would cause John Wayne to weep, but correlations have to be squared to learn how much performance they predict. People who do not understand statistics are easy to fool.

Wrap Up

Personality can be a very important thing to measure pre-hire because it can provide considerable insight into job performance. However, before you can trust a personality test to predict performance in your job, you have to clearly define what you want to predict; identify the personality factor that causes it; find a test developed specifically to measure that factor; either conduct your own study or transport someone else’s work; and finally, realize that personality is only one part of the puzzle.

Otherwise, if your organization is still using WW1 or WW2, it is probably turning away good employees and hiring bad ones.


12 Comments on “Validating a Personality Test

  1. Dr. Williams, thank you for another thorough and well-thought out article. It has been said that there are lies, damned lies and statistics. We need to be careful not to be over-reliant on statistics because we do tend to measure the things we are looking for. And too often we feel affirmed if or when we find it. Relying too heavily on one form of measurement will skew our perceptions and hinder us from finding and hiring the best.
    Ron Katz

  2. I always get nervous when the ‘p’ word is mentioned by recruiters. Why don’t we say on-the-job behaviours? Surely all behaviour is context related? For example, in one 24 hour period I may go to work, got out drinking with my friends, go to church, then play a competitive match of tennis. I may display four different sets of behaviours across these four different contexts yet, surely, my personality hasn’t changed in that 24 period, has it?

  3. I agree that personality tests can be applied arbitrarily. We have recently started using a tool called PDP (thru Predix) and it helps us define a job model with success indicators at the beginning of each search. Each candidate is then profiled and matched against the baseline job model. So far we like it quite a bit.

  4. There are so many aspects to personality that it is essentially impossible to measure completely and accurately.

    I greatly enjoy reading research into various aspects of personality psychology. I can discuss MBTI with great interest for hours. I can also type most people within minutes of meeting them (in the right situation) with a high degree of accuracy.

    Using that information or system to make or not make a hire would be unethical. I do find that I can predict how well I will get along with someone based on MBTI, but its much more of an art (read: parlor trick) than actual science and I do meet people with highly anomalous personality traits.

    Another popular measure of personality is the Five Factor Test. You can find out how ‘neurotic’ you are (or aren’t) and you can get a glimpse into traits like dutifulness and agreeableness and what they mean. Again, it is fun and interesting and often an excellent tool for self discovery but I would be tempted to refuse to take such a test for an employer (I would probably take it because I always like how mine come out).

    A more interesting aspect of ‘personality’ to emerge in the last few years is Marcus Buckingham’s work with the Gallup Organization on identifying 34 human ‘Talents’ and describing their application. They have a test that will identify an individuals ‘Five Signature Themes’ and list your 5 most accessible talents.

    One would think that you could simply identify which talents were needed in a given role and test for them. Not so fast – Buckingham makes a compelling case that there are MANY paths to success and often two successful people will have radically different talent profiles – indeed, there are over 30 Million possible combinations just to describe a persons ‘Top 5’ and we all use more than 5 talents.

    Success is not the opposite of failure and as was mentioned in the article, there is a danger in thinking that one way to success is the only way.

    The often overlooked factor is the role of the manager in identifying people, assessing them and managing them in such a way that they are free to find their own path to success. The best managers facilitate this process, the worst managers stifle it.

    In our ever more connected and analyzed society, we are being conditioned to ignore our instincts and our intuition. The best leaders through out time did not have ‘personality tests’ or statistical analysis as a tool to help them succeed – they relied on vision, leadership, intuition, courage and experience to identify and cultivate their lieutenants and advisers.

    In an age were we believe the fairy tale that ‘we can become whatever we want’ if only we wish it hard enough and believe in ourselves. We want to believe that a magic formula or a computer program can take the place of human insight.

    Do I think these tools are worthless? No – they help describe the range of human talent and capability and give us language to describe why one person likes to build, one person likes to relate to others and yet another person is happiest when they make all the numbers add up.

    As a culture, we need to become more fluent in the language of talent, aptitude, psychology, ability and accomplishment.

    Excellent article.

  5. I am not familiar with that test, but as with any profile, the key points are:

    Working from a boiler-plate job requisition is filled with assumptions and the potential for hiring mistakes. Mainly, successful people already in the job might have totally different personalities than the one built using the profile.

    Your job is to determine future performance… not personality match. Does this mean matching applicants to average employee personalities? To a theoretical paper profile? Is the paper profile the same as the actual job? Does the profile account for future job changes? What definition of performance are you using? Does the profile of the high performers differ substantially from the low performers? Do the personality factors ’cause’ performance? Will the test create ‘clones’?

    I don’t suggest using anyone’s profile without expert help (i.e., a human being with experience and training in psychometrics)…It takes new graduates in the field 2-3 years of practice before they become competent…I seriously doubt there are any shortcuts.

  6. I would like to add one additional comment to Dr. William’s thoughts on personality testing.

    Over the years I have helped 100’s of companies who used Personality Testing to hire people who supposedly did not past the test but had very successful track records. In many case they were in the top 10%.

    What was the one thing that these test all missed? It was the fact that very successful people do the things that average or unsuccessful people won’t do. For each of us that is different. So the test may tell you what people don’t like to do but they don’t tell you if and how they will do it any way.

    The key to success is not just doing the things you like to do. It is the willingness to do the thing you don’t like to do when they need to be done.

  7. I try not to think of myself as overly sensitive, but I have no idea what statistics and hemlines have in common. I am slightly offended by the mention of what seems to be a sexist remark that has absolutely no bearing on making your point.

    Another point, I would love the reference for the university research you site that states the number of factors to be measured in order to successfully predict performance. It is inherent to the position what makes up success and some jobs require more ?factors? to be successful than others. So how could they arrive at 9 to 10 as the magic number?

    Lastly, should you celebrate a correlation coefficient of .30? Depends on the confidence level? at 99% heck yeah you should celebrate! Especially if the standard hiring practices before were resume reviews, unstructured interviews, and education levels which equate to an average coefficient of less than a .30 for all three combined.

  8. The study between statistics and hemlines is well-known..I used it as an example of mis-used stats. You can read about it here:

    I seriously doubt it was a sexist study…just a bad one. You can ready many more by Googling ‘hemlines and stock market’. It was probably done by a business major and not a psychometrician.

    You might also want to do some research on John Holland’s Self-Directed Search work. It is among the most comprehensive job-personality/job-fit studies ever done and is well-respected in the industry. Add the SDS factors with the B5 research comparing job performance with personality factors, and you get 9-10 key factors associated with job-fit and job performance. Simple math.

    Celebrations about statistics were not the point of the was to show that a .30 correlation is not the same as 30%.

  9. Great article, brings the issues right up front.
    I believe in order to compete globally we need to hire employees who bring diverse ideas to the table. This group of candidates may be excluded using a personalilty test.
    What ‘University research’ is the author referring to in the How Many Factors? section?

  10. Ross,

    I think you have stumbled on to the core problem. A persons inherent personality is really not important – what is important is their ability to do a given job effectively.

    The reason ‘personality’ is such a popular subject is that we believe that it is the most predictable, dominant factor in motivating behavior.

    In terms of the activities you described, your ‘personality’ will affect the way you behave:

    When you go to work, are you just getting through the day, are you trying to make money, are you angling for a promotion or do you most enjoy connecting with others? Do you promote harmony and get along with others, do you plan every possible detail, do you take things as they come or try to control every situation to maximize the chance of a favorable outcome?

    When you go out drinking, what is most important? Is it how you are perceived by others, the atmosphere of the bar, the quality or nature of your beverage, the subject of the conversation or do you relish the chance to be the center of attention or just connect with and support your friends?

    When you play Tennis – do you play to win? Why? Are you driven to win at any cost or will you pace yourself against a lesser opponent to help them enjoy the game more? Do you measure your performance by how you look, how you score or how much you improve each time you play?

    At church – are you there because you like to be seen at church? Does it have deep meaning to you or do you do it because you ‘just know’ that it is the right thing to do? Do you have frequent doubts and if you do, do you feel guilty or do you feel that doubting is part of the spiritual process?

    You just described four activities that someone could do, but the reason people do each of those activities will vary massively. Just knowing you play Tennis does not tell me much about you – but if I know your MBTI type is ENTJ or ENTP, I can begin to predict with some accuracy, some of your motivations. If I know that the Strengths Finder assessment has identified competition as one of your top signature themes, the picture becomes clearer still. Your Five Factor test might show low levels of ‘agreeability’ – with each new detail, it becomes clear to me that you are a competitive and driven and you like to win – especially if it means someone else gets to lose.

    If I know those things about you and you are applying for a job in direct sales, my interest is piqued.

    If you are applying for a job that requires high levels of cooperation, empathy and deference, I might be a little concerned (enough to keep looking)

    Of course there are often multiple paths to success in a given role, but in many roles, there are certain patterns of thought or action that are so core that the best performers need to have personalities that support performing those activities constantly.

    Personally, I could be a gracious, congenial, cooperative party host for an evening or maybe a weekend – I can act that way by choice, but my personality would not support doing that every day and after a day or two, the ‘real me’ would want to reappear and I would be highly stressed to continue to work in a role that is not very natural for me.

    So – our personalities remain largely the same – we can tone them down or turn them up for short periods of time, but it is stressful and over the long haul, if we are in the work role, we will not be working at our full capacity.

  11. My mom always said if one continues to call a person a dog, sooner or later they will bark.

    What do I mean and how does this relate you may ask – well, people’s personalities can tend to change, or become more pervasive in certain environments.

    Individuals tend to adopt to their enviornments – this reminds me of Enron, where many of the employees characters changed and because of the enviornment; they got caught up in the delusion of the company and the hype they were fed. Many aspects can influence a change in personality -environment, peers, culture, events, people..

    So, it makes one really wonder how does personality tests rank when it depends on the culture of work, or the management, and staff.. Especially when cultures can change when for example certain events can take place that may cause unpredicted stressors

    So, is this an argument for personality tests or against them? – if a company is not individualistic, but more micro maybe then with a strong structure that may stand for reason, but what if the company promotes a more creative environment, and individuals are left to their own, be the best they can be.. isn’t then experience going to be the best predicator?

  12. I agree that talented people often have hidden skills or underdeveloped mantras that aid them in their upper cognitive displays

Leave a Comment

Your email address will not be published. Required fields are marked *