How Good Are You at Evaluating People? Part Two

In Part One, of this series, you were asked to circle your favorite selection method and estimate its effectiveness. The following chart illustrates the average effectiveness of each selection method using actual research evidence gathered from controlled studies.

Method

%

Accuracy

%

Chance

1. Handwriting Analysis

None

100

2. Age

None

100

3. Amount of Education

None

100

1. Self Assessment

3

97

2. Projective Tests

3

97

3. Traditional Interviews

4

96

4. Grade Point Averages

4

96

5. Expert Recommendations

4

96

6. Personality Tests

4

Article Continues Below

96

7. Motivation

4

96

1. Reference Check

6

94

2. Biographical Data

9

91

3. Situational Interviews

9

91

4. Behavioral Event Interviews

10

90

1. Mental Ability Tests

25

75

2. Content Valid Simulations

64

36

*Adapted from a meta-analysis conducted by Hunter and Hunter, Psychological Bulletin, Vol. 96, 1984. Percentages have been rounded. ?% Chance? refers to the unexplained variance. You don?t have to have a Ph.D. to guess why most of these figures are so low. For example, among the first three methods, you would have to make a giant leap of faith to believe that handwriting, age, or amount of education could be directly related to job performance. In fact, two of the most incompetent managers I?ve ever known were among the most educated ? one was an attorney and the other had a Ph.D. in business. This is not to say that good managers are not intelligent or well educated, but it does say that having intelligence or education does not necessarily lead to being a good manager. The next seven selection methods provide slightly more job-related data. As you can see, self-assessment, projective tests, and traditional interviews are all self-presentations of how the person would like to be seen. They are easy to fake and provide little job data. Grade point averages tend to predict future grade point averages ? useful for admission to grad school, but not for job performance (unless the person has a history of failing grades). Expert recommendations, personality tests, motivation tests provide about the same percentage of usable data. One would expect these methods to be better predictors, but I believe the low research findings are caused by experts who tend to evaluate a very narrow band of technical knowledge, and because most personality and motivation tests are too generalized to be really useful. The four methods beginning with reference checks and ending with behavioral event interviews show an increase in useful data ? not great, but several times more accurate than the first 10 methods. If you look closely, these methods require more rigor, some training, are more focused on the elements of the job, and are harder to fake. Still, as good as they seem, they leave about 90% of job skill data to chance. The last two methods are the most powerful job-skill predictors. However, if you look at mental alertness tests through a demographic filter, they have a long history of adverse impact. Is this a problem? Not if you have done your homework. Like education, high performers tend to be more intelligent, but the opposite is not always true. The government does not require that you hire less intelligent people, but it does require you to show that the intelligence level required to pass your test is equivalent to the level required for the job. That is good for both applicants and the organization. (Imagine having an office full of Einstein?s whose only job is filing papers). Simulations are the most accurate job predictors. They are widely used to qualify and train aircraft pilots, tank crews, ship crews, transport navigators, military specialists, and a host of other occupations ? including some jobs. Simulations are particularly good for sales, management, customer service, team member, planning positions, and analytical jobs where specialized skills are associated with high job performance. So, where do you go from here? I?d suggest you leave the ineffective methods in the trash. I believe the only reason why they are not challenged more often is because the legal profession is not run like a business. Law offices and courts have simple organizational structures that place heavy emphasis on technical knowledge ? lawyers, a few paralegal assistants, and a support staff. But, just because the legal system is a poor example of ?the selection police? it does not mean your organization skips the consequences of poor selection. You get ?billed for your mistakes? everyday in turnover, low productivity, and wasted training dollars. You have seen the warning on TV, ?Do not attempt this at home!? You should not attempt to tackle the selection problem by yourself any more than you would attempt to remove your own appendix. Hire a professional. Choose someone who is an Industrial Psychologist with experience in selection and assessment. Verify that they are a member of the Society of Industrial and Organizational Psychologists. This person should be able to reduce your different job titles into a much smaller, manageable number of job families, to study each family to identify the critical measurable competencies, validate selection tools, and train you how to use them effectively. Expect to spend in the range of $15k to $25K for each job analysis and selection system ? less than the cost to place a prominent ad in one national newspaper. If you are on a tight budget, you don?t have to do all jobs at once, but there are economies of scale that come from doing several job analyses at the same time. Of course, you can just keep doing the same thing, enjoying the same turnover, and experiencing the same mixed productivity. Selection errors tend to be seen as the cost of doing business and probably don?t show up on any single department budget. On the other hand, you could think of a valid and reliable selection system as the difference between an arranged marriage where your partner is chosen for you, or a marriage where you have considerable say in the selection process. The choice and consequences are yours to enjoy.

Topics

Leave a Comment

Your email address will not be published. Required fields are marked *