Matchmaker, Matchmaker, Make Me a Matching Job Tool

Companies sprout up in bunches. In tough times, for example, lots of companies started up to help job candidates with their resumes. In an improving job market, as social media recruiting expands, a number of companies are working on the employee-referral social media connection.

Now, a growing cluster of new vendors is in the matching-screening business, a field already occupied by companies like JobFox as well as the one called StrictlyTalent I mentioned previously. These vendors are trying to go beyond the job board by serving employers only the candidates who could be a fit, not the 300 who email a resume in.

Just a sample of the new tools includes:

Roundpegg. It has a number of products. One’s a culture survey you take to see if your culture matches what you say about your company. Another is a 1:1 interaction guide to work individually with an employee so they can use their strengths and weaknesses to their advantage when dealing with colleagues. Another is a fit guide measuring values, personality, and communication style for current and prospective employees; with that, you can see “what makes your teams tick” and compare a job candidate to that culture.

Roundpegg’s emphasis is on whether candidates have the characteristics that define success at the organization. “Every company is different,” says co-founder Brent Daily, “you can’t just hire your competitor’s top salesperson and expect them to thrive right out of the gate.” Daily is a founder of Yahoo Green, a project that helped teach him how to go from 0 to 60 very quickly, he says, which has helped with his Roundpegg work. He says that with many products, “employers assesses employees and the results don’t see the light of day again,” but with his product “as people move in and out, you can update performance ratings and markers of success.”

I asked Daily if his tool is anything like HRVision. “The process is not too dissimilar to their offering,” he says, “but A) we provide a living, breathing profile of your team; B) we fit a candidate to multiple levels — company, team, and the direct manager; and C) we’re exclusively focused on knowledge workers, HRVision on high-turnover positions.”

Roundpegg started in May 2009, and shares a spiffy office with Jive, which is also a customer. Roundpegg’s largest customer is the Dish Network, which uses the culture gap analysis. Roundpegg is meeting with Dish this week to talk about the company using the Fit Guide.

Matchpoint Careers. With this site, which uses some assessments from SHL, candidates are given four questionnaires: verbal reasoning, work environment, personality, and numerical reasoning, questionnaires which generate a profile of a candidate.

Meanwhile, employers are asked to describe their company and the job; for example, how much responsibility the new employee will have. Employers are given a “Preliminary Job Profile of the competencies required for high performance in that job, as developed over years of research on thousands of similar jobs.” Employers can modify that profile.

Now about those “years of research” I just mentioned: founder Paul Basile says that decades of studies by the government and private sector show what makes a person effective in a given job. “Why are high performers high performers and others not?” Basile asks rhetorically. The answer, he says, is competencies. That’s what’s used on the back end to create the profiles that match future boss to future employee. The database evolves over time, as more workers are added to the research, shaping the needed competencies. Experience, which is heavily emphasized on resumes, is less important to Matchpoint; what your current employees are like is also less important.

Anyhow, candidates who fit the bill are given information about the job and can join a “shortlist” if they choose. Employers are sent the shortlist of candidates who are a match and who chose to be included on the list. Candidates are ranked according to how strong a match there is.

Over many years, quite a few of them in Europe, working for companies like Boston Consulting Group, DBM, and Computer Sciences Corp., Basile realized there needs to be more of a science to recruiting. He started talking to people — hiring managers and others — about his ideas a year and a half ago and got mostly thumbs-up. He moved from Paris to New York last summer, where the company is based, and has just recently launched the Matchpoint tool. Says Basile: “I proceeded with it, we built it, and it works.”

Clearfit. Here, candidates take a “15-minute personality assessment test” that generates a career report about them. They use this to decide what career’s best for them — it doesn’t get sent to employers.

Employers fill out a short survey about the job title, location of a job, and so on. They can pick the job profile that best fits their job: for example, for a sales manager job, there may be multiple profiles for them to choose from. These profiles are built mainly in-house. If they want, they can customize the job profile based on their top performers, something Clearfit calls a “JobFingerprint.”

Article Continues Below

Now, the matching part. Employers get a list of applicants, how well they’ll fit in to the company, and some interview questions to ask them (see graphic). Suzy may be a “strong fit” and Johnny may be a “weak fit”; someone else may show up with a “distortion” notice, indicating they may not have answered the questions so honestly.

Clearfit has been around a while and boasts 1,000 businesses as clients, many small businesses, but some as big as McDonald’s, which has used Clearfit in managerial and franchisee hiring. Co-founder Ben Baldwin says he has worked to simplify the tool, make it easy to use, and easy and free to test it out. “We thought we were in the assessment business, but we are in the easy-to-use business,” he says. “We did 10 things before, now we do one.”

Baldwin says that tools like his are necessary because after decades of innovation in the recruiting/human resources world, there’s still a lot of guesswork done when hiring, and “the results like turnover measures and accuracy of the actual product — hiring — haven’t improved.”

Clearfit, which has 12 employees, is based on Toronto. Most all of its customers are in the U.S.

Wendell Williams is generally pretty skeptical about these sorts of companies. He says in an email: “While matching sounds intuitively attractive, there is scant evidence showing scores on interest tests actually predict job achievement. The reasons are clear: interest tests are self-reported descriptions (i.e., subject to personal opinion); they seldom evaluate the skills to accomplish them (KSA’s); organizational cultures often change dramatically when new senior management takes command; culture can change when moving from one boss to another; jobs with the same title often are entirely different; individual job expectations can change substantially over time; and, personal preferences change as we age. Don’t even ask me about the negative effects of hiring a group of personality clones on productivity!”

Charles Handler says that if you take a “super analytical” point of view he agrees what Williams is saying. But, he says, as long as companies don’t rely on these screens for a score to make a hiring decision, but rather as one piece of a larger hiring puzzle, they can be useful. For one, Handler says, they can help companies find candidates who may not have been interested in or known about a company. And, he says, they can indeed help narrow down the right person for a given job, so long as the assessment questions are done the right way, a job-specific way.

SuccessFactors’ Steven Hunt, who wrote Hiring Success: The Art and Science of Staffing Assessment and Employee Selection, says without commenting on the specific companies listed above that this sort of thing has been tried before — unsuccessfully. “Job matching is inherently difficult, since jobs change so much over time and across companies.”

“But,” he says, “I think the concept in general is worth exploring, especially given the changing nature of the job market. There are often jobs out there that people might like (especially if they are willing to move to new cities), but they just haven’t heard of them.”

Topics

51 Comments on “Matchmaker, Matchmaker, Make Me a Matching Job Tool

  1. Interesting article. It seems that the most reliable way to find out if a candidate is right for the job (beyond the resume, cover letter, and interview) is to see how he/she behaves in the work setting.

  2. Thanks for the article, Todd. To reply to Wendell’s note, in particular; I completely agree that interest tests most frequently fall short on validly predicting success.

    Our assessment, however, is not an interest test — it’s a normative psychometric assessment. We allow companies to build custom job profiles (based on their top performers) that shift as the job culture shifts. These types of normative assessments are very effective and have been around forever, but are typically too expensive and time-consuming for a small business or new user to use.

    Our customers demand that we let them try before they buy, so we’ve had to make something that’s actually quite complex (normative psychometric assessment) appear very simple 🙂

  3. Nice overview – how many of these services are using either “black box” or highly configurable matching services behind the scenes, such as Burning Glass, Daxtra or Actonomy? Some are better than others, and some are only as good as the rules and configuration that gets put into them..

    Also, how many have something more unique that’s been custom built? I know LinkedIn’s matching technology is mostly custom built for example, and seems to be targeting ads quite well at the moment using the matching service..
    It is a good way to approach recruitment though I’d agree – the flood of unqualified applicants is something every recruiter has to find ways of dealing with, especially in today’s market.

  4. I tend to agree that matching won’t work alone. Candidates are impatient and filling out a too detailed questionnaire will result in abandonment. I also don’t think that matching is worthwhile in weeding out potential candidates – that the proverbial “black hole” still exists, even with matching that promises to give candidates that are more suited for a position.

    Other factors that may be at issue are those candidates who can change up their questionnaire to suit a particular job that he/she is a few points off from being a better match. Being able to manipulate the profile to the job fit is easy and doesn’t give a more qualified candidate – rather, it gives a candidate who knows how the matching works and allows them to be attractive to a company.

    Good article, and great responses.

  5. I believe any worthwhile job-oriented selection system is better than none; however, let’s think about what’s behind the curtain in a job-match system. We’ll use a truck driver example…

    Do you think all drivers in the profile have identical profiles or is it just an average number that fit’s no one in particular?

    Do you suppose all the drivers in the profile were equally skilled, or are you assuming everyone in the sample is a rock-solid performer?

    If the group really includes legitimate high performers, aren’t you narrowing your applicant pool by eliminating fully skilled people (who could become high performers?

    Are high performers always rock-solid corporate citizens or do they often cut corners?

    Are you both using the same definition of high performance?

    Do low performers have a significantly different profile from high performers or are we dealing with averages again?

    Which factors have the greatest impact on performnace or are they treated with equal weight?

    Does the profile have dozens of scales that overlap (marketing mojo) or is it succinct and robust.

    Can you really trust group-level data to make accurate assumptions about an individual in the group (we call this pre-judging)?

    If the EEOC or OFCCP comes knocking, can you proove your job is essentially the same as the generic profile?

    I’m sure you can think of more questions…this is just a starter toward making better informed decisions.

  6. Real job-matching, as established by the Uniform Guidelines on Employee Selection Procedures, starts with a job analysis to determine “what it takes” to perform in each particular job. Then, the process uses valid measures of job-related constructs derived from the job analysis to gauge the extent to which individuals “have what it takes”. Finally, the process compares the results of the job analysis to individual assessment results in order to establish the strength of the match. All three things must be done well in order for the results to be highly actionable.

    Most job postings do not come with a job analysis. Indeed, it’s rather rare for employers to even have up-to-date job analyses when hiring for positions. Strike one. Most of the individual input is self-reported and not based on valid, psychometric measurement of job-related constructs derived from the job analysis. Strike two. And even when the starting point includes a job analysis and the individual assessment includes only valid, job-related measures, the “matching” process must somehow build a bridge between the job analysis and the assessment results; the bridging or mapping process inherently compromises the predictive validity of the job match. Strike three.

    The best approach to real job matching uses a single scientifically designed and validated normative instrument both to analyze the job and to assess individuals with respect to the same set of job-related constructs so that there is no “loose linkage” when subsequently gauging the job match. Moreover, the choice of constructs should include a sufficient number of cognitive, behavioral and interest dimensions and the assessment should be capable of detecting efforts to “fake good” or “polish” the results.

    Any selection process has two types of errors – e.g. presenting a match that does not exist and rejecting a match that does exist. The farther the “matching” process strays from that described in paragraph 3, above, the more sizeable Type I and Type II errors become.

    Those offering job-match “capability” ought to be willing to calibrate their outcomes against the outcomes achieved by those who follow the Uniform Guidelines on Employee Selection Procedures. Those who use ersatz job matching services should think through their compliance exposures, should the matching process result in adverse impact. If so, the employer (not the service provider) would be obliged to show the job-related validity of this “job-matching selection procedure” in order to avoid exposure to regulatory enforcement action for discrimination.

    Richard Melrose
    r.melrose@vision21.us

  7. Interesting. “Real job match” as defined by Richard sounsd as though it would point to a higher chance of genuine match, however of course requires higher input from both recruiter and candidate.

    A more traditional match process as used in many job boards and agencies is all too often focused on job-skill, desired locations, experience levels and salary bands.. Some matching engines such as Burning Glass attempt to focus on natural language keyword analysis before using this in a matching system that, among other things, weights experience against typical job transitions (i.e. a move from Job A to Job B is more likely than Job B to Job C). All of these systems are best placed to work alongside CV parsers to enable quick population of a skills and experience based “profile” that is used in the matching. (Therefore this type of system doesn’t HAVE to have huge forms for candidates Michaela)

    The matching that Richard talks about, and some of the types of forms I’ve seen in some of the sites listed above) focus as much on soft skills, personal characteristics and motivations as on skills and experience.

    Which actually gives the better results, and are there enough automated systems out there that balance both types of data? I’m unsure of the answer to that one.

    This secondary type of system (presumably incorporating various psychometric profiling questionnaires) would definitely mean more data entry from a candidate (you’re not going to be able to parse any of that data automatically from a CV!). If the exact nature of these forms is completely job specific (as Richard’s example sounds like it is likely to be), then this can’t work in a generic sign-up form for a site, it can only really work as part of an application workflow.

    The point here though is that this may not be the VERY BEST indicator achievable for matching to a given job. However if it means you get sent 20 likely candidates that you can then test further, rather than 300 candidates the majority of whom are definitely not applicable, then that’s got to be a good thing. The only major downside is the potential for rigging the system, or the matching being too generic and non-job specific and actually ending up meaning some good candidates get rejected. Candidates can easily lie or exaggerate on CVs and application forms though and a whole host of factors already lead to good candidate getting rejected, so I can’t see that this is much of a step backwards..

  8. Richard is not talking “theoretically”. He is talking about best practice, what the EEOC and OFCCP recommend, what leading organizations actually do to maintain the quality of their work force, and what employers will be asked to produce should they be audited.

    Yes, it is more work. Yes, it takes more input from both the organization and the candidate. Yes, thoroughly understanding both job requirements and accurately measuring candidate skills leads to an employee who fits the organization, the manager’s personality, the work group culture, and (best of all) arrives fully skilled for the job.

    I’ve seen it work for many organizations. It’s like choosing a mate: better front-end screening delivers better quality. Organization-wise, it means fewer people to produce the work, less labor problems, less turnover, less training, and less time to productivity.

    I’d be anxious to hear anyone explain which part of best practices they disagree with…other than taking more work for the recruiter.

  9. Gareth wrote: “If the exact nature of these forms is completely job specific (as Richard’s example sounds like it is likely to be), then this can’t work in a generic sign-up form for a site, it can only really work as part of an application workflow.

    Actually, the assessment which the individual takes can be generic (universal), and the results of that assessment can, nevertheless, provide an information rich view of each individual. That’s because the individual’s answers determine the results.

    The same assessment that collects individual results can also characterize jobs in terms of the range of scale scores most associated with high job performance, as well as the relative importance of particular scales. Those determinations can come from the scale scores exhibited by top performers in the job, as well as well as job analysis survey results from individuals who are very knowledgeable about the job and its performance requirements.

    This compatible method of describing people and jobs can dramatically improve the performance of talent markets – i.e. getting “the right people in the right seats”, as Jim Collins urged.

    The low cost of high quality online assessments is such that employers who pay ‘retail’ prices to support their own standalone selection procedures can reasonably expect 10x short-term ROI. Spend a buck and get ten back! So, with a variety of plausible business models, there’s more than enough room for a more broadly based, high quality, job-matching system to quickly create far more value per assessment than let’s say the fee of a professional résumé writer; moreover, the assessment results (unlike the résumé) do not need to be fine tuned for different jobs or rewritten every year or two.

    An applicant would invest far less time in completing an assessment than in writing a couple of good cover letters. And unlike the single-use cover letter the quality assessment has enduring value, because high test-retest reliability is a prerequisite of high validity.

    That said, I would not expect change to come quickly. Résumés, job descriptions and unstructured interviews have been the target of subject matter experts’ frequent attacks, for some time. Even so, most employers still regard them as mainstays of their hiring process.

    Richard Melrose
    r.melrose@vision21.us

  10. Apologies – I fully appreciate that it is best practice. My use of quotes around the term Real Job Match was only to differentiate it from those other systems in use, not to imply it’s not real or of less use at all.
    I’m sure it is indeed far better and far more likely to work (as I stated). My point is that it requires more job specific input from the both recruiter and candidate than current systems.

    I do not know enough specifics about the online assessments you are discussing to know whether they can indeed be generically applied as you say, but it would certainly be good if you could do this – in which case for a candidate it becomes a “complete once” process, with the workload on the recruiter to set their scoring correctly for the specific job/scheme. As i think we’re slowly moving towards an “online professional profile” for jobseekers, which organically grows with them and pulls in blogs, testimonials and possibly video (LinkedIn being the best known but far from perfect example of this), perhaps a set of standardized assessment results of the sort you mentioned would be a perfect addition to this profile. (Standards are great from my points of view!). If these were indeed truly standardized, then an “apply with my LinkedIn profile” process would even allow your assessment to be automatically pulled in and scored.

    I like your definition of this as being like choosing a mate, as slightly similar tests do seem to be used on some dating sites for matching.

  11. Generic evaluations can be quite useful…providing they are performance-validated and face-valid. Let’s take problem solving tests for example. We’ll arbitrarily divide them into basic, applied, and abstract.

    One would not give any kind of problem-solving test to a candidate applying for a job where problem solving is unrelated to job performance. An engineering or financial analysis test might be appropriate for engineers or analysts. An abstract problem solving test might be appropriate for top leadership positions where everything depends on discovering abstract relationships.

    Score-setting is another issue.In some jobs, a 50% score might be acceptable. In another case, it might take 90%-plus…it all depends on how strongly scores affect job performance (and how performance is defined). This is one of the objectives of a validity study.

    Finally, the test items should at least resemble the job. This lends credibility (i.e., face validity) to the test.

    So generic tests can work, but one size does not always fit all. They must be carefully chosen…just like every other hiring tool.

  12. Previous commentators have touched on a very key point: The science of matching, like all sciences, must be correctly applied.

    Matching absolutely must start with rigorous job analysis, not just a few quick questions posed to an employer. Different jobs draw on different strengths. The more precisely we can define the job, the better idea we have of who can best fill it. It makes sense to draw on the decades of analysis that have been done here. And the best of those analyses reflect the range of potential performance-prediction combinations for a job. Clearly there is not one size that fits all.

    But jobs change over time, don’t they? Sure. But the proven predictors of performance change less. Think of a Fortran programmer long ago vs a PHP programmer today. The specific technical skills have changed (eg programming language, the hardware and interfaces) but ongoing research has not shown great changes in the factors that differentiate the top performers from the pack.

    For the people side of the match, we have to look for traits that are the same as (or very, very close to) the real performance predictors of a job. These factors are not simple or transparent. We need lots of deep, varied, reliable and relevant information from candidates. And that means more work for the candidate. Clearly, as CEO of Matchpoint Careers, I am banking on candidates seeing the payoff, both in the psychometric feedback they get immediately and in the great-fit jobs they will get in time. Isn’t getting the job of your dreams worth an hour or two of your time?

  13. Great discussion! A good number of our staffing company clients have found better candidates and employees using our integrated SuccessCheck module, a personality and cognitive testing tool based on many of the best-practices principles discussed here.

    Here is a link to one case study — http://www.vcgsoftware.com/article-EmploymentGroup-SuccessCheck.asp

    The payoff is a win-win-win for the employee, the employer and the staffing/recruiting company.

  14. While these are exciting and intriguing technologies, I think we’re overlooking the basics. There’s only two major areas of recruiting- Individual Contributors and Managers, and each of these requires difference processes for sourcing and selecting…and I’m not sure any of the service above generate substantial value propositions in this paradigm.

    When recruiting for individual contributors, there’s really two sub-sections that exist: specialized and non-specialized. If you’re looking to fill specialized individual contributor roles, keyword matching is a very basic and effective process for identifying good candidates. A resume either has the terms SAP, PLM, CAD, and Engineering Change Management (ECM), or it doesn’t. If it does, it’s worth following up with.

    When recruiting for non-specialized individual contributors, there’s a little more of the herding mentality, with a major focus on decreasing turnover. For these types of roles, work history (length of service with each employer) is really critical. Again, a basic and effective way to screen candidates.

    As we look deeper into recruiting for manager roles (and I use this term broadly to cover supervisors, managers, directors, etc- anyone who oversees other staff or areas of business), we find more vagueness and variation in sourcing processes. However, at this level, I have to fall back to the mantra of first WHO, then WHAT. FIRST who, THEN what. FIRST WHO, THEN WHAT.

    That is, when looking to bring in leaders at any level of an organization, it’s necessary to make sure you’re getting the right fit into your management culture. Everyone here knows the number one reason why good people leave your organization is a bad manager.

    Bringing this back to the article here, I would offer that it’s more important for these positions that you keep your requirements broad and more generalized. The next great leader of your organization may not have a specific software competency (which can be learned). Defining fit into a job description is nearly impossible…almost as impossible as defining personality and morals and vision within a resume. Sourcing candidates for management roles requires strong referrals and some level of insider knowledge of an individuals style and previous experience. These are characteristics that cannot be mapped between job postings and online resumes or applications.

    While I love new technologies, I’ve never been an early-adopter, and routinely chosen to see which resources truly prove value before buying in. As I consider these types of services, I just can’t quite understand where the “substantial” value proposition is, or how they “significantly” enhance current recruiting practices.

  15. What technology, in the form of the services discussed in Todd’s article does when used well, is to allow us to take the understanding gained by years of experience and the latest developments in profiling technology and make these more accessible to candidates and employers. I would not argue that this is perfect, but it offers recruiters a cost-effective solution that is leagues better than most recruitment practice.

    Whilst it can be argued that any volume system will not be as good as one tailored to an employer’s specific needs, this argument is always be based on retrospective evidence – until the data is there validity is assumed, it cannot be taken for granted. Tailoring a system to an employer’s specific needs is no guarantee that it will be better than a comparable off-the-shelf system, though it may increase the odds of this.

    What systems such as those offered by Matchpoint Careers (I cannot claim to be impartial in this debate) do is to make industry-leading technology available to candidates and employers. Good systems do not assume ‘one size fits all’ and support employers in tailoring the profiles to their requirements. They also do not preclude the use of ‘experts’ to further enhance recruitment – in fact this is something that Matchpoint Careers advocates. The potential benefits of this have already been identified by contributors to this discussion, so I won’t go over them again. But think, what would be the effect on GDP if this technology increased the validity of every hire by just 0.01…?

  16. ISTM that the more carefully defined and standardized a position is, the more applicable and effective any type a matching technology might be. If only most most positions were like this!

    Cheers,

    Keith

  17. Angus, I appreciate your perspective and articulation of the positioning and value of these services. As I mentioned, I am not averse to technology, but tend not to be an early adopter, as I wait to see if the assumed validity turns into realized validity, and therefore realized value. As such, I am open to changing my opinions based on new information.

    I would be very interested and appreciative of your thoughts on two other points. First, in your response you mention “industry-leading technology”, a term that seems to be used as ubiquitously in technology forums as “last chance sale” is used by car dealers. How do you define and/or support claims of being “industry leading”, when, as you mention, there is little data to provide proof of concept. (In complete honesty, Angus, I am truly very interested in your perspective on this, and hope that you don’t find this question to be offensive or inflammatory.)

    Second, how would you define the problems that these technologies provide solutions for. I think if I could understand what was actually driving a need for these services, I may feel differently about them.

    Thanks again for your interaction and insights!

    jason

  18. There is a great deal of good insight in the comments thread. Just one point – Martin, group dynamics are indeed a part of the puzzle and absolutely not ignored in the Matchpoint Careers concentration on proven science. They are included in every part of assembling the job profile.

  19. Paul, I would be impressed beyond measure if your solutions involved instrument or simulation based assessment of existing team members as part of the construction of the job profile, and then used various algorithms to match people with groups to predict overall group performance(understanding that such nested multi-variable problems would require large data sets and long experience to establish validity).

    I think that assesment vendors who start to walk that road are going to eventually gain extreme competitive advantage over those who are merely attempting to assess individuals and jobs. It may be a decade away, but my gut says it’s going to happen.

  20. I do not see individual assessment as presenting an inherent conflict with group assessment or job design or leadership assessment or market research or anything else that provides actionable information to improve the quality of business decisions and the value of ensuing enterprise results. They are all complementary from my perspective.

    In addition to job-matching assessments that resolve the individual, the job and the boss, my colleagues and I routinely deliver team assessments capable of analyzing the strengths, weaknesses and balance of teams or prospective teams, as well as examining team leadership choices and providing leadership insights, based on twelve factors that underwrite team performance. The foundation for this capability is an individual assessment of all prospective team members. From there, analysis and reporting features enable an unlimited number of “what-if” team-building exercises, and no additional charge.

    We deliver analyses of employee/manager fit – regarded by most as an extremely important, albeit two-person, group.

    We deliver 360 degree multi-rater feedback surveys that can accommodate Self (manager at the center of the 360) plus up to 3 Bosses, up to 12 Direct Reports, up to 12 Peers and Up to 12 Others (e.g. customers, suppliers, whatever). Those surveys establish organizational perceptions, with respect to 8 universal management competencies and 18 associated skill sets, covering both personal and interpersonal dimensions. Multiple 360s can be readily rolled up into larger group (organizational) analyses.

    Beyond work groups, we deliver comprehensive, enterprise-level, research-based, employee engagement surveys that can be sliced and diced by Client-identified groups (function, department, business unit, country, etc.) and which cover the full range of the employment deal (34 elements), employee segmentation (6 segments), as well as workplace attitudes, experiences and activities, among other very highly actionable measures.

    Undoubtedly, there are many other things to measure about individuals and groups and we do some of those, as well.

    To me, these are an impressive set of capabilities, if for no other reason the size of the ROI they earn for clients. But, I do not expect to find Martin “impressed beyond measure”, nevertheless.

    Richard Melrose
    r.melrose@vision21.us

  21. How much (overall/per person) do effective assessment tools cost? Could a medium size HW store that wants to hire a good bookkeeper afford one? What about a small marketing firm that wants to hire a receptionist? A self-funded 12-person startup that needs to hire a SWE?

    Cheers,

    Keith

  22. Keith,

    Yes, yes and yes.

    Depending on the assessment, list prices for one-up quantities range from $30 to $250. Discounts for purchasers of over 300 assessment typically amount to 50% or more.

    So what’s the return on those investments? Employers should expect 10x, short-term ROI and additional dividends throughout the employee lifecycle.

    The turnover cost of the smallest hiring mistake at an entry-level, full-time hourly job generally exceeds $5,000. The assessments necessary to systematically avoid such mistakes would likely run $300 to $400 per hire and would accomplish much more than just reducing turnover.

    Bottom line: high-performance assessments aren’t just affordable, they’re downright profitable!

    Richard Melrose
    r.melrose@vision21.us

  23. Richard, I actually am quite easy to impress: I believe virtually everything I read, and I think that is what makes me more of a selective human than someone who doesn’t believe anything.*

    Little doubt that process you describe does yield terrific ROI, and I also know that these concepts are in the toolkits of high-end management consulting firms. I also have little doubt that they are somewhat expensive and one-off for each engagement- thats where I am looking to be impressed-when those methods and techniques can be effective in bits and bytes at a fraction of the current expense.

    *attrib David St. Hubbins

  24. Jason, thanks for your comments and the reminder that I need to be careful in the language I use! Still, I did use it, and believe it, and here’s why.

    Matchpoint Careers uses SHL assessments. These are normally recognised as being pretty good, and we have the latest versions and some other pretty new developments from them. The science behind these is good and they have proven validity (not just internal reports but also from peer-reviewed journals). We also use an extensive database of jobs for which competency profiles have been developed by trained consultants (not ‘do-it-yourself’ approaches). There is also user-friendly technology to help employers refine these profiles to find a best-fit for their job. Finally, we have proven algorithms that put together the job profiles and candidate profiles.

    To come back to your first point, each of the aspects of described above is well established and proven. What is new and unique is the way that we put them all together, giving employers access to a large database of pre-profiled candidates.

    Your second point, about what is droving the need for technologies such as ours, is well highlighted by Keith’s questions and Richard’s responses. We know use of appropriate assessments usually provides a good ROI, but in Keith’s example, what is likely to have happened to get the stage of using assessments? Typically the job would have been advertised (that’s a cost), applications or resumes received and screened (that’s a cost and usually pretty unscientific), then assessments might be used on a selected groups of applicants (another cost).

    The recruitment pipeline tends to put the ‘quick and dirty’ selection methods up front to screen down the applicant pool, but these are not the best predictors of subsequent performance. The result: you inevitable miss some people with great potential in the early sifts.

    Now, let’s turn things around. Instead of selecting people to be profiled, let’s profile everyone. Use the tools that are the best predictors right up-front and select on the basis of how well the profiles match the job. All scientific, quick and you get to profile 1000s of applicants for a similar cost to profiling a dozen. Ultimately, what’s driving our enterprise is the belief that recruitment can be better, both for employers and candidates, and the knowledge that we now have the technology to make this a reality for everyone.

    Jason, if you’re interested in trying it out, contact Paul Basile (paul.basile@matchpointcareers.com) and he will be happy to show you round the system.

    PS: Thanks for the questions; not at all offensive and it’s a great opportunity to put our case.

  25. Replying to Keith’s and Richard’s posts, a good normative psychometric assessment typically starts at $150, then reduces based on volume. However, one needs to add the set-up cost to that (i.e. cost of customizing the job around your top/mid/bottom employees) and that can present a much larger cost, often $5,000 to $20,000 if they’re not very large studies.

    This is one of the reasons why we price ClearFit at $349 for unlimited use for 30 days … after your free trial. We’ve tried to make it really easy to “try before you buy,” then cost very little following that.

  26. I have no idea in this discussion who is a test expert and who is a marketer. But I cannot help notice no one has answered the questions that any responsible user should be asking:

    Do you think all job-holders in the profile have identical scores or is it just an average number that fit’s no one in particular?

    Do you suppose all the job-holders in the profile are equally skilled, or is performance all over the board?

    If the group really includes legitimate high performers, please explain why a job-match system would not eliminate fully skilled people who could become high performers?

    If the profile does, indeed, include high performers, are they rock-solid corporate citizens who do the right things? (In my experience, top performers tend to cut corners).

    Are both the vendor and the buyer using the same definition of “performance”?

    Do low performers have a significantly different profile from high performers?

    Which factors have the greatest impact on specific performance …which aspect… how much weight are they given?

    Does the profile have dozens of scales that overlap (marketing mojo) or is it robust enough for one profile to be substantially different from other jobs?

    Can you really trust group averages to make accurate assumptions about whether an individual fits the job?

    If the EEOC or OFCCP comes knocking, can you prove your job is essentially the same as a generic profile?

    As I said, these questions represent what any well-informed buyer would want to know…I’d like to see someone answer these questions.

  27. Martin, These assessments are neither “expensive” nor “one-off for each engagement”. See my reply to Keith Halperin, in this thread, above.

    If these solutions were twice as costly, they would still delight customers with 5x short-term ROI, plus plenty of additional employee lifecycle dividends. Very few companies have higher return investments available to them.

    These assessments are available to every employer in the world, 24/7/365 from any Internet connection and assessment takers require no administration.

    There’s no hardware or software to buy, learn or maintain; no contracts to sign; no consulting experts to pay; and no special training required (brief, plainly written User Guides suffice).

    The web-based, user-friendly employer’s dashboard can have any business (whether Fortune 25 or Mom & Pop) up-and-running (i.e. scheduling, delivering, scoring, analyzing, reporting and archiving their own assessments), within 24 hours.

    One part-time administrative person can readily handle the scheduling and reporting demands for a 1,000 employee company – i.e. typically less than 2 days per month (10% duty), for a rather active user – e.g. 20 post-hire assessments per month and 60 pre-hire assessments per month.

    Moreover, as I have mentioned, previously, these assessments lend themselves to small-scale, yet still very profitable, experiential learning. One personal smart-phone two-year contract generally represents a bigger dollar commitment, with a much fuzzier ROI, especially on a phone/plan upgrade.

    Additional Comments:

    These assessments are not new; they’ve been out there for two decades and on the Internet for a dozen years and their heritage in some cases is as old as dirt. Oh and, lest I forget to mention, they are scientifically designed to facilitate compliance with the Uniform Guidelines on Employee Selection Procedures – i.e. using valid, job related measures to make best practice selections. NO assessment maker can take that burden off of the employer, but the right assessment design choices and mission specifications, together with current, large-scale validation studies can go a long way toward making it easier for compliant employers to demonstrate the validity and job-relatedness of their selection procedures.

    By the way, for those who are not yet concerned about compliance, note: the biggest pure assessment company reportedly generates about $200 million in annual revenue. That’s just less than half of the $404 million in monetary benefits collected from employers in fiscal 2010 by the EEOC, after filing 99,922 discrimination charges during the period.

  28. Richard, forgive me but I have a mental disconnect between

    “my colleagues and I routinely deliver team assessments capable of analyzing the strengths, weaknesses and balance of teams or prospective teams, as well as examining team leadership choices and providing leadership insights, based on twelve factors that underwrite team performance”

    and

    “These assessments are available to every employer in the world, 24/7/365 from any Internet connection and assessment takers require no administration”

  29. Are both statements factual at the same time Richard ?

    Do you and your colleagues routinely deliver team assessments via Internet- bits and bytes without associated professional services ?

  30. Wendell,

    I will gladly address at least some of your pertinent questions, tomorrow.

    I must admit, however, that even after 35 years of marriage, I still have trouble answering some two-part questions like “Do you want to go with me, or stay home?”.

    Other confessions: I am “test expert”. I am also a “marketer”. I advise business leaders on building enterprise health and value (a teacher for the C-suite). I am many other things, too … just as you are.

    Let’s have some fun; perhaps, we can teach some things of great value.

    Best regards,

    Richard Melrose

    P.S. As much as I enjoy teaching, I value learning, even more.

  31. I’m just an auto-didact with a fast computer and voice dictating equipment- FWIW I really only know a lot about Sailing, The Third Reich, Aerospace, and Recruiting Software…..

  32. Martin,

    Nothing at all wrong with being self-taught. Many of the best of us are, in large measure. I cannot recall seeing “hard knocks 101” on any university syllabus.

    But tell me, was your most recent post a retreat or just a side step?

    In 1970-1972, in the defense electronics arena, I collaborated on the design and development of a real add, real multiply, 100Mhz throughput, parallel pipeline processor that could withstand 9Gs in an airborne platform, weigh less than 200 lbs, and fit in a box 1.7 feet on a side, while dissipating less than 1,500 Watts. The then, best available commercial computing platform from IBM (System 360) took up a reasonably large-sized room, and could not come close to meeting the performance (throughput) specifications, let alone the size, weight, acceleration and heat dissipation requirements.

    I have sailed (for over fifty years) and I am particularly fond of boats with un-stayed carbon fiber masts.

    I am well acquainted with the third (and fourth) Reichs, having seriously studied WW II for decades and in the late 90’s served as Geschäftsführer of a medium-sized, highly regarded, German engineered products manufacturing company, with more than 550 employees.

    I grew up on Long Island (NY) in the 50’s and 60’s; my father was, as VP and General Counsel, the last employee of Republic Aviation Corporation (RAC), of P47, F85, F105 fame. My uncle led the missile and space systems divisions for RAC. I have stayed abreast of global aerospace developments ever since. During the Cold War, I held Top Secret Defense (DoD) and Intelligence (SI/SAO) clearances and knew the Soviet state-of-the-art (threat), as well as anyone.

    As you might imagine from my posts, I am keenly interested in (and at least reasonably well informed about) enhancing recruiting and selection processes through science and technology.

    And your point is … ? I thought we were talking about job-matching assessments.

    Richard Melrose

  33. Ahhh Richard you are a tough one. I sensed your prior post as a pause to align us, so I responded with information to reveal the contours of my own style and interests. Not a sidestep, more of a frontal turn. Now that I read your follow-up, I’m sure that were we to meet in person, we could pass hours of happy conversation.

    I own a few boats, my favorite of all time being a Laser2 hull with *a freestanding, rotating carbon mast* and a Hoyt boom. It was marketed (very small run) as the Expedition 14.5- mine is modified with a split luff mainsail that opens wing on wing for downwind work and reefs to any size I want. Up on a plane in 15kts is about as happy as I ever get.

    Each summer I take it to Good Harbor Bay, MI (where my lovely wife’s family has a beach home) and we sail the lakes and bays of Leelanau County, which is as as perfect a spot as can be imagined in August. I really enjoy the Hiller Museum (speaking of the end of Republic Aviation) at San Carlos Field, and I was at the Wright Brothers shrine on Dec 3, 2003 (alone as it turned out) for the 100th anniversary of powered flight. My flying skills, such as they were (no more piston flying…or motorcycles…with a family) were formed in my late father’s Commander 114, which met a bad end one evening in zero/zero with a WOT into a house – with a family friend (alone) at the controls. My wedding music was entirely taken from the soundtrack of Apollo 13.

    My ethnic heritage and historic sense demanded a full study of the Hitler years and I’m hyper-alert to the echoes of what remains the central fact/mystery/shadow of the modern world, even though it was probably not even the third or fourth rank among historic horror shows. I like to say efff Godwin at least once a week, as a matter of fact, I did so earlier today.

    My first experience computing was in a large lab at Cleveland State in the mid ‘70’s, when my dad would go down on Saturday mornings and buy me a coke while he ran reel to reel tapes thru 4K of RAM on a time-share. Later he bought a Diablo 3200 for the price of a house, with floppys the size of turkey platters and a massive 32K of RAM. The IBM PC came out and depreciated that house to salvage value in 18 months, but not before the old man made a few bucks running data processing jobs up and down Chagrin Blvd. (Cleveland, OH) and my brother (thank the stars) got his start programming.

    So now that it’s established that we are very much birds of a feather, please do educate me about your services 😉

  34. I’d like to address Wendell’s questions from yesterday. But before I do, I want to say that as good as this particular job-matching assessment is, I am not suggesting that it be used in isolation, but rather as one of several procedures that support a thoughtful and demonstrably capable selection process. There are other valid assessments, structured interviews, simulations, work samples, etc. available as tools to contribute to a body of highly-actionable information for selection.

    With reference to cutting lumber, the old saw (pun intended) is measure twice, cut once. With people and jobs, measurements are more complicated; so, measure in as many ways as are valid, job-related and viable; the differential between the value of a good hire and the cost of a bad hire is always much bigger than the investment in underwriting success.

    This assessment does provide structured interview questions, as part of its standard reporting. Those questions are based on departures from the range of scale scores exhibited by top performers in the job of interest.

    The assessment also enables a “Managerial Fit” report which describes the potential working relationship between applicant and prospective boss, along seven business-related constructs.

    This assessment covers cognitive, behavior and interest domains, which simplistically answer the questions: Can the person do the job? How will they go about it? and Will they want to do (remain interested in doing) the job?

    This assessment also provides a distortion factor designed to detect “faking good” or “polishing”.

    Finally, for anyone interested in diving more deeply into the details, there is both an 85-page Technical Manual and a 12-page Executive Summary thereof, which cover the assessment’s theories, development and psychometric characteristics.

    OK, here we go. I have put Wendell’s questions in upper case, in an effort to make the Q&A easier to follow. I also took the liberty of breaking up a couple of the two-part questions.

    I HAVE NO IDEA IN THIS DISCUSSION WHO IS A TEST EXPERT AND WHO IS A MARKETER. BUT I CANNOT HELP NOTICE NO ONE HAS ANSWERED THE QUESTIONS THAT ANY RESPONSIBLE USER SHOULD BE ASKING.

    DO YOU THINK ALL JOB-HOLDERS IN THE PROFILE HAVE IDENTICAL SCORES? No, a concurrent study of true top performers will exhibit a range of scale scores; that range may be narrower or wider from one scale to another. Among top performers there may be “outliers” as well as “clusters” of scores, on any particular scale.

    IS IT JUST AN AVERAGE NUMBER THAT FIT’S NO ONE IN PARTICULAR? No, it’s a range, within which most top performers scores fell, or very nearly fell.

    DO YOU SUPPOSE ALL THE JOB-HOLDERS IN THE PROFILE ARE EQUALLY SKILLED? No, top performers will exhibit ranges of knowledge, skills, abilities, behaviors and interests.

    IS PERFORMANCE ALL OVER THE BOARD? If top performers have been selected based upon objective, high-priority (important) measures that define performance in the job, then dispersion of scale scores tends to indicate that a particular construct is less important than one where the scale scores cluster tightly. In that case, for that scale score, the profile will have a “target” (scale score range) that more people from the working population will be able to hit.

    IF THE GROUP REALLY INCLUDES LEGITIMATE HIGH PERFORMERS, PLEASE EXPLAIN WHY A JOB-MATCH SYSTEM WOULD NOT ELIMINATE FULLY SKILLED PEOPLE WHO COULD BECOME HIGH PERFORMERS? Because, the assessment does not measure performance, rather it measures constructs directly related to one’s ability to perform, one’s approach to performance and one’s interest alignments. Hiring managers can see and record performance (e.g. sales). They need help figuring out who can deliver performance.

    IF THE PROFILE DOES, INDEED, INCLUDE HIGH PERFORMERS, ARE THEY ROCK-SOLID CORPORATE CITIZENS WHO DO THE RIGHT THINGS? (IN MY EXPERIENCE, TOP PERFORMERS TEND TO CUT CORNERS). Corner cutting or worse tendencies are assessable, but they are also often more observable than other performance-related criteria.

    ARE BOTH THE VENDOR AND THE BUYER USING THE SAME DEFINITION OF “PERFORMANCE”? Yes, however, it often takes some time to (a) agree upon objective, measurable and meaningful performance criteria for a job and (b) to assure that those are indeed the criteria used to identify the top performers.

    DO LOW PERFORMERS HAVE A SIGNIFICANTLY DIFFERENT PROFILE FROM HIGH PERFORMERS? Yes, or you do not have a performance profile.

    DOES THE PROFILE HAVE DOZENS OF SCALES THAT OVERLAP (MARKETING MOJO) OR IS IT ROBUST ENOUGH FOR ONE PROFILE TO BE SUBSTANTIALLY DIFFERENT FROM OTHER JOBS? This particular “profile” involves twenty scales from three substantially independent domains. Within each domain, value creation (i.e. fostering better selection decisions) depends, to a large extent, on the independence of the chosen constructs.

    WHICH FACTORS HAVE THE GREATEST IMPACT ON SPECIFIC PERFORMANCE …WHICH ASPECT… HOW MUCH WEIGHT ARE THEY GIVEN? The weight given to each scale depends upon the extent to which particular measures on that scale distinguish between top performers and contrast performers. If all top performers score 3, 4 and 5 on a particular scale and very few contrast performers do, that scale rises in influence. If, on the other hand, on a particular scale top and contrast performers both score in pretty much the same range, in accordance with working population proportions, then that scale declines in importance (weight). All of that weighting is automatic as part of the assessment software. Even with weighting, however, the impact of any one out of the twenty scales is relatively low. It is the combination of all twenty scales that presents information rich “pictures” of both individuals and jobs.

    CAN YOU REALLY TRUST GROUP AVERAGES TO MAKE ACCURATE ASSUMPTIONS ABOUT WHETHER AN INDIVIDUAL FITS THE JOB? No, and we don’t. Averages are often points of zero probability – i.e. a two and a ten are not the same as two sixes. Each assessment takers scale scores are “normed” to the working population. So, if Johnny scores an 8 on the STEN scale for Verbal Reasoning, we know, with a certain confidence factor, that Johnny is in the top 16% of scorers on that scale; we know, with a much higher confidence factor that Johnny is in the top 25% of Verbal Reasoning scorers, drawn at random from the working population.

    IF THE EEOC OR OFCCP COMES KNOCKING, CAN YOU PROVE YOUR JOB IS ESSENTIALLY THE SAME AS A GENERIC PROFILE? The profile developed and used with this assessment is NOT generic; it characterizes THE particular job for which it is used. Within the same bank, for example, the profile for a loan officer in New York City will almost certainly differ from the profile of a loan officer in Topeka. Likewise, the profiles for loan officers for ABC Bank and XYZ Bank, may differ, even when both jobs are in NYC. As to the EEOC, the employer needs to demonstrate (and, with the recommended process, readily can) that, in the presence of adverse impact, the selection procedures used constituted valid, job-related measures, consistent with business necessity, in accordance with the Guidelines, and that those procedures were applied uniformly to all applicants (candidates). Try making that statement for unstructured interviews and the follow-up “group grope” exercises that serve as the back end of most companies’ hiring processes. The front-end 15-second resume reads and application reviews cannot meet the standard of the Guidelines, either.

    AS I SAID, THESE QUESTIONS REPRESENT WHAT ANY WELL-INFORMED BUYER WOULD WANT TO KNOW…I’D LIKE TO SEE SOMEONE ANSWER THESE QUESTIONS.

  35. Thank you for the reply…Sorry for the dual-item format, I’m trying to put each question in context.

    You mentioned “top performers” in your reply…What (or whose) performance criteria was used to select these people for the profile?

    You mentioned using STEN scores. Are the STEN’s based on a general population or against the target job and what is the margin of error?

    In looking at job match, are we to assume job success depends on how many group-level dimensions the candidate matches?

    Most test experts know group-level data cannot be used to make individual judgements (i.e., blue people tend to be taller that green people…you are green, therefore, you must be short). Yet, isn’t matching individuals to group-level data what the job-matching program you describe does?

    It seems improbable that twenty dimensions of performance would be independent. I would really appreciate receiving a copy of the technical manual you offered. You can send it to my attention at ww1080@comcast.net.

    Demonstrating validity transportability requires a job analysis showing the client’s position is essentially the same as the validated one…how does job matching account for this?

    If using a test of mental ability, how does job-matching account for disparate impact?

    Hiring tests are a high stakes event. How do you ensure a specific candidate did not have outside assistance?

    Like any hiring and placement tool, IMHO readers should know the inherent pros-and cons of a job-match system before buying into it.

  36. @Angus:
    “Now, let’s turn things around. Instead of selecting people to be profiled, let’s profile everyone. Use the tools that are the best predictors right up-front and select on the basis of how well the profiles match the job. All scientific, quick and you get to profile 1000s of applicants for a similar cost to profiling a dozen. Ultimately, what’s driving our enterprise is the belief that recruitment can be better, both for employers and candidates, and the knowledge that we now have the technology to make this a reality for everyone.”

    I’m not sure this is what you meant, but ISTM this could be a potential alternative/adjunct to resume/experienced-based hiring. In this scenario, instead of sending in a resume to a company, recruiter, or a job board, someone could fill out a “Universal Assessment Test,” and be hired based on what/she can do well, as opposed to what they have done.

    1) I am not sure that such a “Universal Assessment Test” is possible, feasible/affordable, or can be made without bias.

    2) There is an incredible amount invested in the hiring status quo, who would vigorously oppose such changes, if only because it would make many powerful people look quite foolish in retrospect.

    3) Consequently, I would expect the US officially using the metric system, comprehensive peace in the Mideast, and my hair to grow back before such a major change is made.

    🙁

    Keith

  37. Wendell,

    We are probably going to bore the audience to tears or worse.

    I have sent you the Profile XT Technical Manual which should give you what you need to answer many of your own questions.

    YOU MENTIONED “TOP PERFORMERS” IN YOUR REPLY…WHAT (OR WHOSE) PERFORMANCE CRITERIA WAS USED TO SELECT THESE PEOPLE FOR THE PROFILE? The criteria used to select these people are those short-list objective, measurable performance criteria that the individual employer (e.g. hiring manager and hiring manager’s reporting chain) deems most important in gauging the job performance of individuals in that job. As I mentioned, above, the performance criteria definition process often involves “engineering agreement” before settling in on what performance really means in that job context. Some sales organizations conclude that only “Revenues” matter others prefer “Throughput” (revenues less purely variable expenses); others want to include additional criteria that address performance in related activities including: prospecting, customer service, administrative, mix management, solution development, project management, expense management, team play and others. If one or more of those activities generate considerable value, other than through sales, there might be merit in considering demonstrated (objectively measurable) capability in that activity as part of the top performer selection criteria. At the end of the day, however, most employers value Throughput from sales people a whole lot more than anything that those activities might produce. Said differently, most sales organizations would take the slightly weird loner who generated over $1 million in throughput every year over the “model citizen” who averaged $500K per year and performed all those other tasks with aplomb.

    YOU MENTIONED USING STEN SCORES. ARE THE STEN’S BASED ON A GENERAL POPULATION OR AGAINST THE TARGET JOB AND WHAT IS THE MARGIN OF ERROR? The STEN scores are individual (top-performer or candidate) scores which are normed to the general population of working people in the US (N=639,231).

    IN LOOKING AT JOB MATCH, ARE WE TO ASSUME JOB SUCCESS DEPENDS ON HOW MANY GROUP-LEVEL DIMENSIONS THE CANDIDATE MATCHES? No.

    MOST TEST EXPERTS KNOW GROUP-LEVEL DATA CANNOT BE USED TO MAKE INDIVIDUAL JUDGEMENTS (I.E., BLUE PEOPLE TEND TO BE TALLER THAT GREEN PEOPLE…YOU ARE GREEN, THEREFORE, YOU MUST BE SHORT). YET, ISN’T MATCHING INDIVIDUALS TO GROUP-LEVEL DATA WHAT THE JOB-MATCHING PROGRAM YOU DESCRIBE DOES? No.

    IT SEEMS IMPROBABLE THAT TWENTY DIMENSIONS OF PERFORMANCE WOULD BE INDEPENDENT. I WOULD REALLY APPRECIATE RECEIVING A COPY OF THE TECHNICAL MANUAL YOU OFFERED. YOU CAN SEND IT TO MY ATTENTION AT WW1080@COMCAST.NET. Naturally, the twenty constructs exhibit some interdependence. You can look a the inter-correlations on page
    3-11 of the Technical Manual. 50% of the inter-correlations are below 0.3 and 25% are below 0.2

    DEMONSTRATING VALIDITY TRANSPORTABILITY REQUIRES A JOB ANALYSIS SHOWING THE CLIENT’S POSITION IS ESSENTIALLY THE SAME AS THE VALIDATED ONE…HOW DOES JOB MATCHING ACCOUNT FOR THIS? The assessment does not rely on a previously validated performance model. It builds “job-specific” performance models based on top performer scale score ranges and Job Analysis Surveys for the particular job of interest. The assessment can provide a starting point for performance model refinement, when no (too few) top performers are available to facilitate a concurrent study. That “canned” performance model mirrors the Department of Labor O*Net Content Model Job Analysis for the “closest” position(s) out of nearly 1,000 on file.

    IF USING A TEST OF MENTAL ABILITY, HOW DOES JOB-MATCHING ACCOUNT FOR DISPARATE IMPACT? The Appendices of the Technical Manual show that the average Overall Job Match Percentages (OJMP) vary between a low of 67.519% (age 65+) and a high of 72.324% (Hispanics), where the OJMP ranges between 25% and 95% and has an aggregate mean of 70.036% (N=907,326. Thus, ethnicity, gender and age play negligible roles in the OJMP outcomes.

    HIRING TESTS ARE A HIGH STAKES EVENT. HOW DO YOU ENSURE A SPECIFIC CANDIDATE DID NOT HAVE OUTSIDE ASSISTANCE? When the assessment taker doesn’t know how the behavioral and interest questions roll up into the scale scores, there’s no incentive or practical way to have some one help. If an assessment taker answers with a “theory” about what the “right” answers are, he/she will be tripped up by the distortion measure. Assessment takers can “cheat” in the cognitive sections. But of the three domains, cognitive is the most observable to those conducting a structured interview. It’s real hard to fake being smart when your not and few applicants would think about faking stupid to improve their chances of landing a job. An applicant, who exhibits an eighth grade vocabulary and doesn’t understand the interviewer’s questions probably did not earn the 9 in Verbal Skill (top 5%) and the 8 in Verbal Reasoning (top 16%). If suspicions arise, it makes sense to re-administer the assessment in an employer-controlled environment. The assessment has high test/re-test reliability. You will get the right answer.

    LIKE ANY HIRING AND PLACEMENT TOOL, IMHO READERS SHOULD KNOW THE INHERENT PROS-AND CONS OF A JOB-MATCH SYSTEM BEFORE BUYING INTO IT. Amen.

    Richard Melrose
    r.melrose@vision21.us

  38. Keith,

    Now you’ve got it!

    That is, and has been, precisely the opportunity. And it’s not just about one assessment. It should be several, including scores on game-like, on-line RJP simulations. Actionable talent information trumps résumé. Process gets systematized and streamlined. Time to hire plummets. Quality of hire soars. Dinosaurs become extinct.

    The way to overturn the status quo is to take a page or two out of Seth Godin’s “ideavirus” concepts (the most downloaded eBook ever; Seth knows what he is talking about). TALENT can actually drive the transformation, with everyone else having to fall in line.

    Who knows how long the transformation might take? In 1866, the US Congress authorized the use of the metric system and supplied each state with a set of standard metric weights and measures. We’re still waiting for that one to go viral.

    I think we have a much better chance with assessments trumping résumés, if great employers and top talent lead the charge.

    Richard Melrose

  39. Hi Wendell, here are my responses to your questions. I see that Richard has also responded, but I have not read his responses yet as I didn’t want to be unduly influenced by what I had just read (even though I did pick up the capitalization tip from a quick scan!).

    DO YOU THINK ALL JOB-HOLDERS IN THE PROFILE HAVE IDENTICAL SCORES OR IS IT JUST AN AVERAGE NUMBER THAT FITS NO ONE IN PARTICULAR?

    Job holders have a range of scores, as is desirable to encourage diversity. What we have found is that for some roles the level of importance for certain competencies are very consistent across the same jobs in different organizations whereas others are more variable. This can be because 1) these competencies truly vary between organizations or 2) they are far less crucial to job performance.

    DO YOU SUPPOSE ALL THE JOB-HOLDERS IN THE PROFILE ARE EQUALLY SKILLED, OR IS PERFORMANCE ALL OVER THE BOARD?

    This assumes that profiles come from the job holders, but not all approaches use this methodology. Assessing job holders who are identified as very capable (I appreciate capability is easier to pin down for some roles than others) is one way of determining profiles, but in-depth assessment by people trained to conduct job analyses is one we (Matchpoint Careers) use. An advantage of not relaying too heavily on current job holders is that allows us also to look to the future demands of the job rather than just being chained to the past.

    IF THE GROUP REALLY INCLUDES LEGITIMATE HIGH PERFORMERS, PLEASE EXPLAIN WHY A JOB-MATCH SYSTEM WOULD NOT ELIMINATE FULLY SKILLED PEOPLE WHO COULD BECOME HIGH PERFORMERS?

    This is far less likely with a system that ‘selects blind’ such as Matchpoint Careers, as you are not eliminating candidates on superficial factors that have nothing to do with performance but everything to do with bias and prejudice.

    IF THE PROFILE DOES, INDEED, INCLUDE HIGH PERFORMERS, ARE THEY ROCK-SOLID CORPORATE CITIZENS WHO DO THE RIGHT THINGS? (IN MY EXPERIENCE, TOP PERFORMERS TEND TO CUT CORNERS).

    In my experience this is something that needs to be considered in the context of the specific organization. One of my previous companies conducted research which found that good managers were not detail focused; those that were could not pick their heads up from the detail to take a wider view. We worked with the company to help them understand this and ensure that there were people in place to ensure the detail was attended to when necessary. Cutting corners can be okay as long as it’s understood and there are checks and ways to manage this in place – it needs a whole team approach.

    ARE BOTH THE VENDOR AND THE BUYER USING THE SAME DEFINITION OF “PERFORMANCE”?

    The definition of performance has to come from the buyer, though typically needs to be refined and focused into something more specific and measurable. Ultimately, after a bit of discussion, yes.

    DO LOW PERFORMERS HAVE A SIGNIFICANTLY DIFFERENT PROFILE FROM HIGH PERFORMERS?

    This is integral to the competency profiling exercise. Those factors that don’t differentiate between the two, or only differentiate poorly, are given much lower weighting the selection. If the tools you are using don’t show a difference, you’re looking at the wrong things. To me, this is what profiling is all about.

    WHICH FACTORS HAVE THE GREATEST IMPACT ON SPECIFIC PERFORMANCE …WHICH ASPECT… HOW MUCH WEIGHT ARE THEY GIVEN?

    This is very much down to the individual job (which is why we use profiles developed for specific roles) and what are the measures of success agreed with the client. The greater the demonstrable link between factors and performance, the greater the weighting given to them.

    DOES THE PROFILE HAVE DOZENS OF SCALES THAT OVERLAP (MARKETING MOJO) OR IS IT ROBUST ENOUGH FOR ONE PROFILE TO BE SUBSTANTIALLY DIFFERENT FROM OTHER JOBS?

    I would not claim that all factors are statistically or even conceptually distinct from each other, but they are certainly sufficiently pure to reveal a diverse range of profiles between different jobs. In the methodology we use, this is also partly down to the expertise of those profiling the jobs.

    CAN YOU REALLY TRUST GROUP AVERAGES TO MAKE ACCURATE ASSUMPTIONS ABOUT WHETHER AN INDIVIDUAL FITS THE JOB?

    No, not completely, and that’s why it is important not to look just at averages but the level of agreement on the importance of different factors. The higher the agreement, the more confidence and weight can be placed on the factor for selection.

    IF THE EEOC OR OFCCP COMES KNOCKING, CAN YOU PROVE YOUR JOB IS ESSENTIALLY THE SAME AS A GENERIC PROFILE?

    From Matchpoint Careers’ perspective, we would not want to try to do this. Generic profiles form a starting point for job profiling, but these are modifiable by employers for their specific organization

  40. Hi Keith,

    I know Richard has already replied but I want to as well.

    1) I agree. Not one test but there are a number of pieces that fit together to get us close. We all know the consistent findings for general mental ability (Schmidt and Hunter, etc), the big 5 offers a unifying approach to many personality models and competency frameworks, though less researched from an academic perspective, tend to have greater commonality that difference.

    Even these are far from perfect, and I’m not convinced all the gaps are going to be filled by new assessments. More likely they will be filed by better ways of using them and of understanding the complexities of the interactions between the individual, their colleagues and the work environment.

    2) Yes, and I don’t expect change to happen over night. But I don’t think it’s about making people look foolish. It’s about doing the best we can with the tools we have at the time. Technology is a still a relatively new tool and it allows us to do things we would never have thought of only a generation ago. They’ll be early adopters and later adopters, but I do believe it’s here to stay. Only 20 years ago virtual relationships were limited to a pen-pal you’d never met, but look at where we are now. It does not mean we were wrong to only have relationships with people we actually met!

    3) Being from the UK, I could not possibly comment on the first two and I have absolutely no understanding of alopecia.

  41. Thanks, Angus. I think that makes sense- not one revolutionary technique or test but a number of less-significant techniques/tests which together amount to a significant improvement.

    As far as making people look foolish, I agree that’s not the intent, but anything that punctures the balloon of pomposity will have that effect. (If I firmly stand by something in defiance of the facts and people see that, I look pretty foolish, e.g. “Saddam Hussein has weapons of mass destruction and is behind 9/11.) As I frequently point out, I believe that recruiting (and much else) has a thin shell of rationality and pragmatism, and a substantial and domninant core of greed, arrogance, fear, and ignorance/incompetence, not that those who are driven by such motives believe they are. That’s one of the main things that amuse me about recruiting.

    Regarding not commenting on things about which you have no understanding: Angus, if everyone did that on ERE, there’d be hardly anything here at all!

    RAH-AH-HA-HAH!!

    Keith “Scottish in First Name and Spending Habits” Halperin

  42. There are powerful ideas and certainly people behind them in this comments stream who I would love to meet one day. Fortunately, Angus and I work together so I know him. I loved reading your backgrounds, Martin and Richard. And Keith and of course the ever-questioning Wendell…

    Aerospace. I have undergraduate and graduate degrees in that and have worked for two contractors for the NASA shuttle. I don’t know anything about sailing (except how wings/sails work, a little) and while the Third Reich has never earned a deep study from me, I can claim to have known a contrasting extreme, I suppose – the Soviet Union, from several trips there and employment at a partly-Soviet owned research institute. And I can claim to know Europe where I lived and worked for, I don’t know, 25 years or so in four different countries.

    I read a moderate number of articles, comments and blogs, but never has one provided so much learning as this one. With such quality of thinking, there WILL be a revolution in recruiting.

Leave a Comment

Your email address will not be published. Required fields are marked *