Anatomy of a Test Vendor

A few weeks ago, I was asked about a certain test vendor. Being a good researcher, I visited their site and was absolutely astounded by the misinformation contained therein. “Wow,” I thought, “this could be the basis of an entire article about what NOT to believe about testing!” Here are some of the areas where I foresaw major problems for employers who chose to use their tests.


What they state: Their tests have passed a review to ensure a fully legal program. They can ensure users that they will “fully comply” with the new EEOC rules. A personal letter from an attorney states the test does not violate the law. The test complies with all EEOC regulations. What users should know: According to the Uniform Guidelines on Employee Selection Procedures, under no circumstances will the general reputation of a test or other selection procedures, its author, or its publisher, or casual reports of its validity, be accepted in lieu of evidence of validity. Specifically ruled out are:

  • Assumptions of validity based on a procedure’s name or descriptive labels
  • All forms of promotional literature
  • Data bearing on the frequency of a procedure’s usage
  • Testimonial statements and credentials of sellers, users, or consultants
  • Other non-empirical or anecdotal accounts of selection practices or selection outcomes

Validation Procedures

What they state: This test vendor states that “some people” believe in construct validity and others prefer criterion validity as the best approach. They further state that “common sense, good business sense, and concurrent validity on a local level” is the best approach. They do this by benchmarking the top one-third and bottom one-third of producers on a local level. What users should know: This is absolute nonsense. Here is a partial list why:

  • Validity is the process of determining whether a test legitimately predicts job performance, not common personality traits.
  • Validity is not a “crap shoot” to determine if something, anything, correlates with job performance. It is supposed to be a “validation.”
  • The Guidelines are clear. They suggest using either content (the nature of the job) or criterion (performance on the job) validity. They recommend against using construct (deep-seated psychological structures) validity.
  • “Concurrent” is not a type of validity. It is the name given to a real-time design using today’s employees to validate the test (i.e., evaluating current behavior as opposed to predicting future behavior).
  • How would you divide salespeople into top and bottom performance thirds? By new accounts? By account expansion? By existing accounts? By customer complaints or service? By bad debts? By repeat orders? Give me a break! Dividing employees based on “performance” is pure amateur thinking.
  • The Guidelines include almost 4,000 words explaining an acceptable process of validation. They include requirements for job analysis, fairness, representative samples, critical knowledge skills and abilities, statistical sampling, statistical procedures, and so forth.
  • There are more factors associated with job performance than personality. Even people with “good” personality traits can fail in the job because they have the wrong skills.
  • Use some common sense. Even if one could identify the “high performers,” it is likely they would have different personalities.
  • If the test was really valid, wouldn’t it be nice to know the candidate’s test was also significantly different from the low group? How about asking the number of individuals in the high group who actually met the high profile or the low profile?

Validity of the Advertised Test

What they state: In a national survey conducted on salespeople and commissions, “All of the above values show that there is at best a weak correlation between profile scores and commissions earned. Scores obtained on a national level would be unsatisfactory predictors of commissions earned.” What users should know: What kind of “spin statement” is that? Sales commissions are the number one indicator of sales performance, and the vendor publicly admits its test scores are an unsatisfactory predictor on a national level? It takes five key sales skills to make a good salesperson (i.e., relating, communication, questioning, presenting, and managing the relationship). If these cannot be measured nationally, doesn’t it seem like this test is looking at the wrong things?

Accuracy of the Test

What they state: Both managers and test-takers generally agree with the results of the test. What users should know: Well, duh! The test-taker just checked a dozen items stating he is determined. Should there be any surprise that his or her test scores were high in “determination”? Besides, are we measuring agreement with the test results or are we measuring future job performance?

Theory of the Test

Article Continues Below

What they state: The test was developed based on Hippocrates’ 2,400-year-old theory of temperaments. What users should know: Hippocrates also believed that black bile, yellow bile, phlegm, and blood were responsible for all physical and mental health. Furthermore, we have learned a few things about testing and hiring in the last 2,400 years.

Test Reports

What they state: The test uses 60 items to generate a two to fifteen page report. What users should know: Any reasonable person should suspect that 60 questions cannot possibly generate an accurate portrait of human personality. Any test promising that much data about a candidate should be examined for abundant filler, boilerplate information, and marketing general nonsense.

Moving Forward

Although this article focused on one single test site, I suggest the comments apply to many applications that ignore best test practices. There is a good reason why there are entire graduate-level courses devoted to doing a job analysis, developing decent tests, legal testing issues, statistics, and experimental design. This field is complex, and filled with opportunity to 1) hire the wrong people, 2) turn away the right ones, or 3) get your company sued. Want to learn more about how to build and use a test? Go to Here is an outline of its contents: Part I: Test Construction, Evaluation, and Documentation

  1. Validity
  2. Reliability and Errors of Measurement
  3. Test Development and Revision
  4. Scales, Norms, and Score Comparability
  5. Test Administration, Scoring, and Reporting
  6. Supporting Documentation for Tests

Part II: Fairness in Testing

  1. Fairness in Testing and Test Use
  2. The Rights and Responsibilities of Test Takers
  3. Testing Individuals of Diverse Linguistic Backgrounds
  4. Testing Individuals with Disabilities

Part III: Testing Applications

  1. The Responsibilities of Test Users
  2. Psychological Testing and Assessment
  3. Educational Testing and Assessment
  4. Testing in Employment and Credentialing
  5. Testing in Program Evaluation and Public Policy


7 Comments on “Anatomy of a Test Vendor

  1. As usual, Dr. Williams’ article contained a lot of useful information and some legitimate criticism, as well as quite a bit of misinformation. Rather than taking the time to critique this article point by point (which would take a significant amount of time), I can offer the following general commentary. First, if you are researching the validity and lawfulness of a test, evaluating the publisher’s website isn’t a very thorough approach. While some general insight regarding the technical and legal foundations of a test might be gleaned from such a cursory process, an appropriate approach would at least entail: 1. Requesting copies of the validation evidence supporting the test; 2. Requesting information regarding whether the test has typcially exhibited disparate impact, as well as information regarding any litigation involving the test or similar instruments, including the outcome of such litigation; 3. Requesting a copy of the test and user’s manual to evaulate whether the test raises privacy issues, whether it should be administered after a conditional offer of employment has been tendered and whether any test inquiries are unlawful; 4. Determining what type of support a publisher will provide in the event of legal challenge (e.g., free expert testimony) and 5. Having an employment attorney with strong expertise in testing review this documentation and information in relation to relevant laws, agency guidelines and documented job requirements.

    I trust these rather succinct comments provide a bit more insight on how to objectively and thoroughly evaluate whether a test is appropriate for use.

  2. Ouch! A lawyer’s perspective is always helpful when one wants to understand the legal details(even if they were not an intended part of the article).

    To clarify, my point was not to teach law, nor was it to debate how to read and interpret a test manual…it was to illustrate the foolishness of many test vendor claims and why readers should question them.

    Exactly which part of questioning foolish vendor claims do you disagree with?

  3. I wouldn’t buy a test from a company with such an unprofessional web site as that. Looks cheap!

    Dr. Williams, did you have a client mention they went with this test over one of yours, which was the reason for ‘attack’ mode?

    Again, you might win over some people if you had a good ‘offense’ instead of played ‘defense’ all of the time. You do know great salespeople win clients by promoting their product’s value, not by talking about how bad their competition is, don’t you?

  4. Just to clarify the implicit intent of my previous post regarding this article, I don’t think an employer considering the use of an assessment should spend much time on, much less make a decision on the basis of, representations found on a vendor’s website and within its marketing materials–whether these representations are apparently foolish, misguided or appropriate. As Dr. Williams has pointed out many times, the Uniform Guidelines on Employee Selection Procedures (‘Guidelines’) preclude consideration of such information (whether accurate or inaccurate) in
    documenting a test’s validity and defending its lawfulness. Hence, why
    not cut to the chase—ask the publisher
    for documentation that is truly useful in determining whether the test is technically sound and legally appropriate, which will also verify whether the publisher’s representations are accurate. Unlike vendor representations, the Guidelines (See Section 7) and the Department of Labor’s guide on assessments (See ‘Testing and Assessment: An Employer’s Guide to Good Practices’) recognize that such information is appropriate for documenting the technical foundations and lawfulness of a test. As an aside, I certainly don’t support misguided representations by publishers as indicated by my previous post’s statement that Dr. Williams had apparently levied some legitimate criticism within the article.

    I trust this information is helpful.

  5. Thanks for the advice…

    No, I did not lose a client to the company I wrote about…They were just an exceedingly good example of an exceedingly bad hiring test supported by documentation that made it look like a legitimate hiring test.

    I did not realize some folks interpreted my comments as an attack against competitition. It might seem strange, but I don’t look at the test I wrote about as a ‘competitor’… just a very glaring example of a test that may hurt applicants and drain money from organizations.

    I have a low ‘BS’ tolerance and it is hard not to call it like I see it. Sorry.

  6. The sentence that I liked the best in this article is: ‘Besides, are we measuring agreement with the test results or are we measuring future job performance?’

    I can’t tell you how many people want to argue about assessment results because they don’t want to hear what the assessment is telling them. They forget the whole reason for assessing anyone, especially in hiring, is to have a heads-up around behaviors that will both positively and (more importantly) adversly affect a candidate’s performance!


Leave a Comment

Your email address will not be published. Required fields are marked *