We’ve previously written about the use of the Wonderlic aptitude test on NFL draft prospects, pointing out that the popular press and NFL fans as a whole have often unfairly dismissed aptitude tests as irrelevant to future gridiron success. This seems to be based on jock stereotypes about the sport and on a misunderstanding of how tests, and predictive tools in general, work. Virtually every article about the Wonderlic test at the NFL draft mentions Dan Marino, who bombed the Wonderlic and went on to a Hall of Fame career, as evidence that the tests aren’t predictive of success in football. However, this type of anecdotal evidence clearly holds no weight when statistically determining whether or not a test works.
We made a few posts last year about the NFL and whether or not draft order is related to productivity. The core issue for us was a claim Malcolm Gladwell repeatedly asserted that the draft order of NFL quarterbacks (QBs) is unrelated to performance. Well, the issue was raised again over the Labor Day weekend and we were alerted to some more recent material we hadn't seen because to be honest we thought we were done with the whole thing. We found this very sensible WSJ blog from last December, but then we also found this CNBC blog from May of this year. Darren Rovell, the CNBC blogger, reproduced the following table from economist Dave Berri. It purports to show that performance of lower drafted QBs is similar to that of the top drafted QBs. Now to be fair, the table was used to argue that the cost-benefit of the lower picks might exceed that of the higher picks and that is entirely plausible. But Berri also uses a table like this to argue that draft order is not a good predictor of success.
I'll admit I'm in a curmudgeonly mood because I feel like I'm wasting time writing about something so obvious. But we've been implicated in a strange argument that erupted in the blogosphere last week, so I'm compelled to write a few words to clear our name. As we mentioned in our last post, a few days ago Steven Pinker reviewed Malcolm Gladwell's latest book and criticized him rather harshly for several shortcomings. Gladwell appears to have made things worse for himself in a letter to the editor of the NYT by defending a manifestly weak claim from one of his essays – the claim that NFL quarterback performance is unrelated to the order they were drafted out of college. The reason we're implicated is that Pinker identified an earlier blog post of ours as one of three sources he used to challenge Gladwell (yay us!). But Gladwell either misrepresented or misunderstood our post in his response, and admonishes Pinker by saying "we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google."
In my last post I compared a speech given by Malcolm Gladwell in the spring to the content of his new book Outliers, and wondered what had happened to the employee selection angle he had promised in the speech. Well, no sooner did my post go live than my New Yorker magazine showed up in my mailbox with the answer — this week's cover story is an article by Gladwell entitled "Most Likely to Succeed: How do we hire when we don't know who's right for the job?"
This Saturday is the NFL draft, which means that NFL scouts have spent the past months going over 40-yard dash times and college game tapes, and fans have debated which prospect would be the best fit for their team. It also means it's time for media and fans to recycle the usual punchlines about the folly of using an aptitude test like the Wonderlic on NFL prospects. Football, more than any other American team sport, is about physicality, and the idea that performance on an aptitude test could have much to do with success on the football field seems absurd. Skeptics point out that a low Wonderlic score didn't prevent Dan Marino from becoming one of the most prolific passers in history, or Vince Young from making the Pro Bowl in his rookie year. When Criteria works with customers to gather evidence for the validity of our employment tests at their organization, we sometimes hear similar anecdotes. I've often heard HR managers express concern that "one of our best performers did poorly on the test." (Criteria has an aptitude test, the CCAT, that is similar to the Wonderlic.) Such reactions are understandable, but the measure of a test's predictive validity can't be judged from one test score--the only meaningful way to measure a test's ability to predict productivity is to study the correlations between test scores and job performance across a broad sample of people. Based on this standard, the Wonderlic may be a better predictor of performance in the NFL than you might think.