Posted by Josh Millet on Thu, Nov 19, 2009 @ 02:50 PM
We're going to file this one in the "we told you so" file! The other day the famous Harvard psychologist Stephen Pinker reviewed Malcolm Galdwell's newest essay collection in the New York Times, and it wasn't pretty. Pinker savages Gladwell, concluding:
"Unfortunately he wildly overstates his empirical case. It is simply
not true that a quarterback’s rank in the draft is uncorrelated with
his success in the pros, that cognitive skills don’t predict a
teacher’s effectiveness, that intelligence scores are poorly related to
job performance or (the major claim in “Outliers”) that above a minimum
I.Q. of 120, higher intelligence does not bring greater intellectual
achievements. The reasoning in “Outliers,” which consists of
cherry-picked anecdotes, post-hoc sophistry and false dichotomies, had
me gnawing on my Kindle."
Pinker's conclusions echo the arguments we made in this humble blog about a year ago. Read them again here and here. Gladwell is a great story-teller, and a gifted writer--but he should not be considered an authoritative voice on how we conduct social science or public policy, or for that matter employee testing.
Posted by Josh Millet on Thu, Dec 18, 2008 @ 12:24 PM
In my last post I compared a speech given by Malcolm Gladwell in the spring to the content of his new book Outliers, and wondered what had happened to the employee selection angle he had promised in the speech. Well, no sooner did my post go live than my New Yorker magazine showed up in my mailbox with the answer — this week's cover story is an article by Gladwell entitled "Most Likely to Succeed: How do we hire when we don't know who's right for the job?"
In the article the author describes the problems inherent in evaluating talent and predicting job performance, and cites three examples of jobs where he sees this as a problem: pro football quarterbacks, teachers, and financial analysts. I'm going to focus here just on the issue of predicting success for NFL quarterbacks.
Gladwell describes the challenges faced by NFL scouts who evaluate college quarterbacks, and relates the examples of some prominent "can't miss" prospects who became NFL busts. Gladwell is at his most comfortable spinning an anecdote about a single subject, and he structures this article around the story of Chase Daniel of Missouri. But somehow in trying to tell the story of how difficult it is for NFL teams to decide who to draft, Gladwell makes the ludicrous statement that the entire NFL selection process is fraught with error. He concludes that "there are certain jobs where almost nothing you can
learn about candidates before they start predicts how they'll do once
they're hired."
In fact, when one looks at the NFL's record of predicting quarterback success, Gladwell's conclusion is on very shaky ground. The collective opinion of an NFL player's prospects are reflected in the order in which players are drafted by NFL teams. It turns out that draft order is a very accurate predictor of subsequent statistical performance for quarterbacks.
To take the most recent decade as an example, when one looks at all the quarterbacks (67 in all) who were drafted by NFL teams from 2000 to 2004, and compares their overall draft position to their statistics in their first four years in the league, it is clear that on balance NFL teams are very accurate in predicting statistical success in the NFL. Organizational psychologists measure the predictive validity of an employee selection technique by quantifying the strength of the relationship between selection measure and job performance; the strength of the association is expressed as a correlation coefficient. For the whole group, the correlation between draft order and passing yardage is very strong (-.73 — the coefficient is negative because the higher a player is drafted, the lower their draft rank). For those concerned that a measure of total productivity such as passing yardage is somewhat correlated with opportunity, we can consider passer efficiency, as measured by QB rating. Only 51 of the 67 quarterbacks drafted attempted a pass in the NFL, a necessary requirement for calcuclating a QB rating: for this group there was a -.34 correlation between draft position and QB rating. This is still a strong association, and shows a clear, statistically significant correlation between draft order and future statistical success in the NFL.
Like the fans of the teams that drafted them, Gladwell has let the Ryan Leafs (a high draft choice that flopped) and the Tom Bradys (a low draft choice who became a superstar) of the world influence his thinking. These are outliers, a concept with which Gladwell should be familiar given the title of his latest book. (If you take Brady out of the mix the correlations strengthen considerably!) It turns out, in fact, that on average the NFL draft process is highly accurate at predicting QB success, and the draft is based entirely on things that Gladwell dismisses as useless--college performance, scouting, performance in the NFL combine.
If Gladwell had considered any quantitative measures at all relating to the efficacy of the draft he'd have no basis for his conclusion that "a prediction, in a field where prediction is not possible, is nothing but a prejudice." Gladwell, we fear, gets swept up in his own story telling, and in the process badly miscontrues the alleged "quarterback problem."
Gladwell's approach reminds me of a challenge we face every day in discussing our pre-employment testing services with customers and prospective clients. When evaluating a selection instrument, there's a strong tendency to take an anecdotal approach and fixate on the outlier for whom the selection instrument was not an accurate predictor. For example, when a sales manager administers one of our tests to a few dozen existing employees, we'll often hear about the one high performer who didn't fare well on the test. Anecdotes are powerful, and it is sometimes difficult to persuade the manager to focus on how well the assessment predicts performance across the whole group. No selection measure is perfect, and we must be careful when evaluating the efficacy of our selection processes not to follow Gladwell's lead and let anecdotal evidence trump more rigorous analysis.
Posted by Josh Millet on Fri, Dec 12, 2008 @ 12:12 PM
Recently on a plane the guy beside me was reading the same book I
was – Malcolm Gladwell's Outliers. My fellow passenger didn't think
this was remarkable as the airport bookstores had huge displays.
Gladwell has become somewhat of a household name for his skill at
popularizing social science through collecting compelling anecdotes. Blink and The Tipping Point were entertaining enough to read, and
that's why the guy beside me had made an impulse purchase.
I had been more proactive in getting my hands on the book. We here
at Criteria had actually been eagerly waiting for Gladwell's book ever
since May, when we stumbled across a truly odd speech
he gave at a New Yorker conference. Gladwell's speech, which was
explicitly delivered as a sneak preview of the book, covered what he
called the "mismatch" problem. His thesis was that the way employers
evaluate prospective employees — including the practice of
pre-employment testing — is at a complete "mismatch" with what is
required. As evidence he offered three loosely linked examples – sports
combines where amateur athletes are evaluated before a draft;
certification requirements for teachers; and the University of
Michigan's affirmative action program for law school admissions.
Gladwell opens with anecdotes from the National Hockey League's's
pre-draft combine, and then goes on to discuss the NBA and NFL drafts.
He relates the finding that the aptitude test given to the NFL
quarterbacks has no correlation with their performance. As usual, his
evidence for this contention is entirely anecdotal. In a blog post
in the spring we described evidence showing that the test may be
predictive of QB success, and it doesn't help Gladwell's case that he
mocks the fact that Eli Manning and Tony Romo scored well on the test,
while Vince Young and David Garrard did not. Even back in May 2008 he
should have understood that his alleged exceptions weren't exactly
disproving the rule. By the end of the opening example, Gladwell has
shown that he probably doesn't know all that much about sports, and has
launched a puzzling, and difficult to support, argument. We were eager
to see him make his point in print, but this line of thought didn't
make it into the book.
In his speech, Gladwell next moved on to discussing teachers. We
can't disagree with him that good teachers are important, or that it
might be a good idea to broaden the pool from which new teachers are
selected. But we were a bit confused by his argument about hiring
standards. Teacher quality, he tells us, is much more predictive of
student achievement than classroom size, and so it is worth investing
in. But according to Gladwell, there are no criteria to predict whether
someone will be a successful teacher. This is quite a jump, and again
one is left with only colorful anecdotes for what is a very sweeping
point with broad social significance.
Finally, Gladwell points out that even though the University of
Michigan Law School had, in the past, given extra consideration to
minority applicants and made concessions on testing standards, there
was no evidence of differences in "success" years after graduation.
Gladwell does not address that the admissions tests were not designed
to predict success after graduation, but rather performance in the
first year of law school. Nor does he address that there are
statistical problems with evaluating a selection tool on a sample that
was selected in the first place using that tool. But most important,
Gladwell seems to be arguing that based on the experiences of the
University of Michigan, law schools should abandon altogether their use
of test scores to select applicants.
Well, somewhere between his May speech and the November publication
of Outliers Gladwell must have realized he had his own mismatch
problem. His evidence didn't match his thesis. He must have changed the
direction of his book significantly, because Outliers is barely
relevant to employee selection. Instead, it's a motley collection of
examples arguing that exceptionally successful people are not entirely
self-made, and that their ascent is due also to extraordinary good
fortune with regard to the opportunities they were presented with.
(This is hardly a shocker of a thesis, and is right up there with
Blink's bold contention that first impressions are often correct,
except when they are not.) But where's the employee selection angle?
Where his speech proclaimed that it was "time to shut down the
combines", his book only discusses the mildly interesting fact that the
birthdays of professional athletes tend to cluster near critical
cut-off dates for selection into elite programs. Where he promised to
show how the process of hiring and training pilots is completely at
odds with what is required, he only ends up discussing how cultural
factors can have a tragic influence on the dynamics between pilots in a
cockpit. Rather than discuss the process of hiring teachers, he
describes one successful charter school, and the demands it makes of
its students. He also speculates on
cultural and linguistic factors that might correlate with math
perseverance.
We're left to wonder why the change in focus from the speech to the
book... could it be Gladwell realized that after all there is
substantial evidence for the effectiveness of aptitude testing as a
predictor of job success? Or did he just decide that "what determines
exceptional success?" is a more interesting question (and easier to
write about) than "how can we hire better?"