Posted by Josh Millet on Fri, Dec 12, 2008 @ 12:12 PM
Recently on a plane the guy beside me was reading the same book I
was – Malcolm Gladwell's Outliers. My fellow passenger didn't think
this was remarkable as the airport bookstores had huge displays.
Gladwell has become somewhat of a household name for his skill at
popularizing social science through collecting compelling anecdotes. Blink and The Tipping Point were entertaining enough to read, and
that's why the guy beside me had made an impulse purchase.
I had been more proactive in getting my hands on the book. We here
at Criteria had actually been eagerly waiting for Gladwell's book ever
since May, when we stumbled across a truly odd speech
he gave at a New Yorker conference. Gladwell's speech, which was
explicitly delivered as a sneak preview of the book, covered what he
called the "mismatch" problem. His thesis was that the way employers
evaluate prospective employees — including the practice of
pre-employment testing — is at a complete "mismatch" with what is
required. As evidence he offered three loosely linked examples – sports
combines where amateur athletes are evaluated before a draft;
certification requirements for teachers; and the University of
Michigan's affirmative action program for law school admissions.
Gladwell opens with anecdotes from the National Hockey League's's
pre-draft combine, and then goes on to discuss the NBA and NFL drafts.
He relates the finding that the aptitude test given to the NFL
quarterbacks has no correlation with their performance. As usual, his
evidence for this contention is entirely anecdotal. In a blog post
in the spring we described evidence showing that the test may be
predictive of QB success, and it doesn't help Gladwell's case that he
mocks the fact that Eli Manning and Tony Romo scored well on the test,
while Vince Young and David Garrard did not. Even back in May 2008 he
should have understood that his alleged exceptions weren't exactly
disproving the rule. By the end of the opening example, Gladwell has
shown that he probably doesn't know all that much about sports, and has
launched a puzzling, and difficult to support, argument. We were eager
to see him make his point in print, but this line of thought didn't
make it into the book.
In his speech, Gladwell next moved on to discussing teachers. We
can't disagree with him that good teachers are important, or that it
might be a good idea to broaden the pool from which new teachers are
selected. But we were a bit confused by his argument about hiring
standards. Teacher quality, he tells us, is much more predictive of
student achievement than classroom size, and so it is worth investing
in. But according to Gladwell, there are no criteria to predict whether
someone will be a successful teacher. This is quite a jump, and again
one is left with only colorful anecdotes for what is a very sweeping
point with broad social significance.
Finally, Gladwell points out that even though the University of
Michigan Law School had, in the past, given extra consideration to
minority applicants and made concessions on testing standards, there
was no evidence of differences in "success" years after graduation.
Gladwell does not address that the admissions tests were not designed
to predict success after graduation, but rather performance in the
first year of law school. Nor does he address that there are
statistical problems with evaluating a selection tool on a sample that
was selected in the first place using that tool. But most important,
Gladwell seems to be arguing that based on the experiences of the
University of Michigan, law schools should abandon altogether their use
of test scores to select applicants.
Well, somewhere between his May speech and the November publication
of Outliers Gladwell must have realized he had his own mismatch
problem. His evidence didn't match his thesis. He must have changed the
direction of his book significantly, because Outliers is barely
relevant to employee selection. Instead, it's a motley collection of
examples arguing that exceptionally successful people are not entirely
self-made, and that their ascent is due also to extraordinary good
fortune with regard to the opportunities they were presented with.
(This is hardly a shocker of a thesis, and is right up there with
Blink's bold contention that first impressions are often correct,
except when they are not.) But where's the employee selection angle?
Where his speech proclaimed that it was "time to shut down the
combines", his book only discusses the mildly interesting fact that the
birthdays of professional athletes tend to cluster near critical
cut-off dates for selection into elite programs. Where he promised to
show how the process of hiring and training pilots is completely at
odds with what is required, he only ends up discussing how cultural
factors can have a tragic influence on the dynamics between pilots in a
cockpit. Rather than discuss the process of hiring teachers, he
describes one successful charter school, and the demands it makes of
its students. He also speculates on
cultural and linguistic factors that might correlate with math
perseverance.
We're left to wonder why the change in focus from the speech to the
book... could it be Gladwell realized that after all there is
substantial evidence for the effectiveness of aptitude testing as a
predictor of job success? Or did he just decide that "what determines
exceptional success?" is a more interesting question (and easier to
write about) than "how can we hire better?"