Comments on: Gladwell’s New Yorker Article on Hiring http://blog.criteriacorp.com/2008/12/18/gladwells-new-yorker-article-on-hiring/ Thu, 01 Dec 2011 22:25:27 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Gladwell's New Yorker Article on Hiring | Criteria's Employee … « How to select the best employees for your organization http://blog.criteriacorp.com/2008/12/18/gladwells-new-yorker-article-on-hiring/#comment-268 Gladwell's New Yorker Article on Hiring | Criteria's Employee … « How to select the best employees for your organization Fri, 05 Nov 2010 08:12:24 +0000 http://blog.criteriacorp.com/?p=93#comment-268 [...] posted here: Gladwell's New Yorker Article on Hiring | Criteria's Employee … Uncategorized   his-new, last-post, malcolm, outliers, speech, speech-given, spring, [...] [...] posted here: Gladwell's New Yorker Article on Hiring | Criteria's Employee … Uncategorized   his-new, last-post, malcolm, outliers, speech, speech-given, spring, [...]

]]> By: Ken Shirley and Howard Wainer http://blog.criteriacorp.com/2008/12/18/gladwells-new-yorker-article-on-hiring/#comment-40 Ken Shirley and Howard Wainer Tue, 27 Jan 2009 12:54:52 +0000 http://blog.criteriacorp.com/?p=93#comment-40 In his article, "Most Likely to Succeed", Malcolm Gladwell fails to mention a key feature of the analyses of both NFL quarterbacks and teachers on which he bases his conclusions: both sets of data come from observational studies rather than controlled, randomized experiments. The hackneyed phrase "Correlation does not imply causation" is relevant. The best college QBs are typically assigned to the worst teams because of the rules of the NFL draft, which most likely hinders their chance for pro success, and weakens the association between draft position and pro success, at least for first- round draft picks. In schools, the assignment of students and teachers to classrooms is also non-random, and thus estimated teacher effects are not necessarily causal effects, but are rather observed associations between teachers and test scores that could be explained by a host of confounding variables, such as the underlying aptitude or attendance habits of the students, or the classes that teachers choose to teach according to their seniority. (The point is that teachers, parents, and students all may have their own motivations for gaming the system in their own favor). It makes for good story-telling to point out that a college football star turned out to be a flop while an undrafted college player became an NFL star, or that a teacher with "eyes in the back of his head" succeeded in spite of little formal training, but these anecdotes are not credible evidence for a general conclusion. The truth is almost surely that given past performance, there is more variability in future success than most people believe, but not the total unpredictability that Gladwell suggests. Gladwell should have been more careful to point out that the quantitative studies he cited suffer from the fact that they are based on observational data, which makes drawing conclusions from their analysis infinitely more problematic than it would be were the data from a randomized experiment, as most statisticians, social scientists, and economists know. In his article, “Most Likely to Succeed”, Malcolm Gladwell fails to mention a key feature of the analyses of both NFL quarterbacks and teachers on which he bases his conclusions: both sets of data come from observational studies rather than controlled, randomized experiments. The hackneyed phrase “Correlation does not imply causation” is relevant. The best college QBs are typically assigned to the worst teams because of the rules of the NFL draft, which most likely hinders their chance for pro success, and weakens the association between draft position and pro success, at least for first- round draft picks. In schools, the assignment of students and teachers to classrooms is also non-random, and thus estimated teacher effects are not necessarily causal effects, but are rather observed associations between teachers and test scores that could be explained by a host of confounding variables, such as the underlying aptitude or attendance habits of the students, or the classes that teachers choose to teach according to their seniority. (The point is that teachers, parents, and students all may have their own motivations for gaming the system in their own favor).

It makes for good story-telling to point out that a college football star turned out to be a flop while an undrafted college player became an NFL star, or that a teacher with “eyes in the back of his head” succeeded in spite of little formal training, but these anecdotes are not credible evidence for a general conclusion. The truth is almost surely that given past performance, there is more variability in future success than most people believe, but not the total unpredictability that Gladwell suggests. Gladwell should have been more careful to point out that the quantitative studies he cited suffer from the fact that they are based on observational data, which makes drawing conclusions from their analysis infinitely more problematic than it would be were the data from a randomized experiment, as most statisticians, social scientists, and economists know.

]]>