We made a few posts last year about the NFL and whether or not draft order is related to productivity. The core issue for us was a claim Malcolm Gladwell repeatedly asserted that the draft order of NFL quarterbacks (QBs) is unrelated to performance. Well, the issue was raised again over the Labor Day weekend and we were alerted to some more recent material we hadn’t seen because to be honest we thought we were done with the whole thing. We found this very sensible WSJ blog from last December, but then we also found this CNBC blog from May of this year. Darren Rovell, the CNBC blogger, reproduced the following table from economist Dave Berri. It purports to show that performance of lower drafted QBs is similar to that of the top drafted QBs. Now to be fair, the table was used to argue that the cost-benefit of the lower picks might exceed that of the higher picks and that is entirely plausible. But Berri also uses a table like this to argue that draft order is not a good predictor of success.
Performance of Quarterbacks Selected from 1980-2009
WP100 = Wins Produced per 100 plays
Performance Adjusted for Average Observed in Each Season
|Picks||Seasons Observed||Total Plays||Relative Wins||Relative WP100|
Punch-line: Quarterbacks selected between picks 11-50 outperform picks 1-10 (and cost less)
Source: Dave Berri/Stumbling On Wins
Because we blog about issues in employee selection and we want to be helpful to our clients and anyone else interested in the topic, we just have to say please don’t analyze your data by summarizing it in a table like this one. The real unit of analysis is individual QBs and whether they are good selections. The table aggregates into “QB years,” so some QBs are contributing 5 to 10 years of data and some are contributing just one. And what happens to the QBs who have no data to contribute? They disappear. Compare the first row of the table (draft order 1-10) and the third row (draft order 51-90). Loosely speaking, you’d think that there would be about 4 times as much data from a range of 40 draft positions compared to a range of 10. But instead, there are fewer “QByears” and less than half as many plays in the larger pool compared with the elite pool. It’s because a huge proportion of the lower draft order QBs never made any contribution at all. The average performance documented in the fourth row of the table is the average performance of the lower drafted QBs who succeeded in the NFL.
Berri argues that aggregating this way is necessary to avoid a self-confirming “opportunity bias”. The legitimate point is that highly drafted QBs will get more playing time, and so total production might be a misleading metric of success. But the table goes to the other extreme and says that anyone who never played in the NFL was just denied the chance to show what they could do, and they should be omitted from the analysis.
If you are evaluating your own employee selection system, you have to think carefully about the correct way to organize and analyze your data. Any selection mechanism, including pre-employment testing, yields false positives (people who were significantly overrated by the selection system relative to their contribution) and false negatives (people who were underrated by the selection system relative to what they did contribute – or what they could have contributed had they been selected at all). Before you say, “Hmm, I think the weekly sales of my staff who scored low on the selection test is pretty similar to the weekly sales of those who scored high” you have to consider the full sample. What fraction of those hired with low scores didn’t pass training, or were fired, or quit? If you find that the rates of attrition and failure are similar for the low and high performing groups, then maybe you should indeed wonder if the selection tool matters all that much for predicting sales. But if you find that attrition in the lower group is double or triple that of the higher group, then you have to realize that you are only including the best of the low qualifiers in your comparison. Comparing the high and low selection groups on total sales would immediately reveal the hidden risk and cost associated with the lower qualified group. Only you will know, for the purposes of your specific business setting, whether any opportunity bias that might sneak in from considering total productivity is of more or less concern than the bias that comes from pretending the failed hires never happened.