by Josh Millet and Eric Loken.
This week the website Calcudoku revisited the question of intelligence differences among users of different web browsers. You may recall the frenzy last summer when a phony report circulated that Internet Explorer users were less intelligent. That hoax should have been easily spotted because the fabricated data suggested a massive gap on the IQ scale. Anyone with any knowledge of psychological testing should have immediately understood that the data were impossible. The Calcudoku report is interesting because it uses real data on time to completion of the online puzzles, and also concludes that Internet Explorer users are not as numerically talented. Google CEO Larry Page was, not surprisingly, excited by the findings.
We have access to much more definitive data on this topic. We provide online pre-employment assessment solutions with a suite of tests covering a broad range of skills and aptitudes. One of our tests, the CCAT, measures critical thinking skills and is often assigned to applicants for higher level positions (managers, analysts etc.). We have six years of data (more than 1.3 million tests) and recently started tracking browser and operating system information.
Why should any of this be interesting? Many observers last summer and this week lament the emphasis on measures of trait intelligence. Last summer's hoax was motivated by a software engineer exasperated about wasted hours spent making web code compatible with old browsers like IE 6. He felt that it was stupid to waste so many person-hours, and so he concocted a hoax to blame the users for being stupid. It was an attempt to use "low intelligence" as an insult. Sensitive as we are to the misuse of intelligence tests to justify prejudice based on gender or race, we have wondered about the social value of any of these discussions.
However, for many people (not all), computer platform and web browser represent a choice. They represent behavioral choices, and it is of some interest to see if those choices are associated with a trait measure of intelligence. This is certainly relevant to how companies like Apple and Google target their markets. Furthermore, we reasoned that it is better to share good data rather than allow fake or incomplete data to spread. So without further ado, here we go...
The CCAT is a timed test with maximum score of 50. Scores greater than 40 are rare, and the mean for our sample is in the low to mid twenties. (Keep in mind that on our website, this test is selected by employers trying to fill jobs with greater than average complexity and responsibility, so the pool is tilted to the upper end of the distribution already.) When we break out the results of 14,264 tests by browser, we see clear differences. Internet Explorer users are lagging Chrome, Safari and Firefox users by approximately a ¼ standard deviation. The difference is highly statistically significant (F(4,14259) = 66, p < .0001), and it is of modest practical significance (it would correspond to something like 3 or 4 points on an IQ scale).
We'll discuss it a bit more below, but first let’s look at operating systems. When we isolate the Mac OS X users (1706) from the rest of the pack (almost exclusively Windows), we see a similar and slightly stronger difference. The mean score for Mac users was 25.26 and the mean for the rest 22.83. This one-third standard deviation difference was of course highly statistically significant (t(14262) = 12.4, p < .0001).
|Mac OS X||25.26||7.25||1,706|
So what does it all mean? On the one hand, it's not worth getting worked up about these results. After all, the groups are only separated by a small amount – 2 to 3 questions out of 50 for an average difference – and this means that the within group variance is much greater than the between group differences. The overlap in the distributions is high, and it is only with marginally higher probability that we would expect any randomly chosen Chrome user to outperform an IE user. No employer should interpret browser use as predictive of the ability of a single prospective employee. And no individual computer user should feel that their browser choice reflects something definitive about their abilities. On the other hand, such robust differences in group means have a basis, and there are implications for the tails of the distribution. About 2.2% of applicants score 40 points or higher on this test. Even though IE users outnumber all the other browser types combined in the overall sample, they are a 2:1 underdog among those scoring 40 or above. Among Chrome users, more than 1 in 25 scored 40+, while among IE users it was 1 in 75.
The upshot is that there is definitely substance to the recent claims about intelligence differences among users of different browsers and operating systems. We find that Chrome, Firefox and Safari users significantly outperform IE users in a pre-employment assessment designed to measure higher order thinking skills. As Larry Page's post suggests, Google (and Apple) might instinctively feel some pride in appealing to a higher ability demographic. This has come about either through clever marketing, or because the products have an intellectual appeal. But let's not get carried away with any of this. By definition Google and Apple are hungry for market share, and that means selling to everyone. Care to bet how things will look 15 years from now? It's possible we'll be saying "Google Chrome is the new IE," and the tables in this post will look awfully out of date.