We’re excited to unveil a new and improved version of our Employee Personality Profile, our most popular general personality test. Below is the email communication we sent to our customers, which details some of the improvements we’ve made. All of the information contained on the old EPP report will still be there, but we’ve made some big improvements that we think will really make it easier to get job-specific insights into what the report is saying about a candidate’s suitability for a particular job.  Here’s an excerpt from the email describing the specific changes:

_________________

Tomorrow we will release an exciting upgrade to one of our most popular personality tests, the Employee Personality Profile (EPP). The enhanced version of the EPP will be available to all HireSelect subscribers, and includes the following enhancements:

12 traits instead of 10. In addition to the ten existing traits, we’ve added two new ones: Openness and Stability. These are two of the “Big Five” traits that will already be familiar to HireSelect users who use the Criteria Personality Inventory (CPI). Detailed descriptions of these traits are on the sample score report for the enhanced EPP.

Redesigned  score report. We updated the graphics and layout of the EPP score report to make it cleaner and more intuitive.

Job-specific benchmarks.  The most important improvement is that we’ve added a series of job-specific benchmarks to the EPP score report.  These benchmarks suggest how closely someone’s personality profile fits with each specific job.  Each benchmark includes suggested score ranges for 4-6 traits relevant to performance and job fit for a particular position.  Individuals are assigned a “job match” score for each position based on how closely their trait scores compare to the benchmark.  These benchmarks are based on composite profiles we created through validity studies and by analyzing hundreds of thousands of EPP results.  By clicking on the position for which a candidate is applying, you’ll get an indication of how closely a candidate’s personality profile matches the typical profile for people who are successful and comfortable in a given position.

__________

If you’re interested in reading more about the new EPP, and checking out a sample score report, click here.

During grad school I taught for the Johns Hopkins Center for Talented Youth which offers advanced summer classes to 7^th and 8^th grade students who score above the national average on the SAT.  In a short three week session, these youngsters gobbled up the Harvard undergraduate Intro to Psych course.  It was fun to work with such bright minds, and I often wonder what became of the students I met.

There are plenty of data on the overall success of kids who participate in these talent search programs.  Last week Aimee Groth at Business Insider shared a table from the Duke Talent Search showing measurable differences in lifetime achievement based on the rank ordering of kids within the talent search pool.  Kids in the top quartile had earned more doctorates, secured more patents, and were more likely to have tenured professorships than kids in the lowest quartile.

The Business Insider folks ran the title “This Table Proves Just How Much SAT Scores Predict Future Success”.  We get it that dramatic titles are good for web traffic. And yes, data like these (and tons more of it in the published literature) should be helpful to people like Keld Jensen who declared in a very poorly reasoned article on Forbes.com that “intelligence is overrated”.

Although her title was an oversell, Groth’s table does challenge the “threshold hypothesis” which claims that after a certain point, individual differences in intelligence don’t matter anymore.  The 7^th and 8^th grade talent searches that use the SAT are doing something very obvious and effective.  They take talented middle school kids and challenge them to a high school test.  The effect is like zooming in with a microscope – resolution increases, and individual differences become apparent in a group that previously looked the same because they all scored at the top of the national middle school standards.  Is it just random in this selective group some kids score 400 on the SAT math section and some score 700? Of course not.  Score differences like that represent real ability differences, even among the elite, and research study after research study has shown that the relative advantage persists to some degree.

This is not to subscribe to some kind of IQ determinism. So what if there is a correlation between 8^th grade test scores and life success? The young kids are not simply given their doctorates and patents.  They need to stay in school, make good choices, find something they love, and work hard at it.  Personality factors like grit, perseverance, inquisitiveness, motivation, integrity, resilience (not to mention external factors like social support and plain old good luck) play a huge role in determining success, and many an IQ hare is passed by a tortoise in the race of life. But there’s no denying that well designed tests can reveal meaningful ability differences.  In the pre-employment setting, cognitive tests have consistently proven themselves as the most effective predictors of future success.

Here’s a link to a press release we put out today announcing the newest addition to our test portfolio. The Workplace Productivity Profile (WPP) is a personality assessment designed to be used for lower and mid-level positions for which an employer needs trustworthy, reliable, and conscientious employees. The WPP has actually been live in HireSelect for more than a month, as we soft launched it in March. I am going to do a blog post with more details on integrity tests later, but if you are interested in learning more about it check out this info on our integrity test.

Everyone wants to compare themselves to Netflix, whose data-driven, personally tailored movie suggestions improve customer satisfaction and retention.  Among the latest domains to see this trend: “learning analytics” in higher education. The basic idea is to use institutional data to help students successfully navigate towards their college degrees.  Doesn’t sound controversial yet – data-driven decision making is usually just plain common sense.

But the details can get a little tricky. Consider the following effort  out of Austin Peay State University in Clarksville TN.  What caught our attention was that in addition to sensible suggestions for ways to meet course requirements en route to graduation, the system also predicts the grades students are expected to receive in their upcoming classes.  The author of the article is impressed with the accuracy of prediction, saying that end of semester GPA was predicted within 0.02 on the 4 point scale, and that individual class grades were predicted within .6.  Furthermore, the probability of getting a C or better was predicted with 90% accuracy.

Seems like one of those numbers is very likely wrong, and the others are not that impressive given a little thought.  There is no way that end of semester GPA is predicted to within 0.02 unless previous GPA is used as part of the prediction, in which case the weighted sum might not move much.  To predict individual class grades, keep in mind that offering a certainty of +/- .6  means offering a range as wide as A to B– .  A prediction interval that wide, centered on the course average from previous semesters, will cover most of the students in the class.  Add in a tweak for the student’s own prior GPA and voila, good prediction. There’s not much magic required to achieve that level of accuracy and I’m sure any university could replicate it in a heartbeat.

But would that be wise? Injecting grade predictions into the student decision making process is likely to affect choices.  Oh, of course the student will decide which courses to take based on interests, goals, sense of personal identity, and other thoughtful and reflective criteria.  But like all rational agents they will also want to maximize their utility and it would be hard to ignore differences in expected reward (grade) “all else being equal”.  That kind of feedback could easily operate as a market force changing the distribution of supply and demand.  And instructors might begin to feel pressure to make their classes easier if they see students voting with their feet.

Isn’t it already good practice to notify a student during a course that their performance on assignments and tests has them at risk for failure? Sure, but such feedback is focused on those who are at risk for facing the consequences (financial and otherwise) of failure, and is based on evaluation of effort and achievement in the course.  Whatever the predicted outcome for a student before a course begins, a grade must still be earned.   Netflix predicts how someone will rate a future movie experience based on how they rated previous experiences.  But a grade (or salary or other reward) must be earned, and the prediction accuracy rests entirely on the assumption that future effort will resemble past effort.   A college student makes a sequence of choices – they choose which classes to take and then they choose whether to expend the necessary effort to succeed.

And here’s where it is unclear how much an individual’s course selection should be driven by their own prior success, and by comparison with the average successes of others. Should we tell people their actual likelihood of overcoming difficult challenges?  The smart money will always bet against anyone trying to lose weight or quit smoking. Should predicted failure be emphasized up front?  Maybe, as it’s certainly not irrelevant to have full knowledge. But what about the risk of discouraging effort, and of enabling complacency? For all the virtues of helping college students avoid getting themselves into classes that are too difficult, there is considerable risk for driving a market for classes that are too easy.  It sounds great to make data-driven decisions, but sometimes introducing data into a system can create unintended feedback dynamics.

Here’s some other suggestions for so-called “learning analytics” in higher ed: we could apply “learning analytics” to actual learning, and guide instruction and assessment within classes in a tailored and evidence based manner.  Or we could build a course selection system with a much broader view.  If it’s fair game to predict grades in upcoming courses, how about also giving hard data on the career success of students who have taken those courses? That would extend the decision making horizon and add more context to the process. And how about also expanding a course selection system to the broader higher education market, including online providers?  Imagine an advising system that suggested a dozen alternatives for meeting a linear algebra or intro stats requirement, each much cheaper than the cost of the resident instruction class at the student’s current school.  That would be a disruptive technological innovation.

by Josh Millet and Eric Loken.

This week the website Calcudoku revisited the question of intelligence differences among users of different web browsers. You may recall the frenzy last summer when a phony report circulated that Internet Explorer users were less intelligent.  That hoax should have been easily spotted because the fabricated data suggested a massive gap on the IQ scale.  Anyone with any knowledge of psychological testing should have immediately understood that the data were impossible. The Calcudoku report is interesting because it uses real data on time to completion of the online puzzles, and also concludes that Internet Explorer users are not as numerically talented. Google CEO Larry Page was, not surprisingly, excited by the findings.

We have access to much more definitive data on this topic.  We provide online pre-employment assessment solutions with a suite of tests covering a broad range of skills and aptitudes. One of our tests, the CCAT, measures critical thinking skills and is often assigned to applicants for higher level positions (managers, analysts etc.).  We have six years of data (more than 1.3 million tests) and recently started tracking browser and operating system information.

Why should any of this be interesting? Many observers last summer and this week lament the emphasis on measures of trait intelligence. Last summer’s hoax was motivated by a software engineer exasperated about wasted hours spent making web code compatible with old browsers like IE 6. He felt that it was stupid to waste so many person-hours, and so he concocted a hoax to blame the users for being stupid. It was an attempt to use “low intelligence” as an insult.  Sensitive as we are to the misuse of intelligence tests to justify prejudice based on gender or race, we have wondered about the social value of any of these discussions.

However, for many people (not all), computer platform and web browser represent a choice.  They represent behavioral choices, and it is of some interest to see if those choices are associated with a trait measure of intelligence. This is certainly relevant to how companies like Apple and Google target their markets. Furthermore, we reasoned that it is better to share good data rather than allow fake or incomplete data to spread. So without further ado, here we go…

The CCAT is a timed test with maximum score of 50.  Scores greater than 40 are rare, and the mean for our sample is in the low to mid twenties. (Keep in mind that on our website, this test is selected by employers trying to fill jobs with greater than average complexity and responsibility, so the pool is tilted to the upper end of the distribution already.)  When we break out the results of 14,264 tests by browser, we see clear differences.  Internet Explorer users are lagging Chrome, Safari and Firefox users by approximately a ¼ standard deviation. The difference is highly statistically significant (F(4,14259) = 66, p < .0001), and it is of modest practical significance (it would correspond to something like 3 or 4 points on an IQ scale).

We’ll discuss it a bit more below, but first let’s look at operating systems. When we isolate the Mac OS X users (1706) from the rest of the pack (almost exclusively Windows), we see a similar and slightly stronger difference.   The mean score for Mac users was 25.26 and the mean for the rest 22.83.  This one-third standard deviation difference was of course highly statistically significant (t(14262) = 12.4, p < .0001).

So what does it all mean?  On the one hand, it’s not worth getting worked up about these results. After all, the groups are only separated by a small amount – 2 to 3 questions out of 50 for an average difference – and this means that the within group variance is much greater than the between group differences. The overlap in the distributions is high, and it is only with marginally higher probability that we would expect any randomly chosen Chrome user to outperform an IE user. No employer should interpret browser use as predictive of the ability of a single prospective employee. And no individual computer user should feel that their browser choice reflects something definitive about their abilities. On the other hand, such robust differences in group means have a basis, and there are implications for the tails of the distribution.  About 2.2% of applicants score 40 points or higher on this test. Even though IE users outnumber all the other browser types combined in the overall sample, they are a 2:1 underdog among those scoring 40 or above. Among Chrome users, more than 1 in 25 scored 40+, while among IE users it was 1 in 75.

The upshot is that there is definitely substance to the recent claims about intelligence differences among users of different browsers and operating systems. We find that Chrome, Firefox and Safari users significantly outperform IE users in a pre-employment assessment designed to measure higher order thinking skills. As Larry Page’s post suggests, Google (and Apple) might instinctively feel some pride in appealing to a higher ability demographic. This has come about either through clever marketing, or because the products have an intellectual appeal. But let’s not get carried away with any of this. By definition Google and Apple are hungry for market share, and that means selling to everyone. Care to bet how things will look 15 years from now?  It’s possible we’ll be saying “Google Chrome is the new IE,” and the tables in this post will look awfully out of date.

Some newspapers and radio stations recently picked up a story that Facebook profiles can be revealing, and can yield information more predictive of job performance than typical self-report personality questionnaires or even an IQ test.

So first off, this is clearly an interesting idea. A rich Facebook profile contains information about a person’s actual behavior, and past behavior is the best predictor of future behavior.  Every parent knows that a quick Facebook search can reveal a lot about a potential new babysitter.  One wonders if that could scale to the corporate level for hiring, and whether it would be ethical to do so?

But let’s start with the surprising assertion, at least as represented in the LA Times story, that the Facebook profile ratings were better predictors of job performance than an IQ test.  A most consistent finding from the last 50 years of organizational psychology research is that cognitive ability is the strongest predictor of job performance, sometimes followed closely by measures of conscientiousness (and recently there has been interest in perseverance or grit).  So has the Facebook study upended all this established research?  Not at all, and the reason lies in the enormous gap between the claims about the study’s outcomes, and the details of what was actually done.

The researchers had two college population samples.  In Study 1 they had job performance ratings for the part-time college jobs of about 10% of the original sample.  But in study 1 they did not have any IQ or cognitive ability measure.  In Study 2 they gathered Wonderlic’s measure of cognitive ability, but this time they had no job performance data but rather college GPA which they say is correlated with job performance.   And it should be put into context too that only some of the college students were careless enough to have publicly available facebook profiles.  All in all this particular research has very little of value to add about predicting job performance in any real world setting.

Ultimately the clues we reveal about ourselves as we socialize and work on the web will prove to be highly predictive of job performance.  After all, internet marketing based on search and social network behavior is enormously successful.  It is obvious that job performance and health and any number of other future high-stakes outcomes will be predicted with increasing accuracy from online behavior, and a great number of ethical questions will ensue.  But those fascinating developments will come from better data and better designed studies than the material recently referenced by the LA Times.