I finally got around to seeing “Moneyball” this weekend, the movie adaptation of the Michael Lewis book of the same name. The movie documents the role played by Billy Beane, General Manager of the Oakland A’s, in transforming the way baseball teams drafted and evaluated players a decade ago. Beane and his staff pioneered the application of sophisticated statistical analysis to the process of player selection. In so doing he was able to help his chronically underfunded Oakland As compete with the big budget teams like the Yankees, whose payroll was four times that of the As. His methods have since been imitated by many other teams, including the Boston Red Sox, who used it to win two World Series championships.

The lessons of Moneyball have obvious implications that reach beyond baseball, and it has garnered some lively discussion in HR circles. Beane’s breakthrough was that he found objective, quantifiable ways to measure player potential that turned out to be more accurate predictors of on field success than the collective wisdom of baseball scouts and insiders. This is exactly the promise of pre-employment testing. Well designed tests provide employers a way to gather objective, reliable data that predicts performance more accurately than traditional, more subjective methods of employee selection such as interviews.  Most of our clients are small and medium-sized businesses, for whom hiring smarter is one of their best chances to compete with the bigger “Yankees” of their respective fields.

This holiday shortened week has brought some weak economic news to the fore, and yesterday the stock market took a steep loss.  Some analysts are pointing to tomorrow’s monthly payrolls numbers as an important event that could significantly impact the markets.  As avid readers of this blog will remember, our Hiring Activity Index (HAI) is a metric based on the proportion of our customers who are actively conducting pre-employment testing in a given month. The HAI touched an all-time high for the month of May.  We were also stable and high for March and April.

We’re not sure whether this strength reflects the hiring environment, or a new maturity in our business model as we build a loyal customer base.  Either way, we’re pleased with the indicators and expect that the jobs number tomorrow will likely be decent. The consensus estimate seems to be that we’ll have added between 90,000 and 200,000 non-farm payroll jobs: if the HAI is any guide, as it has been in the past, we’d be surprised if the number isn’t on the high end of that range.

Today is that time of year again, the NFL draft. Not quite the same with the labor situation overhang, but that doesn’t seem to have slowed the perennial debates about draft order and the professional prospects of various members of this year’s draft class.  Our blog posts on the NFL draft are always among our more widely read posts, and we’re very interested in the draft because it is such an iconic example of NFL teams devising methods to tackle the challenge we think about every day: devising employee selection systems that help organizations hire better and derive long-term competitive advantages.  I don’t have much new to add on the draft this year, but did want to highlight a really interesting article on a new entrant to the field.

http://www.slate.com/id/2292312

If you have thoughts on this approach let us know in the comments section.

Just a quick heads up that we introduced a new computer skills test today. If you want to read the press release it is here. A lot of customers have been asking about a quick assessment of basic computer literacy, so we created one. It’s designed to determine whether a candidate has fluency with basic computer skills that many employers take for granted–using the internet, email, etc.  Very often employers find out too late that they should NOT be taking these skills for granted, because even in the US, they are far from universal.  The name of the new test if the Computer Literacy and Internet Knowledge Test (CLIK).

Okay, first an upfront explanation of why we are even blogging about this. We are a web-based business that generates vast amounts of data. We continuously monitor and analyze our data, and even sometimes blog about what we find. So when we saw that a blog from OKCupid was the source of headlines such as, “The Curse of Being Cute” we had to see what they had done with their data. They did this.

OKCupid is an internet dating site. They have millions of users, and their users have rated each other on attractiveness (on a 1 to 5 scale, low to high) and sent messages to each other.  Of course getting a lot of 5s was most predictive of getting lots of messages. A second claim by the company, and the one getting most of the attention, is that being rated 1 was the next best thing to being rated 5 when it comes to getting more messages, and being rated 4 could even have a negative impact on he flow of messages. Hence the “curse of cuteness” and the discussion that being being polarizing can be beneficial.

To translate it to an employment setting, let’s suppose that on a job search site, prospective employers can rate potential applicants on a 1 to 5 scale. And suppose that it’s also possible to track the number of interview offers made to job applicants. Then here is the graph offered up by the folks at OKCupid:

Of course as an employment blog we’ll read the x-axis as average employer rating and the y-axis as interview requests.

Notice the clear upward trend. Notice also that it is an accelerating upward trend and not just a straight line relationship. The OKCupid folks calculate the positive slope, but then noticed that there was also still a lot of scatter around the straight line they drew. They found that in addition to the average rating being predictive of getting messages, the variability of the ratings was also predictive.  All else being equal, it was better to have variability in the profile of ratings. Imagine two people who both have an average rating of 4.0. If one had all ratings of 4.0 (i.e., no variability) and another had some variability, the second person would be expected to get more messages. (Hopefully you see that this is because that person must have more 5 ratings than the other person…but we’ll get to that.)

Okay OKCupid, so far so good. But next comes the egregious error. They decided to construct a regression equation that used all the individual rating profiles to predict the number of messages received.  As inputs to their regression they put the number of 1 ratings, the number of 2 ratings, the number of 4 ratings and the number of 5 ratings (they left 3 out because it caused some redundancy).  Here’s the equation they came up with:

Now it would have been helpful for them to have given the uncertainty in those regression weights, but let’s just see how they interpreted them. Correctly, they noticed that the largest positive weight was given to the 5 ratings. Indeed, there is almost a 1-1 payoff in that for every additional highest rating someone receives, they are expected to get .9 more messages. But then they went on to claim that the lowest category appears to have the next highest positive association. They actually conclude that the next best thing to getting a 5 rating is to get a 1 rating. By their logic, it would be better to be a person who has 10 of the worst rating (expected messages = k + .4*10 = k + 4) than to be a person who has 100 of the 4 rating (expected messages = k – .1*100 = k -10). Really??

Where did they go wrong? First they looked at the individual parts of the equation without realizing that it has to be looked at as a whole.  You can’t just imagine adding or subtracting specific ratings without thinking about the whole profile.  The number of 1, 2, 3, 4 & 5 ratings received are interrelated – the more high ones you have, the fewer low ones and vice versa.  It’s not easy to imagine dialing up and down the number of low ratings while holding everything else constant.  So because the rating counts are so interrelated, the regression weights are also interrelated, and they should not be interpreted on their own.  Otherwise you quickly come to silly examples such as it is better to receive all low ratings than to receive all 4 ratings.  That just can’t be true.  Second, they are ignoring that receiving a lot of 1 ratings has two meanings: it might mean you’re very unpopular, but it also might mean your profile has been rated a whole lot of times and when that happens ratings will accumulate in all the slots from 1 through 5.  When predictors in a regression have double meanings like this, you often see some paradoxical behavior.  In this case, 1s on their own are probably neutrally or negatively related to number of messages; but when the number of 4s and 5s are also included, things can change.  And finally, the association between messages and ratings, as seen in the first graph, is non-linear meaning that the pattern of regression weights will favor the highest category.  The weights were [.4, -.5, -.1, .9].  It’s not a coincidence that the graph has a non-linear feel, and that those weights trace out a curve.

Students in an introductory statistics class would easily spot the errors in the OKCupid analysis and realize that the authors had jumped to an unsupported conclusion. But judging from the retweets and the Facebook shares, people are just running with the study’s conclusion and not bothering to evaluate the evidence. Let me be clear – if you are a job seeker, it would be a ludicrous strategy to say, “Gee, if I can’t get prospective employers to rate me as the best possible candidate, the next best thing I can do is get them to rate me as the worst possible candidate.” Such a strategy would be just flat out….well, I don’t want to be rude but I’m thinking of a word that rhymes with “cupid”.

I was interviewed the other day by Ric Franzi, who does a twice weekly business show where he interviews CEOs of companies based here in SoCal. It’s actually fairly extensive–25 minutes long, as we get into how Criteria’s employee testing services help small companies address their hiring challenges. If anyone is interested here’s the interview.

A recent New York Times article highlighted the challenges faced by a small business owner in devoting sufficient time to hiring new staff. The challenges are particularly acute in the current economic climate, it was argued, because the number of applicants per open position has risen dramatically.  An unemployment rate that hovers stubbornly around 10% has certainly meant more competition for fewer jobs. For job searchers, this makes it even more important to stand out in a crowded field. For employers, this “buyer’s market” means that although there is plenty of talent available, it can be difficult and time consuming to find the right person for the job when there are so many applicants.

Of course, pre-employment testing is a very efficient way to filter through large applicant pools. Because companies use our HireSelect pre-employment testing software to help them do just this, we are able to gather data that offers real-time insights into hiring trends. For example, we can track, for every job opening on our site, the number of test takers who show up to take the tests.  This presumably works out to a reasonable measure of the number of applicants per position.  Since I haven’t seen much hard data out there about the extent to which applicant pools have grown, I thought it would be interesting to share our findings. Our data confirms the anecdotal reports of much bigger applicant pools. In fact, as the graph below shows, amongst small businesses the average number of applicants per job has just about doubled since 2008, when the recession began.

Just to be clear on what the data represent, we’ve actually tallied test batteries by company, and test takers per battery.  Often larger companies set up a test battery with the intent of multiple hires.  That was our rationale for restricting our sample to smaller companies (companies of fewer than 50 employees), as they are more likely to be making only one hire per position opening.  We should also note that our company is relatively young and growing rapidly, so the pool of companies and how they use our services has evolved since Q3 2008.  All that considered though, the basic point remains that by a constant metric on our site we have seen a large increase in the number of test takers showing up to take pre-employment tests on our site, even when considered on a per-opening basis.