Criteria’s Employee Testing Blog

Microsoft Skills Tests Now Live on HireSelect

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Well, we've now put the Microsoft skills tests live on HireSelect (read about it in the press release.)  The integration of these tests took longer than we expected, and right now they are in beta release, because we are still gathering customer feedback and making sure they are bug-free. 

One of the great things about delivering software over the web is that when we make changes we get immediate feedback from our customers. If a new feature isn't intuitive, we hear about it right away and can immediately improve it; if there's a bug that we didn't catch in QA, we learn about it right away once a customer encounters it. While constantly upgrading our software does introduce the possibility of minor technical glitches, this is far outweighed, we believe, by the fact that our customers don't have to wait six months for the next release to see new features they want, as is the case with companies that are still distributing their software in shrink-wrapped CDs. 

What is a Sales Aptitude Test?

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

In my last post I described our customer service test and the kinds of personality traits that it measures. People who have high levels of cooperativeness, patience, and personal diplomacy tend to be well suited for customer service roles.  The use of personality tests is even more widespread, however, in helping select salespeople, because there's a lot of research that shows that people with certain personality traits tend to be successful in sales roles across a wide range of industries.  Most personality tests that are designed to help select salespeople look for outgoing, fairly aggressive people that tend to be competitve and highly motivated. This general profile of a stereotypical sales professional is probably not all that surprising. But what kinds of research underlies this type of "sales profiling?"

The sales aptitude test featured in HireSelect is called the Sales Achievement Predictor. The professors who created the test validated it in part by comparing the 15 personality traits it measures to job performance data for various samples of salespeople.  The highest performing salespeople tended to be competitive, outgoing, highly motivated, assertive individuals.  For example, in a sample of 156 real estate sales professionals whose test scores were compared to their job performance, the highest correlations were observed in the following traits: Achievement, Motivation, Initiative, Assertiveness, Competitveness, Goal-Orientation, and Extraversion (the correlations were .53, .43, .42, .38, .38, and .36, respectively.)  Interestingly, low or even negative correlations were observed for Cooperativeness and Patience, suggesting that when it comes to sales being too patient or too cooperative can sometimes be a liability rather than an asset.  We've conducted numerous case studies with our customers that essentially confirm these findings: the most successful salespeople tend to be competitive, assertive, and relatively impatient individuals--in short, nearly the opposite of the type of people who are best suited to customer service.  To be sure, the type of personality that is best suited for a particular sales role can vary from one organziation to another, and from one industry to another, depending on the nature of the sales process and the sales culture in a given environment. But the basic building blocks of what personality traits you should look for in selecting sales people are remarkably consistent across all industries. 

Click here to read more about our sales aptitude test.

What does a Customer Service Test Measure?

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

In a series of blog posts last year I discussed the evolution of employment personality testing, from the Myers-Briggs to the "Big Five" to more job-specific personality inventories. The last of these continue to grow in popularity, especially in the fields of customer service and sales.

Our customer service test is now one of the most widely used tests in HireSelect. One reason for this is that customer service representative positions tend to have higher than average turnover, and so HR managers are constantly looking for tools that can help remedy this problem. But another reason we're seeing these tests grow in popularity is that many organizations seem to be placing a heavy emphasis on cultivating a "culture of customer service" across an entire organization. We hear this regularly from customers as diverse as medical care providers, banks, and non-profits.  The result is that applicants for managerial and administrative positions may be asked to take these tests if their jobs will involve frequent interactions with customers and/or the public.  So these tests are not just used for Customer Service representatives anymore. 

So what does a customer service personality test measure?  Our customer service test is called the Customer Service Aptitude Profile (CSAP), and it measures a series of personality traits that are widely regarded as important for successful interactions with customers.  These traits include such things as patience, cooperativeness, and personal diplomacy.  Because customer service personnel are foot soldiers on the front lines of an organization's efforts to build and mantain its reputation and brand, companies look for individuals who will be able to answer questions and resolve customer complaints in a helpful, compassionate way. Impatience, competitiveness, and uncooperativeness may be traits that can lead to success in some fields (such as sales, actually--more on that in my next post), but they are not traits you'd look for in customer service representatives.  

Click here to see a sample score report for the CSAP. 

Hiring Activity Index Ticks Higher Again in March

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

There was significant interest in my post about the Hiring Activity Index last month, so I thought I'd follow up. The post was even picked up by the widely read blog of Columbia University statistician Andrew Gelman, who made a chart out of the horrible table I posted (mea culpa). There were some great comments on Dr Gelman's blog about the possible utility and limitations of the HAI as an economic indicator, some of which I'll address in a second. After Dr Gelman featured our post it was subsequently picked up in a widely read economics blog written by a University of Oregon economist and then again here too.

It's cool to see that smart people find our data interesting, and so we wanted to provide an update and some more information on the HAI. First, the update. With the final March numbers now in, the Hiring Activity Index nudged upwards very slightly again this month, to 62.3% from 61.4% in February. To me this is an encouraging sign that the February jump in hiring activity by small businesses was not just a blip. If the data we're seeing means anything, the hiring situation for small and medium-sized businesses has begun to rebound.

There were some interesting comments and questions about the HAI and its potential utility as a leading economic indicator. We do sell our software on a subscription basis, and someone pointed out that if non-active subscribers didn't renew because of the downturn, this could artifically inflate the HAI because it is based on the percentage of our customer base that is actively doing pre-employment testing in a given month. This is a legitimate point, but I will say that while low levels of use are a reason that customers sometimes do not renew, we haven't see non-renewal rates climb much since November, when the HAI dropped by 10 points. It was also suggested that higher numbers of job-seekers may result in applicants for positions that may not have been desirable previously--this is theoretically possible, but I don't see much evidence for it. What is most certainly true is that companies are getting far more applicants per open positon, as I previously blogged about here. However, since the HAI is based on the percentage of companies testing in a month, not the overall volume of tests, this shouldn't influence the HAI unduly, and wouldn't in any case explain the plunge in November and (partial) rebound in February.

Computer Skills Tests Coming

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Yesterday we announced a partnership with another test publisher, Skillsarena, that will allow us to add Skillsarena's Microsoft skills tests to HireSelect.  You can read the press release here.

Many of our customers have been asking us about computer skills tests for a while now, and we believe we've found the right partner to fulfill this need. We think the tests will add significant value to HireSelect, and we expect that we will finalize integration of the tests in late May. 

Are Small Businesses Starting to Hire Again?

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

A recent Department of Labor Report noted that mass layoffs at large companies surged in January. Not surprising, really, since we already knew the unemployment rate for January rose to 7.6% and job losses seem to be increasing by the day. But how is the job picture with respect to small and medium-sized businesses (SMBs)?

In my last blog post I mentioned a statistic that we track internally here, the Hiring Activity Index. The HAI is essentially a measure of how actively our customers (made up mostly of SMBs of between 10 and 500 employees) are administering pre-employment tests through our system (and presumably, therefore, hiring).  It is useful to us because it provides an overview of the breadth of usage of our pre-employment testing software.  It is calculated in a very simple way: the HAI is the percentage of our customers who are actively hiring (administering tests) in a given month.  From January 2008 (when we began tracking the HAI) to October 2008 the HAI remained very steady, within a few points of 65%. (If this seems low, consider that even in the best of times many 30 or 40 person companies will not be hiring every month.)

But as the financial markets plummeted and the unemployment rate surged in November, the HAI sunk about ten points, and by January reached its lowest level since we started tracking it, 53.28%. (The monthly readings for the HAI since January 2008 are listed below.)  Small companies, like their larger counterparts, had severely curtailed their hiring.  The financial news has not gotten much better since then; mass layoffs are continuing, we learned that the US GDP shrank 6.2% in Q4,and the stock market's descent shows no signs of abating.

So I was very pleasantly surprised to see a fairly strong uptick in the HAI in February, to 61.41%. It is only one data point, to be sure, but it suggests that for SMBs the hiring picture improved somewhat in February. Could it be an upwards blip in a downward trend? Of course, but the eight point jump in the HAI is the biggest we've seen since we started tracking the index. For those, like me, inclined to think that the current recession, although brutal and severe, will not be as long-lasting as some suppose, the February HAI reading is cause for hope. I don't expect that January's 7.6% figure for the overall unemployment rate is the end of it--we'll almost certainly see it get north of 8% soon. But as big public companies in the worst hit industries (financial services, construction, etc) continue to shed jobs the February HAI reading offers a glimmer of hope for the job market. Small and medium-sized businesses did not lead us into this recession, but they may just lead us out of it--and don't look now, but it may have already started. 

Hiring Activity Index

The Hiring Opportunity of a Lifetime?

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

At Criteria we develop and deliver pre-employment testing software that our customers use to help enhance their hiring process. Because our service is web-based we can track our customers' usage patterns pretty closely, and this provides us with great insights into hiring trends across the U.S. (we currently have hundreds of customers in 46 different states.)

One of the metrics we track is a monthly "Hiring Activity Index" that essentially measures the percentage of our customers who are actively administering pre-employment tests, and therefore, presumably, hiring. From January to October of 2008, the Hiring Activity Index was remarkably consistent, always hovering around 65%.  When the turmoil in the financial markets caused the unemployment rate to surge in November, however, the Hiring Index dropped a full ten points, and by January it was down to 53%. In the past few weeks, however, we've seen pre-employment testing activity surge to the highest level we've ever seen: it looks like the total number of tests delivered through HireSelect in the month of February will be 25-30% higher than in any previous month.

What gives? If significantly fewer companies are hiring now than were three months ago, why is the total level of applicant testing on our site surging? There's no question that some companies in certain industries (construction, financial services) have stopped hiring for the moment, but the data shows that companies that are hiring are screening far more applicants for each position than they were previously. Companies that had 6 applicants for a position now often have 20; where 25 applicants for a position was normal, some companies are now getting 60 resumes.

The silver lining for employers in this giant economic mess, therefore, is that there is a glut of job seekers on the market right now. Those companies that are hiring have their pick from candidate pools that are stocked with talent. The upside of the downturn, it seems, is a real opportunity for HR managers to land great employees. 

Don't Ask, Don't Tell: The New Rules of the SAT and College Admissions

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Today's blog post is by Dr. Howard Wainer, who is the Distinguished Research Scientist at the National Board of Medical Examiners, as well as Professor of Statistics at the Wharton School of the University of Pennsylvania. Dr. Wainer received his Ph.D. from Princeton Univeristy, has won numerous scholarly awards, and spent 21 years as Principal Research Scientist in the Research Statistics Group at the Educational Testing Service.  He is also, as far as we know, the only member of Criteria's Scientific Advisory Board to have swam the English Channel.

On September 22, 2008, the New York Times carried the first of three articles about a report, commissioned by the National Association for College Admission Counseling, that was critical of the current college admission exams, the SAT and the ACT. The commission was chaired by William R. Fitzsimmons, the dean of admissions and financial aid at Harvard.

The report was reasonably wide-ranging and drew many conclusions while offering alternatives. Although well-meaning, many of the suggestions only make sense if you say them fast.

Among their conclusions were:

1. Schools should consider making their admissions "SAT optional," that is allowing their applicants to submit their SAT/ACT scores if they wish, but they should not be mandatory. The commission cites the success that pioneering schools with this policy have had in the past as proof of concept.
2. Schools should consider eliminating the SAT/ACT altogether and substituting instead achievement tests. They cite the unfair effect of coaching as the motivation for this — they weren't naive enough to suggest that because there was no coaching for achievement tests now that, if they became more high stakes coaching for them would not be offered. Rather, they argued that such coaching would be related to schooling and hence more beneficial to education than is coaching that focuses on test-taking skills.
3. That the use of the PSAT with a rigid qualification cut-score for such scholarship programs as the Merit Scholarships be immediately halted.

I will not attempt to discuss all three of these here, just the first one — if there is sufficient interest shown in this topic this entry will be followed by others. 

Has the admissions process been hampered in schools that have instituted an SAT optional policy?

The first reasonably competitive school to institute such a policy was Bowdoin College, in 1969. Bowdoin is a small, highly competitive liberal arts college in Brunswick, Maine. A shade under 400 students a year elect to matriculate at Bowdoin, and roughly a quarter of them choose not to submit their SAT scores. In Table 1 is a summary of the classes at Bowdoin and five other institutions whose entering freshman class had approximately the same average SAT score. At the other five institutions the students who didn't submit SAT scores used ACT scores instead. 

To know how Bowdoin's SAT policy is working we will need to know two things. First, how did the students who didn't submit SAT scores do at Bowdoin in comparison to those students that did submit them? And second, would the non-submitters performance at Bowdoin have been predicted by their SAT scores?

The first question is easily answered by looking at their first year grades at Bowdoin. These are shown in Figure 1 below.

We see that non-SAT submitters did about a standard deviation worse than students who did submit SAT scores. And so, we can conclude that if the admissions office were using other variables to make up for the missing SAT scores, those variables did not contain enough information to prevent them from admitting a class that was academically inferior to the rest.

But would their SAT scores have provided information missing from other submitted information? Ordinarily this would be a question that is impossible to answer, for these students did not submit their SAT scores. However, all of these students actually took the SAT, and through a special data-gathering effort at the Educational Testing Service we find that the students who didn't submit their scores behaved sensibly. Realizing that their lower than average scores would not help their scores at Bowdoin, they chose not to submit them. Below (Figure 2) is the distribution of SAT scores for those who submitted them as well as those who did not.

As it turns out the SAT scores for the students who did not submit them would have accurately predicted their lower performance at Bowdoin. In fact the correlation between grades and SAT scores was 12% higher for those who didn't submit them than for those who did.

So not having this information does not improve the academic performance of Bowdoin's entering class — on the contrary it diminishes it. Why would a school opt for such a policy? Why is less information preferred to more? There are surely many answers to this, but one is seen in an augmented version of Table 1 (below).

We see that if all of the students in Bowdoin's entering class had their SAT scores included the average SAT at Bowdoin would sink from 1323 to 1288, and instead of being second among these six schools they would have been tied for next to last. Since mean SAT scores are a key component in school rankings, a school can game those rankings by allowing their lowest scoring students to not be included in the average. I believe that Bowdoin's adoption of this policy pre-dates US News and World Report's rankings, so that was unlikely to have been their motivation, but I cannot say the same for schools that have chosen such a policy more recently.

The Obama Effect?

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Last week the New York Times published an article on a possible Obama effect on test scores of black test takers. It was unusual for a major newspaper to publish a story on a social science study before that study has been published, let alone reviewed. But when you hear that so-and-so reported their results at some national conference, that isn't really peer reviewed either. The conference organizers have often only seen a 200 word description of what the researchers thought they would present. So although unusual, it's not entirely out of line to try to get the first step on a story like this, and the Times did circulate the study to some academics to get professional opinions.

Let me say at the outset that I hope the central result is true. The authors claim that they gave a short academic aptitude type test to black and white test-takers. When they administered the test last summer, they noted a difference between average scores for blacks and whites. However, after (now) President Obama had received his party's nomination and given his acceptance speech, the difference in scores disappeared. The theory is that Obama's rise has had a positive motivating influence on test taking performance.

The story has legs because there is a well-documented body of research on test performance, and how it can be affected by contextual cues. You can start with the cultural beliefs about aptitude tests in general. If there is a belief among one target group that the tests always show underperformance, then that belief can have a self-fulfilling aspect. Researchers have experimentally manipulated that contextual clue by describing tests differently to participants before they take them. Researchers have also manipulated the race and gender of the test administrator and done a variety of clever tricks to see to what extent performance can be affected by context. One enterprising team actually had women of Asian heritage take a math test, randomly dividing them into one group who answered a questionnaire designed to get them to think of their female identity, while the other group answered questions about their Asian identify. Guess what? One group underperformed relative to the other, and because the study was conducted as a randomized experiment, the authors are allowed to infer that their contextual manipulation caused the differences in performance.

So I'm sympathetic to the study described in the Times, and I fully appreciate the research tradition it comes from. That said, there are a couple of warning flags about the study. First, it is unclear from the Times piece whether there was any reference at all to Obama before the participants took the test. If not, then the story must be that if there was a difference in performance over time it was because Obama was "in the air". That's true enough – he certainly was in the air. The country was electrified. But most studies on test taking performance try to make the contextual cue more closely connected to the test taking event. Lots of things happened from last summer to now...millions of jobs were lost, the stock market tanked, Tom Brady was injured, and the seasons changed.

But the more worrisome concern is the quality of the data.  Based on the Time article, it seems like there were four tests, and at each occasion there were maybe 20 black participants. Furthermore, the age range of the participants was around 50 years. I don't want to make your eyes swirl with statistical mumbo jumbo...but let me throw out these two points. The degree of sampling variability from occasion to occasion would be huge. Would you trust the results of an opinion poll that gathered a group of 20 participants? So why trust the results of a test taken by 20 people? It's all the more problematic that the researchers are trying to prove a lack of difference. With such a small sample size, and such wide variability of participants in age and occupation, it becomes very difficult to prove that a difference exists. But as I have to remind my PhD students everyday – failing to prove that there is a difference is not the same thing as proving that there is no difference. Their eyes swirl at me too.

Come to think of it, it makes you wonder why everyone is looking at the data in this particular way. The story is that on the one testing occasion before Obama's meteoric rise, there was a black white difference, and then it disappeared over the next three testing occasions. The implicit reasoning is that something has happened. But why privilege the summer result so much? Why not ask "What was happening last summer that made a black-white difference show up?" Why assume that that result is somehow "true" and that it has recently "disappeared"?

At any rate, more data is already in hand. There have been several administrations of the SAT during the election run, and even one since President Obama's inauguration. Let's take a look at the national trend based on millions of scores. I'd be very happy if there is something to write about. I personally expect that there will be something to write about over time, but I also believe that the evidence is going to take some time to develop. Let's hope the New York Times is still paying attention then, and not just trying to front-run another study that has barely been mailed out for review.

Transparency and Truthfulness (in Advertising)

Email Article |  digg it |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

First, a quick disclaimer: As you might have guessed from the pretentious title of this post, I'm not going to write today about pre-employment testing, statistics, or anything relating to psycyhometric testing. But this rant, er..post...does touch on principles that are crucial to how we try to operate our company and that seem increasingly imperiled in the age of Bernie Madoff, balance sheets hiding toxic debt and opaque finacial instruments that no one can seem to value, and so on. Being transparent and open with customers, employees, and investors has such obvious benefits that it still strikes me as weird when people don't understand this. But first, let me tell you the story that got me thinking about transparency as a business imperative.

Several months ago Criteria moved into a new, larger office overlooking Santa Monica Boulevard, with a view of the Santa Monica mountains to the north. Things were going pretty well in the new space, until just after Christmas, when we returned to discover that over the holiday our landlord had covered the windows of the entire 3rd and 4th floor windows of our building with an enormous 150-foot vinyl "supergraphic" advertisement for a multinational bank that will remain nameless. Happy Holidays! Apparently the sign was part of a larger trend of "supersign" advertisements going up suddenly across Los Angeles over the holidays, a trend reported in the LA Times on Jan 1.

The sign had an awful effect on our office space: it allowed very little natural light into the office, and of course our view was ruined. On the fourth floor the sign was a really obnoxious shade of orange, so that in the late afternoon my office felt like being inside a lava lamp. We were all very angry.  Calls and emails to the building owner were ignored and often unacknowledged. Incredibly, when we did finally get a response, the building managers and owner at first seemed to imply they were not aware that the sign was up, and would have to consult with the "sign company." It would be down in two months, we learned next. This response prompted more angry emails, this time more emphatic and sarcastic than the last. Some of our co-tenants wrote letters to the City Council, and eventually a fire marshall came in and pronounced the sign a violation of the fire code, because it hindered entrance and exit from the building. 

Finally, this week the LA Times ran a story entitled 'Supergraphics'  Anger Tenants in L.A. Office Buildings that featured an interior picture of the office of our neighbor Pat, a physical therapist who had been instrumental in rallying opposition to the sign.  (Thanks Pat!) Sure enough, the next day workers appeared and took down the sign, just like that. 

What most annoyed us about "signgate" was that the landlord put up the sign with no warning, the building manager denied any knowledge of it being up, and we were provided with no credible explanation for what was happening or when we could expect it to be reversed. I'm not saying we'd have been ok with it if we had been told in advance, but at every stage the increasingly angry tenants in our building could probably have been at least partially mollified by a little more credible, timely information about what the owner planned to do to respond to our concerns.  A little more transparency would have gone a long way.

At Criteria we practice a kind of "radical transparency" that might be too much for some companies to handle. We try to dispense as much information to customers, employees and investors as possible. We allow prospective customers to access a comprehensive, fully functioning version of our pre-employment testing software for 30 days prior to purchasing it, with no obligation.  We publish real-time sales figures on our company intranet, so that every employee can tell exactly how much money we've made in a given quarter, month, or day. And we provide detailed updates to investors not quarterly but monthly, so that they know exactly how we're doing at all times. 

This kind of transparency might not be practical for very large companies, and is not without risk. Occasionally a small company that has no intention of buying our software will use our tests very actively for a month during a hiring spree. (Our VP of sales calls such companies T&Rs, or "test-and-runs.")  But in general allowing prospects to fully immerse themselves in our software through the free trial inspires confidence in the product and is a very cost-effective way to market our services.  By making sure all employees can see the company's sales data in real-time we encourage data-driven decision-making and help everyone understand how their roles directly impact the company's success. And we give lots of information to investors because, well, we like our investors.

Obviously, there are some limits to how far you can take the transparency ethos. But in the digital age it just doesn't seem to make much sense to try to restrict the amount of information given to customers, employees and investors--its going to come out eventually. In the coming months it is expected that President Obama's administration will make a point of pressing for more transparency in Washington and on Wall Street. Here's hoping landlords are listening too.