Posted by Wayne Chuen on Wed, Nov 05, 2008 @ 06:31 PM
As a kid, it was fairly well ingrained in me that when taking a test, the goal was to try to achieve the highest possible score. If I received low scores on a test, then it meant I didn't sufficiently understand some concept in algebra or properly comprehend the significance of a novel like Catch-22. In other words, tests served to point out deficiencies, so the higher the test score, the better. However, with pre-employment testing, and personality tests in particular, this is not necessarily true.
One of Criteria's personality tests, the Criteria Personality Inventory (CPI) measures the "Big Five" personality traits - Agreeableness, Conscientiousness, Extraversion, Openness, and Stability. Suppose you gave the CPI to two candidates: Amy and Bob. Amy scored in the 25th percentile on Agreeableness and 33rd percentile on Extraversion while Bob scored in the 90th percentile on Agreeableness and 84th percentile on Extraversion. Who is the better candidate?
With how we're accustomed to thinking about tests, it would be easy to reason that since Bob scored higher, he must have done better on the test, and thus, must be the better candidate. However, this is not how personality tests work. The key is to examine the results in tandem with the job description. If you're testing for a position — such as a Customer Service Representative — that requires frequent and pleasant interactions with others, then Bob may be the best candidate because his results point to a personality that is more cooperative and outgoing. If, on the other hand, you're testing for a position that requires objective problem-solving skills and involves little social interaction — such as data analysts, for example — then Amy may be the best candidate because her results point to a personality that is less dependent on social interaction. Because there are no "high" or "low" scores on personality tests, the CPI should be viewed as an evaluation of candidates' personalities, rather than a test they need to ace.
For this reason we recently redesigned the score report for the CPI. Based on customer feedback, we believe the new score report helps emphasize the point that there are no "high" or "low" scores on the CPI. The most significant difference you will notice is the Results Summary graphs. These graphs show where on the spectrum the candidate scored in each of the "Big Five" personality traits. At a glance, you can see how candidates compare to the general population. The Results Snapshot gives you a quick description of the candidate's personality traits based on the results, and the full text for each trait emphasizes how the score should be interpreted. Finally, we added an Interview Guide where appropriate. These questions were designed to help you further probe a candidate if his/her results don't seem to correspond with the type of personality demanded by a certain job.
We hope you'll find these new score reports more useful and informative. If you have any questions, give us a call or leave a comment below.
Posted by Josh Millet on Thu, Oct 02, 2008 @ 12:21 PM
Just Who Lives in Alaska Anyhow?
A recent article in The Week entitled "Sizing Up the Last
Frontier" notes that "after John McCain tapped Alaska Gov. Sarah Palin to be
his running mate, the nation took a sudden interest in the quirky 49th
state," and asks "How is Alaska different than the lower 48?"
Well, Alaska has always attracted hunters, oil drillers, and
gold miners, and men outnumber women there by a margin of 7 to 6. Alaskans consider themselves to be rugged
individualists, and there's good reason for this: the state covers 656,424
square miles and has only 683,000 inhabitants. More than 60% of Alaskans were
born elsewhere; in other words most Alaskans chose to live in the vast,
beautiful and empty expanse they call home.
What kind of people make this choice? In our last blog post
we commented on the Wall Street Journal article that summarized a recent study on personality with 600,000 internet
volunteers from all 50 states and the District of Columbia. We discussed the state level rankings on
personality traits such as agreeableness, openness and extraversion, and
pointed out that there was a tendency for the states with smaller numbers of
respondents to show up at the higher and lower ends of the rankings.
But even accounting for increased variance in small states,
the data from Alaska stands out from the rest. Alaska scored 49th in
extraversion, 51st in agreeableness, 51st in
conscientiousness, 47th in neuroticism, and 49th in
openness. That's consistency--49th state indeed! It also makes you wonder a bit about how seriously
the 4000 Alaskans were answering the survey.
With respect to agreeableness and conscientiousness, Alaska
didn't just rank last in both, their scores were more than 3 standard deviations lower
than the mean—very few states were more than one standard deviation off. Take a
look at the plot below where we've shown how the states look on agreeableness
and conscientiousness. The red dot is
Alaska.
According to the data, then, Alaskans consider themselves much less
agreeable and conscientious than their compatriots in the lower 48. But
before we take any angry calls from the Alaskan Tourism Board, our point here
is not to highlight Alaska's singularity, but to use Alaska as an example of
how a single outlier can affect the conclusions one draws
from a small sample size. What impact could Alaska have on the national scene
when the personality measures are correlated with health, crime and business
measures? This what statisiticians are refering to when they call something influential.
According to the plot,
Alaska is as far removed from the other states psychologically as it is
geographically. It is showing the same
trend as the rest of the data shows, in that low scores on agreeableness seem
to be associated with low scores on conscientiousness. But Alaska's position on the graph
exaggerates that trend. When you include
Alaska, the correlation between the two personality traits is r = .66. When you remove it, the correlation is r = .56. Like we said, in this case Alaska doesn't
change how you would look at the data (you'd come to the same conclusion that
there is a positive association), but it does affect the strength of the
conclusion.
By contrast, look at a
plot of the national data for conscientiousness plotted against neuroticism. The
trend is that the less conscientious a state is, the more neurotic they appear
to be. However, in this case Alaska (the
red point) bucks the trend. Including
Alaska, the correlation is r = -.27; excluding Alaska the correlation is r = -.38. Here, according to standard tests of
statistical significance, whether or not you include Alaska would make a
difference to the inference.
The take home message is
that the personality data in the research study were matched up with dozens of
potential interesting variables, and in some cases small correlations were
observed. You'd have to look very
closely at the individual plots of the data to determine whether Alaska was
exaggerating or diminishing the final conclusions. It's obvious status as an outlier in these
internet data makes it a potential deal maker or deal breaker in the national
data.
The message as regards pre-employment testing is that small to
medium sized companies should examine data on pre-employment assessments and job
performance carefully. In a smallish
sample, say 50 recent hires, there could be influential outliers that make the
assessment look better or worse than it truly is.
Posted by Eric Loken on Tue, Sep 30, 2008 @ 07:53 PM
The Wall Street Journal last week ran an interesting article that summarizes a study that aggregates personality data on a state by state basis. The research is based on a Five Factor approach to studying
personality (Agreeableness, Conscientiousness, Extraversion, Neuroticism/Stability, and Openness). Criteria's pre-employment testing solution features a personality inventory, the CPI, that is built on the same theoretical principles – and other employment personality tests that we offer also focus on a subset of these traits. Armed with over 600,000 internet surveys, the
researchers showed that on average, respondents from North Dakota were the most
agreeable in the country, and those in Alaska – by a huge margin – were the
least agreeable. Alaska also ranked near the bottom on the other four
personality dimensions too — but we'll have more on that in another blog post. But before we jump to any conclusions, let's stop to
consider why small states seem to pop up a lot near the top and bottom of these
lists.
A lot of it has to do with a simple principle of statistics
that as sample size increases, variability due to measurement error tends to
decrease. This is why it is safer to be
in a mutual fund than in a single stock. Although the upside of the single stock is greater, so too is the
downside, and risk management suggests you should invest more broadly to limit
the variance.
So how would this affect 600,000 surveys? Well, when those
surveys are divided across 50 states, very unequal sample sizes occur across
the board. Those agreeable folks from North
Dakota submitted 2,372 surveys, while Californians submitted 71,873. If you were doing sophisticated polling for
an election survey, either sample size should get you a good estimate of
people's preferences. But when you're
assessing something trickier to measure, like personality traits, and when the
samples are either self-selected participants who stumble across the website,
or students told to go to the website for course credit, there can be
considerable variation due to measurement error. This will tend to diminish as the size of the
sample increases.
A graph can save a thousand words. We've plotted the absolute value of the
scores on the agreeableness scale against the population size. You can see the clear descending
triangle. The chances that a state has
an extreme score on agreeableness (either high or low) goes down as the sample
size goes up.
We don't mean to deny that there are interesting patterns in
the personality data reported in the study. On the contrary, we're quite intrigued and expect to make a few more
blog posts on the topic. We're just
reminding people that when the 50 states are ranked from highest to lowest on
anything that has a lot of measurement error, states with smaller samples will
be overrepresented in the top and bottom parts of the list.
None of this should take away from the achievement of North
Dakota in being ranked most agreeable in the country. It's just that if there had been two million
surveys instead of two thousand it would have been even more impressive. Of
course two million surveys from North Dakota would have been an even more serious
problem…
Posted by Josh Millet on Tue, May 27, 2008 @ 12:53 PM
Following up on the discussion I started last time about the Big Five personality traits, I want to provide a little more context on the Big Five and how they relate to the field of personality testing as a whole. The Big Five are personality dimensions that describe the ways in which an individual reacts to other people and to the world around them. For example, the Extraversion/Introversion dimension describes the extent to which an individual is more or less outgoing, gregarious and in need of social stimulation. If a personality test determines that an individual is in the 65th percentile for "Extraversion," this means that the individual is more extraverted than 65 percent of the individuals in the norm group.
The notion of personality "traits" is now fairly widely accepted, and is superceding an older paradigm of personality "types" that originated with Carl Jung and relied on a view of personality that grouped people into one of two distinct types, such as introvert or extravert, thinker or feeler. The traits model is gaining credence in personality research because of growing evidence that suggests that a strict dichotomy between two distinct types does not sufficiently describe the nuances in the extent to which individuals tend to one side or the other.
The best known example of a test based on the older model is the Myers-Briggs Type Indicator (MBTI). Since the MBTI is probably the most widely known and thoroughly studied personality test today, and since we get asked about it all the time, I thought I'd offer some thoughts on it. Or one thought, to be exact. Do not use the MBTI to make hiring decisions! I repeat, the MBTI should not be used for the purpose of employee selection...ever. I say this because the MBTI, which has a large and enthusiatic following, is often used in just this way, even though it shouldn't be. There are many reasons the MBTI should never be used to inform hiring decisions, many of which are described here. But the most important is simply that there's no convincing evidence to link MBTI results to job performance. In order to ward off the anticipated deluge of angry emails from MBTI-devotees, I would just say that if you don't believe me, take it from the MBTI's publisher. Even they do not suggest it should be used for employee selection...they provide a table that lists every conceivable use for a test, but note the complete lack of check marks in the "Selection" column.
There's plenty of evidence, on the other hand, to link the Big Five Traits to job performance for a variety of positions. Conscientiousness, which measures the extent to which an individual is reliable, organized, persistent, and responsible (those who score low in Conscientiousness may be more impulsive and at times unreliable) has been shown to be moderately predictive of success across many job types, but particularly for entry-level positions where characteristics like reliability and punctuality may be more valuable than creativity. Certain Big Five traits are useful for certain types of jobs; for example, extraverts perform better in sales than do introverts, and highly agreeable people are well-suited for customer service but might not make good judges or CEOs, because those jobs require objective decison-making that highly agreeable people may not be comfortable with. Other Big Five traits are much less relevant to employee selection: for example, there isn't much evidence that Openness (the extent to which an individual is imaginative and creative, rather than down to earth and conventional) is predictive of work success, even though it seems logical that people with high Openness scores would be better suited for jobs that require imagination, creativity or abstract thinking.
Alright, that's enough for now. Next time I'll finish up with this thread by discussing ways in which some employment personality tests move beyond the Big Five by measuring more fine-grained traits that have been shown to predict success for specific jobs.
Posted by Josh Millet on Wed, May 14, 2008 @ 04:34 PM
A recent report summarized here suggests that personality testing is the fastest growing segment of the pre-employment testing market. The survey of HR professionals revealed that the percentage of respondents whose firms used personality tests has grown from 21% to 59% in the last five years alone. Unfortunately, there are still a lot of misconceptions about what personality tests are, and how they should be used. Since we get so many questions about how personality tests work, from both HR professionals and job candidates, I thought I'd try to explain some of the basics.
Employment personality tests are designed to measure personality traits that may be related to job performance. Most personality tests consist of a series of self-evaluative "prompts" and ask a test-taker to indicate the extent to which they agree or disagree with the statement. An example of a prompt might be "Meeting new people is enjoyable to me." There are no right or wrong answers to these questions, as the responses can be used to indicate behavioral tendencies that may or may not fit a particular job. For example, the prompt above might be one of many used to calculate an individual's "Extraversion" rating. A high score in Extraversion is not necessarily better than a low one, but an extraverted individual might be better suited for some jobs and less well-suited for others. A classic example is that highly extraverted individuals tend to perform better in sales jobs because not only are they more comfortable dealing with people, they are also better at initiating interactions. On the other hand, highly extraverted people might be less suited for jobs that require very little or even no social contact. They could potentially become bored or frustrated without social stimulation.
Professionally developed and validated employment personality tests attempt to measure constant, fixed "traits" thought to influence an individual's behavioral tendencies across a variety of contexts. The use of personality tests by companies during the hiring process has grown rapidly in recent years, with the dominant framework being the "Big Five." The Big Five are five dimensions of personality that seem to emerge consistently in empirical research: Agreeableness, Conscientiousness, Extraversion, Openness (to Experience) and Stability.
How are the Big Five used to help companies with their hiring process? I'll get into more details on the applied use of personality tests in my next post.