Criteria's Employee Testing Blog

How Much Testing is OK?

In the social media age, when a company’s employment brand is more important than ever, it’s a great idea for companies to always keep the issue of candidate perception in mind when implementing pre-employment testing. Given that the trend is increasingly towards testing early in the hiring process — as we discussed here and here — it’s important to consider the question of how much testing is appropriate when the tests are one of the first points of contact a candidate may have with your organization.

We analyzed a lot of data (about half a million tests) to help answer this question. As the graph below makes clear, candidates complete the tests much less frequently when the length of the test exceeds 40 minutes.

The completion rates for batteries less than 40 minutes in length always exceed 75%. If this seems low, consider that many candidates encounter the test through a link in a job posting, and may simply close the test window after deciding they don’t have the time, ability, or inclination to take the tests. If candidates won’t spend 20-30 minutes applying for a job, chances are they weren’t serious about working for your organization in the first place—we call this group “resume spammers.” Interestingly, this 75%+ completion rate is no different for a very short (less than ten minute) test than it is for a 30-40 minute test.

However, in cases where candidates are asked to take a test battery that is longer than 40 minutes, the completion rates are significantly lower: 66% for 41-60 minute tests, and 60% for tests lasting longer than an hour. It seems that the point at which “test fatigue” begins to discourage candidates can be pinpointed: it’s after 40 minutes. This is why we recommend that our customers keep test batteries under 40 minutes in length whenever possible. This is especially true for remote testing done early in the hiring process. It is impossible to know from the data, although it seems highly likely, that candidates will have higher completion rates for tests given on site, for several reasons—the main one being that if candidates perceive themselves to be under serious consideration for a job, they are much happier to spend a long time being assessed.

Leave a comment

The Secret to Google’s Hiring Revealed: Cognitive Ability

Last summer we reacted to an interview with Laszlo Bock at Google who seemed to say that tests scores and grades were useless predictors for hiring decisions. We said that what constitutes information for hiring purposes at Google may well differ from what constitutes information for hiring elsewhere, and we pointed out that validating a selection tool after it has been used, and only for those who were selected will typically yield lower estimates of the usefulness of that tool.

This week, in a widely read New York Times column, we get a more elaborated answer about Google’s hiring goals. What do they look for? Number 1, says Mr. Bock, is cognitive ability. Although Bock is quick to distinguish this from IQ — he sees it as demonstrating an ability to learn quickly — the fluid intelligence he’s trying to evaluate likely correlates well with traditional clinical, academic, and business oriented measures of cognitive ability. Bock is also looking for leadership and a sense of responsibility.

In short, Google is largely looking for the same things that organizational psychologists have been telling us for decades predict job performance — cognitive ability and personality. Measures of conscientiousness are often the second best predictor of job success (after cognitive ability). Other preferred aspects of personality will depend on the nature of the work and the workplace.

For any given selection process, those making decisions want predictive information. What constitutes predictive information will vary from setting to setting. For a company like Google, the composition of the applicant pool and the nature of the workplace might mean that certain traditional sources of information are less useful, and Google has the resources to invent a new, tailored interview process to gather new information. However, the underlying constructs they are looking at — cognitive ability and conscientiousness — are ones that pre-employment assessments have been highlighting for some time. Every organization must also deal with its own costs — what are the consequences of hiring the wrong person? What are the consequences of failing to hire a qualified person? We mostly think about the cost of hiring the wrong person (false positives), but there is also a cost to missing a diamond. Facebook paid $19 billion to buy what Brian Acton built (WhatsApp) 4 years after they didn’t hire him.
But even if they go about it differently, all companies are trying to maximize the information they have about the cognitive ability and character of the people they hire.

Tagged , , Leave a comment

How Can I Trust Test Scores From Remote Tests?

One of the most prominent trends we’re seeing with our customer base is the move to administer more tests remotely, towards the very beginning of the hiring process. As I’ve talked about previously, there are many advantages to administering tests offsite, mostly to do with the cost and time savings of using objective, reliable data to help you filter out unqualified applicants from large applicant pools. But one question that customers often raise is, “How can I trust test results from remote tests, when I can’t verify the identity of the applicant?” Well, unless you’re using a remote proctoring service (either human or web-cam based) you can’t be 100% sure a candidate took the test without outside help. But what we’ve found in working with customers is that if your messaging to candidates is well constructed, and you retest candidates onsite, you don’t need to worry about the reliability of offsite test results.

We recently did a study with one of our largest customers, who administers aptitude tests remotely at the front of their hiring process, and retests candidates later onsite, if they make it that far. What we found by looking at the data is that the likely percentage of people who didn’t take the test honestly (i.e., without outside help) offsite is actually very small, likely much less than 2% of the applicant pool. One reason this may have been the case is that when the employer sent candidates the invitation to take the test, it explicitly described its retesting policy. We really recommend employers do this, because it cuts down on any incentive applicants may have to cheat in the first place. If candidates will have to take a different version of the test onsite anyhow, they are only wasting their own time if they don’t take the test on their own the first time.

So a couple of important takeaways. Offsite cheating is not really a big problem, at least according to our data; and you can cut down on it further through proper messaging and by retesting onsite, with those candidates who make it that far.

Tagged , , , Leave a comment

What Are Integrity Tests?

Today’s topic is integrity tests; what are they and how should they be used?

Well, the term integrity tests is actually used to describe a couple of different types of tests, but they all help employers manage risk by assessing the likelihood that an applicant will be a reliable employee who will follow the rules. So as I said, there are two main types of integrity tests: covert (personality-based) tests that measure traits linked to rule adherence; and overt tests which assess an applicant’s attitudes towards various counterproductive work behaviors (CWBs) directly. Some integrity tests, like our Workplace Productivity Profile (WPP), combine elements of both to predict employee reliability.

What most integrity tests focus on is an applicant’s tendencies and attitudes with respect to rule adherence. Because of this, the tests can be used to predict behavior with respect to a wide variety of counterproductive work behaviors that employers want to avoid, ranging from tardiness, absenteeism, and time-wasting, to theft, fraud, drug use, and safety violations. Integrity tests are generally most widely used and are most effective for entry-level positions for which overall reliability and rule-following is particularly important. Some of the more common uses of integrity tests in specific situations are:

  1. To reduce risk of employee theft, for example in retail sales
  2. In positions where employees will be working in customers’ homes, such as home health care aides and field service technicians
  3. In manufacturing settings to assess risk for safety violations

In all of these cases, integrity tests can serve as a risk management measure, as they will determine that certain applicants represent a higher risk of engaging in these behaviors based on their responses and personality profiles. Often, employers will use background checks pre-hire for these positions, but background checks are expensive and only target people who have committed crimes in the past. By using integrity tests early in the hiring process, employers can save time and costs and help to minimize risk with respect to avoiding workplace behaviors that damage their organization.

Tagged , , Leave a comment

Why Do Organizations Use Pre-Employment Tests?

As an employer trying to make informed hiring decisions, you obviously want to gather as much relevant information on candidates as possible. The problem is that traditional methods of getting to know candidates — resumes and interviews — often don’t yield much real insight. Resumes are notoriously unreliable, and interviews — especially unstructured ones — are subjective and poor predictors of job performance. On the other hand, properly developed (i.e., well validated) tests are a reliable and objective means of gathering job-related information on candidates. Research shows that cognitive aptitude tests, for example, are twice as predictive of job performance as interviews, and three times as effective as resumes.

So companies use pre-employment tests because they are an efficient, objective way to gather data that predict employee performance. They use this data to make better informed, more defensible hiring decisions. Tests are really about giving you better raw materials out of which to build your talent management process: getting better data on candidates will lead to better talent decisions.

So what are the results of all this in terms of tangible business impact? Well when we first speak with companies, HR professionals, and business owners, their stories are very different, but we often see two common pain points: organizations are spending way more time than they’d like on hiring, and yet despite this, they are still making more hiring mistakes than they can afford. Pre-employment tests can help directly with both of these: by dramatically reducing the time you spend reading resumes and doing interviews, pre-employment tests will help you reduce your time to hire and costs associated with hiring. And by giving you reliable, objective data that predict job performance, tests should help you increase your quality of hire and reduce the number of bad hires you make. The ultimate effect of this will be bottom line improvements like increased workforce productivity and reduced turnover.

Tagged , , , Leave a comment

What is Pre-Employment Testing?

Today I’m going to address the most basic question we get about pre-employment testing: What is it? What are pre-employment tests?

Pre-employment tests are an objective, standardized way of gathering data on candidates during your hiring process. All professionally developed, well-validated pre-employment tests have one thing in common: they are an efficient and reliable means of gaining insights into the capabilities and traits of prospective employees. There are many different types of tests, but they generally fall into three categories: aptitude, personality and skills tests.

Aptitude tests measure critical thinking, problem solving, and the ability to learn, digest and apply new information. Since critical thinking skills are vital to so many jobs, aptitude tests can be used in almost any occupational context, but they are especially useful for mid and higher-level jobs. Because critical thinking and problem solving are crucial in so many jobs, it’s not surprising that research shows pretty unanimously that aptitude tests are the single best predictor of job performance.

Personality tests measure behavioral traits. So unlike with aptitude tests, there are no right or wrong answers on personality tests. They measure traits such as extraversion, openness, conscientiousness, and agreeableness. Personality tests can be valuable indicators of job fit, can help you determine if a person’s behavioral tendencies are a good match from a job fit and culture fit perspective. They help answer questions such as: Will someone be comfortable in this position? Do they have behavioral traits that are linked to success in this role?

Skills tests measure job-related competencies; broad ones like verbal, math and communication skills, or narrow ones like typing and computer skills.

To maximize the effectiveness of pre-employment testing, it is a good idea to use more than one type of test. For example, it’s very common to test aptitude and personality, or skills and personality.

By using tests as an objective, reliable source of job-related data, you can streamline your hiring process and make more informed hiring decisions.

Leave a comment

Can a Candidate Be Too Qualified? Is Using a Max Score Cutoff Ever a Good Idea?

Last week we talked about using minimum cutoff scores to improve quality of hire and reduce time spent on hiring by allowing employers to quickly filter out applicants who lack the necessary capabilities for the job. Today we’re going to talk about a much less common and more controversial practice of using maximum score cutoffs. We generally do NOT recommend using maximum cutoff scores. That is, we do NOT recommend excluding someone because she scored too high on an aptitude test for example. The reason we don’t recommend this is that the science is just not clear enough on the benefits.

Why would anyone consider excluding someone for being too smart? The theory goes like this. Some testing companies believe that scoring above the expected range on an aptitude test can be an indicator that a person will be bored by a particular job and want to move on, and that higher aptitude people will also have more opportunities to find other jobs than will lower aptitude employees. Essentially, the idea is that whereas low scores signify that candidates are a risk for involuntary turnover — because they may not be trainable or able to perform well — extremely high scores can be an indicator of risk for voluntary turnover. But the evidence that overqualified employees represent a greater flight risk is not very strong, and in fact one recent study refutes it pretty convincingly. So we don’t generally recommend using max cutoff scores. However, a study we did for one of our larger customers showed that higher scoring candidates were more likely to leave, so if you can find similar trends at your organization, and if voluntary turnover is a big problem for your company or industry, it may be something to consider.

Tagged , , Leave a comment

Cutoff Scores for Pre-Employment Tests

Customers frequently ask us, “Should I use minimum cutoff scores, and if so, how should I set them?” If you have large applicant pools, using a hard cutoff score is a great way to efficiently filter your candidates, and to screen out those who lack the basic aptitude or skills required for the job.

Ok, so how can you determine where to set your minimum cutoff score? The truth is, setting cutoff scores is part art and part science; there are a number of different factors to consider. One good way to get started is to test a sample of your incumbents in a given position, and then use those results to determine cutoffs for your applicants. Our testing software, HireSelect, contains tools that will generate suggested cutoff scores based on data samples you provide. HireSelect also contains suggested minimum scores for a variety of different positions, based on large samples of test results in our database. Using our national norms and suggested minimum scores is especially useful if your company doesn’t want to test incumbents before getting started, or you don’t have large enough samples of existing employees for a given job.

Whether you use our suggested scores or customize a score range based on your own data, there’s a common misconception about how cutoff scores work, so let’s clear that up first. A lot of people believe that there is some “magic number,” where anyone who scores above it will be a good fit, and no one who scores below it is capable of doing the job. However you set your cutoff scores, this is not how they work.

Let’s look at the example of using a cognitive aptitude test, like our CCAT, to hire salespeople for an organization. The higher you set the cutoff, the more likely people above the cutoff will have the critical thinking and problem-solving skills necessary to perform well in the job. So if your only concern is maximizing your hiring accuracy rate, then you’d want a high cutoff score. However, you usually don’t want to set it too high, because doing so will eliminate many capable applicants and run the risk of filtering out too many qualified people. For hiring managers and recruiters who hire large numbers of people, this would be frustrating and counterproductive. This is why I said earlier that setting cutoff scores is part art and part science. To determine where the cutoff should be, you need to take into account the specific dynamics of your hiring process, such as the size of your applicant pools, your applicant-to-hire ratio, and other factors.

The takeaway here is simple. Using a minimum cutoff score can help you minimize the risk of bad hires; the higher the cut off score used, the lower the risk of a bad hire. We help our clients determine the score ranges that make the most sense based on test results they provide and the specifics of their hiring process.

Tagged , , Leave a comment

Legal Compliance in Pre-Employment Testing

Today I want to talk about the important topic of legal compliance with respect to pre-employment testing. One thing should be made clear upfront. There is a persistent myth that utilizing tests as part of the hiring process increases a company’s legal exposure or somehow entails added legal risk. If you are talking about professionally developed, well validated tests, the opposite is in fact true. The same federal laws that apply to the use of pre-employment tests actually apply to any selection methodologies that you use. There is no extra body of laws that applies only to testing; when using tests you’re bound by the same laws that govern conducting interviews with candidates, checking references, etc. That’s why we believe there are two important reasons that the proper use of testing as a selection tool actually reduces the likelihood of being sued:

First, testing makes the selection process more objective, and fairer for all candidates. Tests are less subjective than interviews, where the personal biases of interviewers are much more likely to lead to discrimination claims. In fact, a recently published study shows that companies are over three times more likely to be sued because of interviews than for their use of aptitude, personality or skills tests. Tests also help you minimize the risk of hiring problem employees, who can often pose a far greater liability risk than tests.

Second, the EEOC stipulates that companies use best reasonable efforts to remove biases from their hiring processes. Using validated tests that do not discriminate according to age, sex, and race, can reduce subjective biases, and inject an objective, reliable data point into the hiring process. This helps with EEOC compliance.

OK, so we’ve cleared up that testing, when done properly, will actually reduce rather than increase your legal exposure related to your hiring practices. Now let’s look at the single most important concept when thinking about legal compliance and pre-employment testing. If you pay attention to this one area, you can be confident that you’re well on the way to a legally defensible testing system. That one thing is the rule of job relatedness. The EEOC’s guidelines surrounding the use of tests are the same as their rules for any selection procedures; the tests must measure job-related skills and abilities. So test selection is critical to legal defensibility. No matter how valid a test is, it won’t be legally defensible if you are using it in an invalid way. For example, a bank teller’s daily work requirements might include adding and subtracting numbers, looking up customer activity, and filling out paperwork. Therefore, using an employment test that measure’s an applicant’s fluency with basic mathematical concepts and ability to read and write would clearly be testing job-related capabilities. On the other hand, using a typing test for a forklift driver who won’t be required to type, is NOT a job-related selection measure. One of the best ways to ensure proper test selection is to do a job requirements analysis that pinpoints the skills and abilities that a job requires. Our HireSelect software has a Job Profiler tool that contains over 1000 such job requirements analyses conducted by the U.S. Department of Labor.

So using well validated tests and making sure the tests you use, as well as all your selection processes, are job-related, is the best way to ensure legal compliance.

Tagged , , Leave a comment

Pre-Employment Testing: What to Expect

When first speaking with customers, we try to establish realistic goals and expectations for a pre-employment testing program. So what kind of results should you expect from using pre-employment tests? By using professionally developed, validated testing instruments you are adding objective, data-driven metrics to your hiring process. Using tests should drive incremental improvements in your hiring results, and minimize the risk of bad hires. It should also dramatically streamline your hiring process, and translate into demonstrable improvements in your business by reducing turnover, lowering hiring and training costs, and improving productivity.

OK, but it’s just as important to be realistic and realize what you should not expect from pre-employment testing. Tests are not a crystal ball, unfortunately. Anyone who tells you otherwise is lying. When you see testing companies say things like, “Never make a bad hire again” or advertise 99.9% accuracy, it means they are either ignorant of how the science behind testing works, or misrepresenting it to sell you their tests — either way run, don’t walk, in the opposite direction. Incorporating tests into your hiring process does not mean you’ll never make another bad hire, only that you will make fewer of them. No test is a perfect predictor. Some people who don’t test well may be great employees, and some that test well may be terrible employees. Research shows that tests are significantly more accurate and reliable as predictors than resumes or interviews, but there is no selection methodology that will be 100% accurate in predicting performance.

That’s why we always recommend that tests should be only one element of a comprehensive set of criteria you use to evaluate applicants. Sure, you can use tests at the top of the process to screen out candidates who aren’t a good fit, but ultimately, organizations that use tests are making their final decisions based on many factors, of which tests should be an important component. A good way to look at is this — no matter how well developed a test is, you should not try to outsource your hiring decisions to a testing company. Tests will help you make better talent decisions, but they are not the only factor in those decisions. You should expect tests to streamline and improve your hiring process, not replace it.

Tagged Leave a comment