5 science-backed best practices for cognitive ability testing

Cognitive ability tests are a great tool for identifying top candidates when used appropriately. However, while they certainly are popular and useful, tests of cognitive abilities should be used carefully. Using these tests haphazardly can potentially create unfair disadvantages or biases toward some groups, meaning you may overlook high-quality candidates or open your organization to legal risk.

This blog will walk you through some best practices, based on the latest scientific insights, for cognitive ability testing. Read on to learn how to build a sound hiring process using cognitive ability tests.

The latest science on the effectiveness of cognitive ability tests 

Are the hiring tools that you’re using job-relevant and helping you select high-performers?

This is the question at the center of criterion-validation studies. These studies look at how successful are different hiring tools like structured interviews or cognitive ability tests in predicting who will be successful in the job. 

While a single criterion-validity study will look at one or several hiring tools, a meta-analytic study will summarize multiple studies across different organizations, roles, and hiring contexts to deliver a sound verdict on the usefulness of a certain hiring tool.

A large and influential meta-analysis summarizing hundreds of existing research papers on the topic of the validity of hiring tools was published by Sackett and colleagues in 2022. This study compared some of the most popular hiring tools, such as structured interviews, job knowledge tests, and cognitive abilities tests. 

You can see a summary of the key results of the study in Figure 1. Existing research suggests that structured interviews are the hiring tool with the highest average validity, followed by job knowledge (skills) tests, empirically keyed biodata, and work sample (simulation) tests. 

Sackett and his team published another large meta-analysis in December 2023, this time focusing only on the role of cognitive abilities. They summarized more than 130 studies and concluded that the average validity coefficient of cognitive ability tests is even lower: between .22 and .26. This lowers the rank position of cognitive ability tests down 7th or 8th place (instead of the 5th place suggested by the 2022 study).

Figure 1. An overview of the original and revised validity estimates for different hiring tools

Does this mean cognitive abilities testing is a no-go when hiring? 

Absolutely not. Cognitive ability tests still have a role to play when assessing candidates in the hiring context, but they are not the “silver bullet” of every hiring process as was once considered. 

Recruiters and hiring managers should be mindful when using these tests, applying a nuanced approach to using and interpreting them during the hiring process. Read on to learn some good practices and how to best integrate cognitive ability tests into your hiring process!

5 science-backed best practices for using cognitive abilities tests

1. Consider whether to test cognitive abilities or more role-specific skills

The science is clear that the more similar the hiring assessment is to the actual job, the more effective it will be in identifying good candidates and predicting job success. So, when thinking about including a cognitive ability test into your assessment battery, you should first consider whether there are other more suitable role-specific tests that could help you identify top candidates for the position.

The influential study of Sackett and colleagues (2022) shows that job knowledge tests like TestGorilla’s role-specific skills tests or software skills tests are better (on average) than cognitive abilities tests at predicting overall job performance. Work sample tests are also great at predicting future performance – tests like coding simulations could help you identify a future top performer while assessing skills they will have to use in their everyday work.

Testing cognitive abilities is still a good option in some cases. Cognitive ability tests are known to be good at predicting candidates’ learning capabilities, for example, and estimating their growth potential. This is why they are often considered when hiring for roles that require a lot of on-the-job learning (e.g., fresh graduates hired for a graduate development program) or when recruiting people with little relevant experience.

2. Choose your cognitive ability test wisely

Before using a cognitive ability test as part of the candidate assessment, it’s important to consider what the cognitive requirements and demands of the role are. 

Start by thinking about what everyday work will look like for the job incumbent. Consider the types of tasks they will perform and what it takes to complete them successfully. Do they have to cross-check information and make sure it is correct? Do they have to solve routine problems with relatively straightforward solutions or figure out solutions to complex problems that have no precedent? Or maybe they need to have a good memory? There are many such questions that you can ask to unpack the role requirements. There’s a more in-depth discussion on how to understand what your future hire will do in our blog on job analysis.

As an example, imagine you’re looking to hire a writer and want to include a cognitive ability test as part of the candidate assessment. The first thing to consider is what kind of work will this person be doing:

• Are they going to do a lot of desk research and summarization? If so, then the Reading comprehension test might be a good choice.

• WIll they have to check textual information for consistency and ensure all details are correct? Are they going to write texts based on detailed technical sources? The Attention to detail (textual) test might be a better choice in that situation.

• Will they have to reason critically with differing opinions and attitudes to write objectively? A Critical thinking test is probably the best option here.

There are many different cognitive abilities, and TestGorilla can help you assess 15 different ones. Our ultimate guide to cognitive ability tests provides a deep dive into each of the cognitive abilities you can assess with TestGorilla, offering recommendations on how to choose between them and when to use them.

3. Use cognitive ability tests as part of a multi-measure assessment 

While a single cognitive ability test can provide insight into a candidate’s proficiency in a particular cognitive ability, including several varied, job-relevant tests into a single multi-measure assessment can help you screen for multiple role-relevant skills. 

Job performance is complex and multi-faceted, and you need a good mix of different skills and hiring tools to assess a candidate holistically. This is known as multi-measure assessment. Indeed, research shows that the validity of the hiring process is maximized when different hiring tools are used together to holistically assess a candidate. By assessing multiple skills that are critical for the role, you’re ensuring that the assessment covers several different skills needed for the job. 

When including more than one test in the assessment, be mindful of their complementarity and avoid skill overlap. Including tests that are too similar to one another will yield little additional information about the candidate (often called “incremental validity” in hiring science). 

As an example, both Problem solving and Critical thinking tests measure similar skills in candidates. The inclusion of one of these tests can provide valuable information to predict candidates’ job performance. However, because these tests tap into a similar cognitive ability, adding the second test adds little additional value to your assessment. 

Rather than including two similar cognitive ability tests, more information can be gathered about the candidate's future job performance by combining one of the two cognitive ability tests (e.g., Problem solving) with a role-specific skills test (e.g., Email marketing). 

4. Consider diversity and inclusion 

Diversity, equity, and inclusion (DEI) have become a strategic imperative and asset for many organizations. Research has shown that diverse workplaces are more productive and profitable. Acting as a filter for candidates to enter an organization, the hiring process is key to facilitating and contributing to diversity and inclusion in the workplace. 

Research shows that cognitive ability tests typically result in higher adverse impact than other hiring tools (e.g., personality tests, job knowledge/skills tests), which could translate into a less diverse workforce. As a general rule (and as per legislation in some regions), when there’s an equally or more valid hiring tool available that carries less risk of disadvantaging some groups of candidates (e.g., protected groups or minorities), it is best to use that hiring tool instead. As such, there may be instances where it is fairer and more effective to opt for a different hiring tool (e.g. a role-specific skills test or a simulation test) rather than a cognitive ability test. 

In short, when using cognitive ability tests, it is important to weigh the benefits (e.g., validity) and risks (e.g., adverse impact) associated with using these tests. 

5. Place cognitive ability tests early in the assessment 

Cognitive ability tests demand deep concentration, often requiring the test taker to respond quickly to tough questions under demanding time limits. While this closely mimics the demands of a fast-paced work environment where sound thinking skills need to be paired with quick decision-making, it can also be tiring for the candidates. When tired or distracted, even the best candidates may score poorly on demanding cognitive tasks. 

When including cognitive ability tests in the assessment, it is best to place them at the very beginning. This way, you’re ensuring that candidates are approaching the tests fresh and sharp, allowing them to do their best. Test order can significantly impact candidate performance on the assessment, as well as reliability and validity of assessment results. 

Read our blog about test order for details about how to order tests in an assessment to achieve the highest candidate performance and satisfaction.

Use cognitive ability tests wisely for a hiring process that’s fair and effective

To summarize, cognitive ability tests are an important hiring tool that every recruiter and hiring manager should know how and when to best use. They can help you identify candidates who have great learning potential, and can shine when given the opportunity to develop. Cognitive ability tests can also help you assess candidates that are expected to go through a lot of classroom training or commit to a lot of on-the-job learning. Typically, these are candidates applying for entry-level positions or fresh graduates with very little experience and work knowledge.

Although they are a valuable hiring tool for identifying future top performers, cognitive ability tests also come with the risks of potentially disadvantaging some groups of candidates in the hiring process. Combining (or sometimes even substituting) a cognitive ability test with other role-specific skills and job-relevant tests can help you create an effective and fair hiring process. Building a truly holistic multi-measure assessment of your candidates is a sound way to future-proof your hiring process.

References

Rock, D., & Grant, H. (2019, March 19). Why diverse teams are smarter. Harvard Business Review. https://hbr.org/2016/11/why-diverse-teams-are-smarter 

Sackett, P. R., Demeke, S., Bazian, I. M., Griebie, A. M., Priest, R., & Kuncel, N. R. (2023, December 7). A Contemporary Look at the Relationship Between General Cognitive Ability and Job Performance. Journal of Applied Psychology. Advance online publication. 

Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040.