Science series materials are brought to you by TestGorilla’s team of assessment experts: A group of IO psychology, data science, psychometricians, and IP development specialists with a deep understanding of the science behind skills-based hiring.
TestGorilla takes a deeply scientific and multi-layered approach to creating tests for skills-based hiring. This series of brief introductions aims to conjure transparency when it comes to how we build our tests and how we advise they are used.
In this post, we’ll introduce you to test order and offer some test order best practices that will help you increase the reliability of your assessments.
When we refer to test order, we’re talking about the order in which tests are placed within an assessment – where an assessment is made up of two to five different tests and/or custom questions.
Note that this is specifically how we understand tests and assessments at TestGorilla. (It’s also common to use the terms interchangeably, or to understand a test as a type of assessment where there are correct and incorrect answers.)
When you’re choosing the order of tests in an assessment that you’re creating, it’s important to consider what the candidate is going through. Taking tests is hard – it’s very cognitively demanding and requires a lot of both deep and time-bound thinking, especially when the test is timed.
Because of this, the order in which tests are taken by candidates can have an impact on their assessment-taking experience and overall performance. But it’s difficult for people who don’t have expertise in testing to know how to properly order tests within an assessment.
Without this guidance hiring managers can make suboptimal decisions about test order, resulting in a worse experience for both the candidates and themselves.
Despite test order’s importance, we don’t expect employers, who aren’t experts in testing or assessment psychology, to know how to do it properly. The purpose of this science series article is to support you with test ordering expertise, so that you can improve your test reliability and candidate experience.
Assessment scientists use the term “fatigue effect” to refer to the cognitive demand that makes taking tests hard – the longer an assessment goes on, the more fatigued the candidate will get. Not only is enhanced fatigue a poor experience for the candidate; it can also worsen the psychometric properties of a test. In other words, when the fatigue effect is in play, the data you get about the candidate is noisier and less reliable.
Because of this, there is a broad consensus among assessment scientists in support of two (very simple!) test order best practices:
More cognitively demanding tests should be placed at the start of an assessment, when fatigue effect is low. The impact of this is that candidates can spend their mental energy on the more difficult tests when they are fresh and energized.
Intuitively, less cognitively demanding tests should be placed at the end of an assessment, when fatigue effect is highest. The impact of this is that when candidates begin to get tired as a result of taking tests, they are working on easier tests that require less concentration.
Our Problem-solving test has been used hundreds of thousands of times in assessments created by a multitude of employers. Due to the widely-applicable nature of this test, assessments that include it have been used to hire for a wide range of job roles. Customers can, and do, place this test anywhere from first to fifth in the assessments they create.
The Problem-solving test is a hard test for candidates. It’s challenging and cognitively demanding, and requires that candidates demonstrate substantial problem-solving skills in a short amount of time (10 minutes). In short, it’s exactly the kind of test that, as best practices suggest, should be placed at the start of an assessment.
We gathered some data in order to understand the impact that test order has on the performance of our Problem-solving test – where ‘performance’ is determined by the reliability of the test (where reliability is a statistical measure of how consistent candidate scores are). Below is a summary of our findings.
Note that when considering the reliability of a test for use in hiring, a reliability score of 0.70 is the minimum acceptable standard, with higher values being more desirable.
Where was the Problem-solving test placed within the assessment? | What was the reliability score of the test under this placement? | What percentage of Problem-solving tests appear in this order? |
1st test in assessment | 0.772 | 23.22% |
2nd test in assessment | 0.735 | 24.043% |
3rd test in assessment | 0.726 | 29.458% |
4th test in assessment | 0.718 | 13.204% |
5th test in assessment | 0.703 | 10.074% |
As you can see, when the Problem-solving test is placed first in an assessment, it has a reliability estimate of 0.772. This score sits nicely above the minimum acceptable score of 0.7.
As the test is placed further and further along in the test order of an assessment, its reliability score steadily decreases. By the fifth slot, it barely meets the minimum accepted standard for test reliability. Test order really does make a difference.
Our data also shows that the majority of people using this test place it within the first three tests of an assessment. This shows there is already some intuitive understanding among TestGorilla users that our Problem-solving test should appear early on in an assessment.
Being intentional with test order can help you make the most of your TestGorilla assessments. It’s low-hanging fruit: by switching your test order around slightly, you can improve the candidate experience and the reliability of your test, enabling you to make better hiring decisions.
The best way to develop your understanding of test order is by simply trying out the tests for themselves and seeing what feels hardest.
Place the ones that you find toughest at the start of your assessment, and move the easier ones to the end. You’ll end up with a more reliable assessment, and your candidates will be happy you took that additional step. Why not experiment with some of our tests for free by signing up for a free plan?
You can check out additional blogs about the science behind TestGorilla here:
Why not try TestGorilla for free, and see what happens when you put skills first.
No spam. Unsubscribe at any time.
Our screening tests identify the best candidates and make your hiring decisions faster, easier, and bias-free.
A step-by-step blueprint that will help you maximize the benefits of skills-based hiring from faster time-to-hire to improved employee retention.
With our onboarding email templates, you'll reduce first-day jitters, boost confidence, and create a seamless experience for your new hires.
This handbook provides actionable insights, use cases, data, and tools to help you implement skills-based hiring for optimal success
A comprehensive guide packed with detailed strategies, timelines, and best practices — to help you build a seamless onboarding plan.
This in-depth guide includes tools, metrics, and a step-by-step plan for tracking and boosting your recruitment ROI.
Get all the essentials of HR in one place! This cheat sheet covers KPIs, roles, talent acquisition, compliance, performance management, and more to boost your HR expertise.
Onboarding employees can be a challenge. This checklist provides detailed best practices broken down by days, weeks, and months after joining.
Track all the critical calculations that contribute to your recruitment process and find out how to optimize them with this cheat sheet.