The impact of different time limits and test versions on reliability in South Africa

Abstract

The empirically developed English comprehension test (ECT) was created for organisational and educational purposes to assess verbal reasoning. The initial version of the ECT had an associated time limit of 45 min, which required individuals to complete it within the specified time, while the later version of the ECT had no time limit. The ECT’s two test versions – a timed and an untimed version – were piloted as part of the development and validation of the ECT. The purpose of this article was to explore the internal consistency of the two test versions and compare the reliability of the timed and untimed versions of the ECT. This study was conducted to establish whether reliability was affected by the different time limit-related requirements. The sample size for ECT version 1.2 was 597 and ECT version 1.3 comprised 882 individuals. The methods used for comparison in this article involved a graphical display of performance relating to both test versions and an exploration of the times recorded for the untimed test version. A reliability analysis was performed to evaluate the internal consistency of the two test versions. The performance of individuals in the untimed and timed versions of the ECT was similar based on the average minimum and maximum scores. The Cronbach’s alpha indicated that verbal reasoning was measured consistently for the two test versions. This result suggested that time did not negatively affect the reliability of the test.