Differential item functioning and cut-off scores: Implications for test score interpretation

Abstract

Psychological and educational measurement instruments are used to make decisions that can have an important impact on the person being assessed. It is therefore essential to ensure that tests are free from bias so that the scores they yield provide a fair interpretation. This study aimed to assess the impact that items showing differential functioning may have on test interpretations based on cut-off scores. To this end a simulation study was conductedin which we manipulated the size of the comparison groups (100, 250,500 and 1000), the magnitude of differential item functioning (DIF) (set at 0.8 for the difference between the difficulty parameters of the two groups) and the degree of test contamination (0%, 10%, 20%, 30% and 40% of items with differentialfunctioning). Overall, the simulation considered 20 conditions, 1000 replications and a 20-item test. Results indicated that the selected cut-off did have an influence, and as the degree of test contamination increased, greater differences between the groups were erroneously detected, both in terms of statistical test significance and effect size obtained. These findings highlight the importance of ensuring that measurement instruments are free from DIF so that the interpretation of scores is both accurate and fair, this being a key aspect of a test's validity.