Interrater reliability and agreement of the NEUMOBACT checklist about infection prevention performance of intensive care nurses in simulation-based scenarios

Abstract

Objective: To analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible. Methods: A validation study with a cross-sectional design, compliant with the GRRAS checklist,among ICU nurses attending a SIMULAZERO course with an Objective Structured Clinical Evaluation simulation format, to verify transfer from theory to clinical practice of knowledge and skills in ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB) prevention. A minimum sample size of 111 pairs of nurse raters was calculated. Interrater agreement was analysed using Gwet’s AC1 for each item and as a total for each of thethree checklists in the NEUMOBACT instrument. Results: A total of 95 pairs of valid NEUMOBACT checklists were completed by 190 raters with amedian age of 29 [25–35] years, 93.7% were female. At the Central Venous Catheter insertion station, Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). Only 2 of the 17 items scored below 0.9. At the Endotracheal Suctioning station, Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items that made up this station, 16 had an agreement percentage above0.9, a further 9 were between 0.821 and 0.884, and item 13 had a value of 0.789. At the Patient Care station, Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items, 17 showed an agreement percentage above 0.9 and 4 were between 0.810 and 0.894. Conclusions: The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses.