Clerkship Assessment Reliability Based on Assessor Type and Assessor-Student Contact Frequency: A Generalizability Theory Analysis

Christopher D Mattson; Celia L O'Brien; Yoon Soo Park; Sandra M Sanguino; Mary E McBride; Brigid M Dolan

doi:10.1097/ACM.0000000000006143

Clerkship Assessment Reliability Based on Assessor Type and Assessor-Student Contact Frequency: A Generalizability Theory Analysis

Acad Med. 2025 Jun 23. doi: 10.1097/ACM.0000000000006143. Online ahead of print.

Authors

Christopher D Mattson, Celia L O'Brien, Yoon Soo Park, Sandra M Sanguino, Mary E McBride, Brigid M Dolan

PMID: 40550141
DOI: 10.1097/ACM.0000000000006143

Abstract

Purpose: Workplace-based assessments (WBAs), such as clinical performance assessments (CPAs), often comprise a significant portion of clerkship assessment systems. This study explores the reliability of CPAs in clerkships based on assessor characteristics, including assessor type and assessor-student contact frequency.

Method: The CPAs of third-year medical students at Northwestern University Feinberg School of Medicine completed during 5 clerkships in academic year 2021 to 2022 were compiled. Analyses compared clerkship assessments per student and breakdown of assessor characteristics by clerkship. Generalizability analysis divided assessments by clerkship, assessor type (resident vs attending), and assessor-student contact frequency (daily vs not daily). Decision studies examined the number of assessments needed per student to achieve reliability of 0.7 (D0.7).

Results: A total of 4,062 CPAs completed by 734 assessors on 179 students were analyzed. The number of attending versus resident assessors and daily versus not daily assessors varied by clerkship. Assessments completed by residents were more reliable than assessments completed by attendings (resident φ coefficient = 0.62; student variance, 8.5%; D0.7 = 19; attending φ coefficient = 0.48; student variance, 5.6%; D0.7 = 34). Assessments completed by assessors with daily contact were more reliable than assessments completed by assessors with not daily contact (daily contact φ coefficient of = 0.63; student variance, 7.6%; D0.7 = 22; not daily contact φ coefficient = 0.31; student variance, 5.7%; D0.7 = 34). Residents with daily contact were most reliable (φ coefficient = 0.38, D0.7 = 16).

Conclusions: Differences in assessor type and assessor-student contact frequency may explain between-clerkship differences in assessment reliability. These findings provide an opportunity to consider assessor characteristics in the design of assessment systems, thus improving assessment reliability and working toward systems that can be trusted by all stakeholders.