Non-technical Skills for Urology Trainees: A Double-Blinded Study of ChatGPT4 AI Benchmarking Against Consultant Interaction

Matthew Pears; Karan Wadhwa; Stephen R Payne; Vishwanath Hanchanale; Mamoun Hamid Elmamoun; Sunjay Jain; Stathis Th Konstantinidis; Mark Rochester; Ruth Doherty; Kenneth Spearpoint; Oliver Ng; Lachlan Dick; Steven Yule; Chandra Shekhar Biyani

doi:10.1007/s41666-024-00180-7

Non-technical Skills for Urology Trainees: A Double-Blinded Study of ChatGPT4 AI Benchmarking Against Consultant Interaction

J Healthc Inform Res. 2024 Nov 14;9(1):103-118. doi: 10.1007/s41666-024-00180-7. eCollection 2025 Mar.

Authors

Affiliations

¹ School of Health Sciences, University of Nottingham, Nottingham, UK.
² Department of Urology, Broomfield Hospital, Chelmsford, UK.
³ Department of Urology, Manchester University Hospitals NHS Foundation Trust, Manchester, UK.
⁴ Department of Urology, Royal Liverpool University Hospital, Liverpool, UK.
⁵ Leeds Teaching Hospitals NHS Trust, Leeds, UK.
⁶ Norfolk and Norwich University Hospitals NHS Foundation Trust, Norwich, UK.
⁷ Centre of Postgraduate Medicine and Public Health, University of Hertfordshire, Hatfield, UK.
⁸ NHS Lothian, University of Edinburgh, Edinburgh, UK.
⁹ Edinburgh Surgical Sabermetrics Group, University of Edinburgh, Edinburgh, UK.

PMID: 39897101
PMCID: PMC11782744 (available on 2026-03-01)
DOI: 10.1007/s41666-024-00180-7

Abstract

Non-technical skills (NTS) are crucial in healthcare, encompassing cognitive and social skills that support technical ability. Traditional NTS training is evolving with the emergence of artificial intelligence (AI) models that can intelligently converse with their users, known as large language models (LLMs). This study investigated the capabilities and limitations of a popular model named generative pre-trained transformer 4 (GPT-4) in NTS training, comparing its performance to that of human evaluators. Urology trainees identified NTS events in simulated scenarios and discussed them in blinded feedback sessions with AI and human consultants. Experts assessed the blinded interaction data, providing quantitative ratings and qualitative evaluations using annotated transcripts. Wilcoxon signed-rank tests compared pre- and post-intervention ratings, whilst Mann-Whitney U tests compared post-intervention ratings between AI and human feedback. Thematic analysis identified strengths, limitations, and differences between AI and human feedback approaches. The AI model demonstrated significant strengths in reinforcing knowledge gathering (p = 0.04), providing accurate and evidence-based feedback (p = 0.013), conveying empathy (p = 0.021), and tailoring explanations to complexity (p = 0.002). However, human feedback excelled in language terminology (p = 0.003), complexity (p = 0.020), and fact-based feedback (p = 0.025). The study highlights the potential for AI to augment assessment of NTS training in healthcare. A blended approach utilising AI and human expertise may boost training efficacy.

Keywords: Artificial intelligence; Healthcare education; Non-technical skills; Pedagogy; Simulation; Urology.

© The Author(s), under exclusive licence to Springer Nature Switzerland AG 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.