Dual stream transformer for medication state classification in Parkinson's disease patients using facial videos

Vasileios Skaramagkas; Iro Boura; Georgios Karamanis; Ioannis Kyprakis; Dimitrios I Fotiadis; Zinovia Kefalopoulou; Cleanthe Spanaki; Manolis Tsiknakis

doi:10.1038/s41746-025-01630-1

Dual stream transformer for medication state classification in Parkinson's disease patients using facial videos

NPJ Digit Med. 2025 Apr 26;8(1):226. doi: 10.1038/s41746-025-01630-1.

Authors

Vasileios Skaramagkas^{1

2}, Iro Boura^{3

4}, Georgios Karamanis⁵, Ioannis Kyprakis^{6

7}, Dimitrios I Fotiadis^{8

9}, Zinovia Kefalopoulou^#¹⁰, Cleanthe Spanaki^#^{11

12}, Manolis Tsiknakis^#^{13

14}

Affiliations

¹ Electrical and Computer Engineering, Hellenic Mediterranean University, Estavromenos, Heraklion, GR-710 04, Greece. vskaramagkas96@gmail.com.
² Institute of Computer Science, Foundation for Research and Technology Hellas (FORTH), 100 Nikolaou Plastira, Heraklion, GR-700 03, Greece. vskaramagkas96@gmail.com.
³ School of Medicine, University of Crete, A. Kalokerinou 13, Heraklion, GR-715 00, Greece.
⁴ Department of Neurology, University Hospital of Heraklion, Vassilika Vouton, Heraklion, GR-711 00, Greece.
⁵ Department of Neurology, Patras University Hospital, Rion, Patras, GR-264 04, Greece.
⁶ Electrical and Computer Engineering, Hellenic Mediterranean University, Estavromenos, Heraklion, GR-710 04, Greece.
⁷ Institute of Computer Science, Foundation for Research and Technology Hellas (FORTH), 100 Nikolaou Plastira, Heraklion, GR-700 03, Greece.
⁸ Unit of Medical Technology and Intelligent Information Systems, University of Ioannina, Ioannina, GR-451 10, Greece.
⁹ Biomedical Research Institute, Foundation for Research and Technology Hellas (FORTH), Ioannina, GR-451 10, Greece.
¹⁰ Department of Neurology, Patras University Hospital, Rion, Patras, GR-264 04, Greece. zkefalopoulou@gmail.com.
¹¹ School of Medicine, University of Crete, A. Kalokerinou 13, Heraklion, GR-715 00, Greece. kliospanaki@gmail.com.
¹² Department of Neurology, University Hospital of Heraklion, Vassilika Vouton, Heraklion, GR-711 00, Greece. kliospanaki@gmail.com.
¹³ Electrical and Computer Engineering, Hellenic Mediterranean University, Estavromenos, Heraklion, GR-710 04, Greece. tsiknaki@ics.forth.gr.
¹⁴ Institute of Computer Science, Foundation for Research and Technology Hellas (FORTH), 100 Nikolaou Plastira, Heraklion, GR-700 03, Greece. tsiknaki@ics.forth.gr.

^# Contributed equally.

Abstract

Hypomimia is a prominent, levodopa-responsive symptom in Parkinson's disease (PD). In our study, we aimed to distinguish ON and OFF dopaminergic medication state in a cohort of PD patients, analyzing their facial videos with a unique, interpretable Dual Stream Transformer model. Our approach integrated two streams of data: facial frame features and optical flow, processed through a transformer-based architecture. Various configurations of embedding dimensions, dense layer sizes, and attention heads were examined to enhance model performance. The final model, trained on 183 PD patients, attained an accuracy of 86% in differentiating between ON- and OFF-medication state. Moreover, uniform classification performance (up to 88%) was obtained across various stages of PD severity, as expressed by the Hoehn and Yahr (H&Y) scale. These values highlight the potential of our model as a non-invasive, cost-effective instrument for clinicians to remotely and accurately detect patients' response to treatment from early to more advanced PD stages.