Interactive Video Acquisition and Learning System for Motor Assessment of Parkinson's Disease

1 Introduction

Parkinson's Disease (PD) is a prevalent neurodegenerative disease that impairs motor functions. The diagnosis and treatment of PD heavily rely on regular motor examinations at clinics. It is cost- and time-consuming to patients, while the routine evaluation processes are repetitive and timeconsuming to physicians.

Telemedicine may provide a promising solution for more accessible movement assessments. It overcomes geographical barriers and links medical providers to patients online [Wootton, 2001; Perednia and Allen, 1995]. Mobile app is an affordable and efficient approach for telemedicine. Symptoms of PD have been assessed on smartphones by accelerometer [Zhan et al., 2018], by self-rating[Bot et al., 2016] and by screen drawing task [Kuosmanen et al., 2020; Tian et al., 2019]. Advanced machine learning techniques could facilitate primary diagnoses and greatly reduce clinician's workload, such as automatic speech performance rating [Nilashi et al., 2018; Tsanas, 2012].

Most of the clinical motor assessment tasks from the Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) Part III [Goetz et al., 2008] could be video captured with the help of an interactive system in the absence of physicians. However, out-of-clinic recording of MDS-UPDRS tests has two main challenges: 1) incorrect movements without professional guidance (due to misunderstanding); 2) poor video quality (dim light or missing body parts from view). Meanwhile, no MDS-UPDRS video datasets are publicly available to promote real applicable machine learning algorithms for automatic diagnosis of PD. A scalable video acquisition system could significantly boost data collection, benefiting both research and clinical practice. To the best of our knowledge, interactively capturing standard MDS-UPDRS videos using smartphone is still a missing part. A specialized mobile app for high quality remote video collection and analysis is in great need for telemedicine of Parkinson's disease.

We developed PD-GUIDER, an interactive video acquisition and learning system for in-home motor assessments of PD 1 . It is the first mobile app for movement recording of standard MDS-UPDRS motor assessments. Our AI engine provides real-time guidance to patients to guarantee correct movement and efficient recording. We tested the effectiveness of the system on healthy subjects and evaluated its performance on patients with Parkinson's disease. High quality videos of standard movement tasks can be easily recorded by patient using PD-GUIDER. We present an efficient telemedicine system equiped with AI engine for motor assessment of Parkinson's disease.

2 SystemArchitecture

The workflow of PD-GUIDER is illustrated in Figure 1. To ensure correct movements, the app shows video demonstrations before each assessment and provides audio instructions. Interactive calibration with audio guidance and onscreen bounding boxes indicating body positions is integrated in the system. Body-keypoint-based auto-recording, oneclick-upload and selfie mode (for individual recording without help, see Fig 2) features further simplify user operations. Hand keypoint detection algorithm provides primary analysis such as finger tap counting. In this way, video collection for motor examination for patients with PD and preliminary motor function analysis are streamlined and standardized using PD-GUIDER.

1 https://youtu.be/f3iIioA5wI4

3 AI Engine

Deep learning models work as AI engines to drive PDGUIDER (see Figure 3). Camera calibration and recording guidance are based on real-time monitoring of user movements. We embedded PD-GUIDER with two pre-trained pose estimation models: PoseNet [Bazarevsky et al., 2020] and HandPose [Zhang et al., 2020]. PoseNet and HandPose estimate body and hand poses from video frames. We implemented the models with TensorFlow Lite [Google Inc.] to run PD-GUIDER on smartphones and infer user movements. The whole system runs fluently on ordinary Android devices.

3.1 Real-time Calibration

Dim environment and moving out of camera are common problems in out-of-clinic recordings, which greatly hinder diagnosis and research. To ensure high-quality recording, we developed real-time calibration. PD-GUIDER detects environmental luminance by computing frame pixel values of background area. On the meantime, it performs "hard calibration", checking if body bounding box (determined by PoseNet detected body keypoint positions) falls inside movement boundary box (preset positions) frame-by-frame. The system displays the boxes in green and grey respectively. Video recording cannot start or would stop if dim environment or wrong body position occurs(see Figure 4 (a) (b)).

To assist users, a preset silhouette presents on the screen (see Figure 4 (a)). It acts as "soft calibration": aligning head within the silhouette ensures proper recording position. With both "soft calibration" and "hard calibration", users are able to self-calibrate smoothly.

Movement boundary box positions vary with different tests. For example, as the out-of-camera problem is common in gait assessment, PD-GUIDER calibrates with a small boundary box to ensure sufficient walking distance, and then changes the boundary box to full frame size as recording begins. For posture assessment (user stands still in front of camera) that is almost motionless, PD-GUIDER sets large boundary box and silhouette for clearer view of the body.

3.2 Movement Analysis and Interactive Guidance

During recording, PD-GUIDER provides audio guidance to users to perform correct movements, and inform the end of recording. Basic analysis from detected keypoints provides preliminary information about the movement. We included preset guidance for all 12 movement tasks as user instructions, and additional video-based guidance for the four errorprone tests (see below). Note that calibration function always runs in the backend to guarantee proper body position.

Preset Audio Instructions. The system plays preset audios to guide user movement performance step-by-step when recording begins. It alleviates the memory load for patients with PD who may suffer from cognitive impairments. Audio instructions could also distract patients from intentionally controlling their symptoms such as tremor.

Real-Time Guidance for Movements. Besides preset audios, we designed task-specific real-time guidance to better assist users. Inspired by state machine, the system recognizes user movements from detected body/hand keypoints, and guides user to target state in time.

For finger tapping task and hand movement task, PDGUIDER would draw a hand box near the user's shoulder given body keypoints from PoseNet. Users are instructed to raise hand to the corresponding hand box. Instead of predicting from the entire frame, the HandPose model infers hand keypoints from the hand box area to improve accuracy. The system recognizes finger tapping or hand open-close behavior by hand keypoint positions and counts movement repeat numbers to determine the end of task. (see Figure 2)

For arising-from-chair task, camera calibration makes enough space for standing by inferring standing position from seated user. Then PD-GUIDER guide user to stand up with arm crossed before chest (if unsuccessful, push off using the chair arms). As user stands steadily, it gives sit-down instruction and automatically ends recording after the user is seated.

For gait task (in which user walks toward camera, turns around and returns to the armchair), calibration process is similar to the arising-from-chair task. Since it is hard for user to determine the turning point during walking, PDGUIDER monitors user's body box size and guide him/her to turn around when the body box reaches preset threshold. As user sits steadily, the recording ends.

Body out of camera!

Turn around, please.

(b)

(c)

Figure 4: Interactive Guidance. (a) Alert of dim environment background in calibration. (b) Alert of out-of-camera error in gait task. (c) Instruction of turning in gait task.

4 Evaluation

We evaluated three aspects of PD-GUIDER : the guidance efficacy and the app usability rated by users, and the video quality examined by experts. Five healthy subjects used the app and confirmed that it functioned normally. We recruited 21 patients with PD to record MDS-UPDRS movement assessment and divided them into two groups: 10 patients used PD-GUIDER (with app group), and the rest didn't (without app group). The with app group showed improved efficacy compared to the without app group: lowered movement error rate (proportion of videos with wrong movements; with app: 9.3%, without app: 16.7%), and shortened total recording time (with app: 34 minutes, without app: 57 minutes; group average; statistically significant). The PD-GUIDER users rated the app with high usability (7 rated useful or very useful, 3 rated neutral; all rated helpful or very helpful). Two experts approved that PD-GUIDER improved video quality, and the proportion of videos missing critical body parts fell below 2% (with app) from above 50% (without app). PDGUIDER also successfully guided PD patients with cognitive problems.

5 Conclusion

We proposed PD-GUIDER, an interactive video acquisition and learning system to assist motor assessments for PD patients with high efficiency and high quality. We implemented cutting-edge computer vision and machine learning techniques into the mobile app to guide, record, and analyze movement videos of clinical motor examinations. AI-based calibration and interactive procedure ensured video quality and the clinical value of recordings.

Preliminary experiment demonstrated the potential of PDGUIDER as an efficient AI-based telemedicine approach for Parkinson's disease. Collecting high-quality and diagnosticlevel videos in large-scale becomes feasible with this system. Currently, our system is limited to single person analysis. To facilitate patients who need assistance in certain tasks such as gait, we would implement multi-person tracking and analyzing tools. PD-GUIDER could be easily modified and applied to speech assessments of PD [Fang et al., 2020] and motor examinations of other movement disorders. Video deidentification methods [Zhu et al., 2020] can be applied for privacy protection concerns. Automatic rating and symptom analysis would further benefit patients with diagnostic suggestions and long-termmonitoring in the future.

Acknowledgements

This work is sponsored by Beijing Municipal Science & Technology Commission (Z211100003521006). The authors thank the National Engineering Laboratory for Neuromodulation for supporting this research. Correspondence to Yanan Sui and Chen Zhang.