---
title: "Intelligent Agent for Assessing and Guiding Rehabilitation Exercises"
authors:
  - "Min Hun Lee"
doi: "10.24963/ijcai.2019/911"
url: "https://ijcai.org/proceedings/2019/911"
pdf: "https://ijcai.org/proceedings/2019/0911.pdf"
venue: "Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence"
year: 2019
pages: "6444-6445"
track: "Doctoral Consortium"
keywords:
  - "Robotics: Human Robot Interaction"
  - "Humans and AI: Human-AI Collaboration"
  - "Humans and AI: Personalization and User Modeling"
  - "Machine Learning Applications: Bio;Medicine"
---

# Intelligent Agent for Assessing and Guiding Rehabilitation Exercises

Min Hun Lee

*Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence*, pages 6444-6445. DOI: [10.24963/ijcai.2019/911](https://doi.org/10.24963/ijcai.2019/911)

## Abstract

Individualized rehabilitation sessions with a therapists are important to improve physical ability of a patient with neurological and musculoskeletal disorders. However, in-home rehabilitation regimens are often prescribed due to therapist’s limited availability. During in-home rehabilitation, patients might become confused whether they correctly follow rehabilitation regimens. To address this problem, this research aims to develop a Socially Assistive Robot that can interactively accommodate expert’s domain knowledge into a data-driven model to assess exercise performance and guide desirable joint positions for improvement.

## Full text

## 1 Introduction

Individualized rehabilitation sessions with a therapist are important to improve physical ability of a patient with neurological and musculoskeletal disorders (e.g. stroke). However, therapists often prescribe in-home rehabilitation regimens due to their limited availability. During in-home rehabilitation, patients might become confused whether they correctly perform rehabilitation regimens and therapists have difficulty with tracking the progress of a patient

To address this challenge, researchers have explored computer-assisted rehabilitation systems with advanced sensor and machine learning techniques [Webster and Celik, 2014]. Prior research demonstrates the feasibility to collect information about patient's in-home exercises (e.g. duration and the correctness of an exercise). However, prior research is limited to being a passive monitoring system, which focuses on visualizing sensor data or detecting incorrect movements offline [Webster and Celik, 2014]. Passive monitoring systems cannot provide patient real-time performance assessment and corrective suggestions for improvement. Moreover, passive monitoring generates an overwhelmingly large amount of patient's exercise data that requires significant effort for meaningful interpretation and interventions. Prior research still encounters challenges to address subject variability and generate personalized suggestions for improvement.

This research proposes an intelligent agent that leverages machine learning and a Kinect sensor to support rehabilitation exercises of patients with motor disorders, through assessing exercise performance and generating corrective instructions for improvements (Figure 1). Our approach is to develop an interactive assessment model that can incorporate expert's domain knowledge into a data-driven model to evaluate the quality of a motion and generate suggestions for improvements. Furthermore, this work will learn a guidance model of performing an exercise, which can predict desirable joint positions to demonstrate and instruct an exercise. An agent with assessment and guidance models will be deployed on a Socially Assistive Robot to interact with patients.

Figure 1: An agent iteratively leverages expert's domain knowledge and data-driven models to assess and guide patient's rehabilitation exercises while presenting an report on patient's performance

![Image](https://ijcai.org/proceedings/2019/fulltext/911/image_000000_36b892c82d139cc15b42c3a1addf84ba60a21614fe83839452421d6254eca85c.png)

This research hypothesizes that such an agent can enhance current rehabilitation practices by transcribing a patient's performance report for a therapist and generating useful instructions for patients. This research will validate this hypothesis by (1) empirically evaluating how well an agent can replicate therapist's assessment and predict joint positions and (2) conducting user studies with therapists and post-stroke survivors.

## 2 Study of Stroke Rehabilitation

To demonstrate the feasibility of the proposed work, we utilize three task-oriented, upper-limb stroke rehabilitation exercises recommended by therapists ('Bring a cup to the mouth', 'Switch a light on', and 'Move forward a cane') [Lee, 2018]. Three performance components ('Range of Motion', 'Smoothness', and 'Compensation') and their evaluation guidelines are specified based on clinical stroke rehabilitation assessment tools (e.g. Fugl Meyer Assessment and Wolf Motor Function Test).

We recruited 15 stroke patients with different level of functional abilities (37 ± 21 Fugl Meyer Scores) and 11 healthy subjects to collect exercise data. We utilized a Kinect v2 sensor (Microsoft, Redmond, USA) to record the trajectory of body joints and video frames at 30 Hz. A patient performed 10 repetitions of each exercise with both affected and unaffected sides. A healthy subject performed 15 repetitions of each exercise with the subject's dominant side. In total, each exercise contains 465 trials: 315 trials of unaffected movements and 150 affected movements. Two therapists watched recorded videos to evaluate collected exercise data on a 3-point scale score over three performance components as ground truth scores.

## 3 Interactive Machine Learning to Improve an Automated Rehabilitation Assessment

In contrast to previous research efforts on detecting incorrect movements [Webster and Celik, 2014], our assessment model can not only detect incorrect movements, but also quantify a performance score [Lee et al., 2019]. The assessment of exercises requires the prediction of a categorical score (e.g. '0: no motion', '1: limited', '2: normal'). We implemented threshold models with binary classifications [Li and Lin, 2007] using Decision Trees and Neural Networks and evaluated models with Leave-One-Subject-Out cross validation on the dataset in Section 2. The experimental results demonstrated the feasibility of automatically assessing stroke rehabilitation exercises with the decent agreement levels (e.g. 0.7571 - 0.8254 F1-scores) and relevance with clinically validated, Fugl Meyer scores (R2 = 0.95).

However, [Lee et al., 2019] has a limitation of being a black box system that cannot explain an output of computer assisted assessment to support therapist's decision making and provide detailed feedback. To address this limitation, we propose an interactive machine learning approach that identifies salient features to generate user-specific feature analysis and elicit domain expert's knowledge. For identifying salient features, we utilize Reinforcement Learning with Double Qlearning [Hasselt et al., 2016] to train an agent that sequentially decides whether another feature is necessary to assess an exercise while receiving a negative reward for requesting an additional feature. Using the selected features, the agent can compute how much patient's affected motions are deviated from unaffected motions to (1) transcribe an interpretable performance report and (2) generate personalized suggestions for improvement. This transcribed report will be utilized to elicit expert's domain knowledge. According to the preliminary experiments, our approach can perform significantly better than basic interpretable models (e.g. Decision Trees or Linear Regressions) and non-interactive Neural Network approach (p < 0.05). In future, we will conduct a study with therapists to explore whether an automatically transcribed report can generate new insights for therapist's decision making.

## 4 ARobotic Rehabilitation Guidance

This research will utilize a Socially Assistive Robot (SAR) to evaluate the effectiveness of supporting a post-stroke patient's rehabilitation exercise session. We will learn an guidance model that can predict desirable sequential joint positions of an exercise with Encoder-Recurrent-Decoder approach [Fragkiadaki et al., 2015] and few demonstrations of individual's unaffected side. Even if learning the entire number of possible motions is challenging, we can leverage the history of prior motions to reduce the set of next possible motions. Furthermore, we represent the body joints on N ×N grid [Fragkiadaki et al., 2015] and derive eight direction actions of joints (e.g. Up, Right). We assume that such discrete action space would be sufficient to guide joint positions, because therapists typically rely on high level directional instructions on joints instead of informing the exact joint positions in continuous values. The guidance module will enable an agent to indicate a next target goal or a robot to demonstrate how to perform an exercise. After integrating interactive assessment and guidance models into a robot, we will follow the study of [Fasola and Matari´c, 2013] to analyze the usability of a system with post-stroke patients.

## 5 Expected Contribution and Conclusion

This research will create an intelligent agent that can mimic the role of a therapist (i.e. monitoring exercise to assess patient's performance and generate guidance for improvement) to enhance current rehabilitation practices. This agent can objectively assess patient's performance and transcribe a detailed report for therapists to track patient's progress. In addition, the agent on a Socially Assistive Robot can assist patients to correctly engage in rehabilitation exercises.

## References

1. [Fasola and Matarić, 2013] Juan Fasola and Maja J Matarić. A socially assistive robot exercise coach for the elderly. Journal of Human-Robot Interaction, 2(2):3-32, 2013.
2. [Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354, 2015.
3. [Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Thirtieth AAAI Conference on Artificial Intel- ligence, AAAI'16, pages 2094-2100. AAAI Press, 2016.
4. [Lee et al., 2019] Min Hun Lee, Daniel P Siewiorek, Asim Smailagic, Alexandre Bernardino, et al. Learning to as- sess the quality of stroke rehabilitation exercises. In Pro- ceedings of the 24th International Conference on Intelli- gent User Interfaces, pages 218-228. ACM, 2019.
5. [Lee, 2018] Min Hun Lee. A technology for computer- assisted stroke rehabilitation. In 23rd International Con- ference on Intelligent User Interfaces, pages 665-666. ACM, 2018.
6. [Li and Lin, 2007] Ling Li and Hsuan-Tien Lin. Ordinal re- gression by extended binary classification. In Advances in neural information processing systems, pages 865-872, 2007.
7. [Webster and Celik, 2014] David Webster and Ozkan Celik. Systematic review of kinect applications in elderly care and stroke rehabilitation. Journal of NeuroEngineering and Rehabilitation, 11(1):108, Jul 2014.