Intelligent Assistant for Elderly

1 Introduction

The IN LIFE project was a 3-year project spanning across 9 European countries and 20 partners [Bizjak et al., 2017]. The main goal was to develop and test various services that increase the comfort and security of the elderly and their carer staff. The solutions developed were tested on 6 large-scale pilot sites where more than 2000 elderly above 65 participated. Here, we present the system that was developed for the Slovenian pilot site and focus on fall detection and activity monitoring modules, focused on a personalized smartwatch.

One of the main fears of the elderly who live alone is that they will not be able to call for help in case of a fall or other accident [Tinetti and Williams, 1997]. Approximately 30% of the elderly over the age of 65 experience a fall each year, and after a first fall the chance for the second one drastically increases. Furthermore, such falls may result in a "long lie", for several hours, which significantly deteriorates the health conditions, even resulting in death in most severe cases [Wild et al., 1981].

A solution would need not only to report autonomously about the fall in case of user inability, but would also need to be as discrete as possible and at the same time preserve the user's independence. An obvious solution is a personalized (virtual) assistant, accompanying the user at all times.

2 Intelligent Assistant

A suitable platform for such a service could be a smartphone, however there are several issues with this in real life. Most of the elderly are still not using these devices and do not want to learn how to interact with them. When performing a baseline study before the pilots in Europe, only one person out of 50 had a personal tablet and a couple of them had a smartphone. Dementia at different stages is another issue, since the elderly often forget even simple commands. In addition, the elderly do not like to wear their devices at home and often forget to bring them with them when leaving home. Therefore, we decided to use a smartwatch device as the core device of the virtual carer system. People are typically used to the concept of wearing a watch and it is harder to forget it somewhere if you have it strapped to a wrist. On the downside, the watch has a smaller screen than phones or tablets and therefore somewhat more limited functionality regarding the interaction with the user through a graphic interface. The smartwatch is connected to the web portal used by formal or informal carers and also allows the elderly automatic or manual calls to the emergency call center.

During the pilots we analysed the usage of the system and interviewed each participant at several occasions in order to learn the strong points and the shortcomings of the system. It turned out that most users with advanced stages of dementia soon forgot about the system and stopped using it. To fix this problem we changed the design philosophy of the whole system.

Among the conclusions are the need to personalize the system to each individual user, target the system for the users with no advanced stages of dementia, and add user-friendly functions even though not safety related. One of the consequences is that the system is designed modularly [Figure 1], and that there are various modules the user can turn on or off independently. The goal is that the user adopts the system as soon as possible, initially not necessary as a safety device, but rather as a fitness tracker or a navigation tool. With this approach, the user gets familiar with the system and when such a need arises, dedicated modules are activated. When dementia progresses, the user is already used to the system. Another advantage of the modular design is personalization. Users can selectively use the modules to fit their needs and wishes, and since not everything is running on the system all the time, battery life and performance are also improved.

Two main modules tested in the pilots were the automatic fall detection and the activity monitoring module. The fall detection module uses the data from the accelerometer to determine whether a fall occurred [Luˇstrek et al., 2011]. As hands are used in most of the everyday activities including fast movement, it is extremely hard to have a high precision for detection, resulting in false alarms during the day [Kangas et al., 2008; Gjoreski et al., 2016]. We focused on the most grievous falls where person becomes unconscious or is otherwise unable to get up or manually call for help. In other cases we reason that the person can press the SOS button and call for help manually.

The rough idea of the pattern during the fall is shown in [Figure 2]. There is some movement before the fall phase, during the fall there is low movement as a person is in a free fall toward the ground, followed by a large peak on all three axis as the person suddenly stops on the floor. The period after the fall is the most important for the algorithm as we check for the movement after. If the person gets up or can otherwise move, there will be movement visible. On the other hand, if the person is unconscious, the period after the fall will contain little to no movement [Bochanovski et al., 2016]. We tested several different ML methods, among which deep neural networks and k-nearest neighbours achieved highest accuracy; over 93% in a laboratory setting. During the pilots, the accuracy was lower due to the nature of the study, but we found out some possible scenarios that we have previously not anticipated, such as the acceleration patterns for wheelchair-bound users, and were able to adjust the algorithm accordingly. For the field application we chose kNN over DNN because it is less computationally complex.

In addition to the fall module, the activity monitoring module was implemented. It monitors the average activity of the user throughout the days [Cvetkovi´c et al., 2016]. The main purpose of this module is to detect when the user is feeling unwell or is not moving due to whatever reason in uncommon circumstances. The module uses statistical model to compare the activity levels from the previous days. The model divides the day into multiple timeslots so it can better model the user's activity throughout different periods of the day, achieving better personalization.

A sub-module of the activity monitoring is a sleep detector, aimed to reduce the number of false alarms during the night (movement during the sleep can often resemble a fall) and to conserve the battery power (by reducing the sampling rate). The method takes into account the time of the day and user's activity levels during the evening/night time. The watch in the night mode still tracks the user's movement: if the movement rate increases significantly (person wakes up to go to the bathroom or similar) over a short period of time (minutes), the night mode automatically turns off in order to be able to detect fall if it happens. When the person goes back to sleep, the watch returns into the night mode again.

Other modules are utility-based and include GPS tracking, geo-fencing, over-the-air settings, data collection, and collaborative learning.

3 Conclusion

A large 3-year EU project aimed at bridging the gap between research prototypes and mass adaptation of systems for the elderly. The experience obtained might not be relevant only for the smartwatch, but for other systems assisting the elderly. During the pilots, the user and carer reaction to the system was generally positive. The large-scale pilots proved to be important in application development. Several ideas designed and tested in laboratory were significantly improved after user's feedback, proving just how important it is during development of such system to be in close collaboration with end users. This demo shows the product close to the version that users would actually want to use everyday. The remaining improvements address mostly the ergonomics of the watch and the improved battery life.

Acknowledgments

The research leading to these results has received funding from the EU H2020 project IN LIFE (grant agreement No. 643442). The details behind the implementations and pilots are explained in the award-winning paper [Bizjak et al., 2017]