Multi-Sensor Mobile Platform for the Recognition of Activities of Daily Living and their Environments based on Artificial Neural Networks

1 Introduction

Following the development of a framework for the recognition of the ADL [Foti et al., 2013] and their environments, the use of the diversity of the sensors available in off-theshelf mobile devices, including motion sensors, e.g., accelerometer and gyroscope, magnetic sensors, e.g., magnetometer, acoustic sensors, e.g., microphone, and location sensors, e.g., GPS receiver, allows the creation of a method for this purpose that should be a function of the different subsets of sensors available in the large variety of the mobile devices existent in the market [Salazar et al., 2013]. The proposed the architecture in [Pires et al., 2016a; Pires et al., 2015; Pires et al., 2016b] is based in the combination of several types of methods, such as data acquisition, data processing, data fusion and classification methods, for further inclusion of the framework in the development of a personal digital life coach

[Garcia, 2016]. The development of this type of systems is included in the large group of Ambient Assisted Living (AAL) systems [Garcia et al., 2014].

The proposed demonstration presents a mobile application that implements an Android library with the proposed methods in [Pires et al., 2017a; Pires et al., 2017b; Pires et al., 2017c; Pires et al., 2018], which the classification methods are simply structured in three stages, these are the recognition of ADL with movement, i.e., walking, running, going upstairs and downstairs, and standing, the recognition of environments, i.e., bar, bedroom, classroom, gym, hall, kitchen, library, street, and watching TV, and the recognition of standing ADL, i.e., watching TV, standing, and driving. Firstly, for the recognition of ADL with movement, the fusion of the features extracted from the motion and magnetic sensorsÕ data is recommended. Secondly, for the recognition of environments, the use of the features extracted from the acoustic sensorsÕ data is recommended. Finally, for the recognition of standing ADL, the fusion of the environment recognized with the features extracted from the motion, magnetic and location sensorsÕ data is recommended.

Several studies [Pires et al., 2017a; Pires et al., 2017b; Pires et al., 2017c; Pires et al., 2018] presented the classification methods that should be used for the recognition of ADL with movement and standing ADL are the DNN, and for the recognition of environments are the Feedforward Neural Networks with Backpropagation, but, during the development and tests of the Android library, we reported reliable accuracies for the recognition of ADL with movement and standing ADL, but the reported accuracies for the recognition of environments are very low. Thus, for the recognition of environments, we tried to implement the DNN method, verifying that it reports reliable accuracies. The difference of these results is caused by the limitations of the mobile devices, related to the low power processing and battery consumption capabilities. The main motivation of this work is to create a reliable method for the recognition of ADL and their environments with local processing in order to provide a feedback to the user in real-time.

Taking in account the limitations of the mobile devices, the results reported by the Android library are reliable and adapted to the number of sensors available in the off-the-shelf mobile devices, reporting an overall accuracy between 58.02% and 89.15%, depending on the number of sensors available.

2 Main Purpose and Implementation

The framework for the recognition of ADL and their environments intends to reduce the main constraints for the recognition of ADL and their environments, because it was already performed with complex equipment with high processing capabilities and constant network connection. However, the commonly used mobile devices may handle the recognition of them with lightweight methods, providing a feedback in real-time.

For the implementation of this project we selected some volunteers aged between 16 and 60 years old with different lifestyle, height, weight and health state, in order to create the classification methods and further testing of the application developed. The dataset collected is publicly available at [ALLab, 2017].

Based on the architecture provided in the Pires et al. [Pires et al., 2016a; Pires et al., 2015; Pires et al., 2016b; Pires et al., 2017a; Pires et al., 2017b; Pires et al., 2017c; Pires et al., 2018], the data acquisition is performed with the mobile application, applying, during the data processing, a low pass filter to the data acquired from the motion and magnetic sensorsÕ data and the Fast Fourier Transform (FFT) is applied to the acoustic data in order to extract the relevant frequencies. The data processing included also the extraction of the features for the classification methods, these are the five greatest distances between the maximum peaks, the average, standard deviation, variance, and median of the maximum peaks, plus the average, the standard deviation, the variance, the median, and the minimum, the maximum of the raw signal extracted from the motion and magnetic sensorsÕ data, and 26 Mel-Frequency Cepstral Coefficients (MFCC), plus the standard deviation, the average, the maximum, the minimum, the variance, and the median of the raw signal from the acoustic sensorsÕ data. Finally, these features are fused and used for the creation of a neural network for further classification of the ADL and their environments with the DNN method, where the acoustic data is used for the recognition of the environments and the data acquired from the other sensors are used for the recognition of the different types of ADL.

In addition, to reduce the constraints of the use of the mobile devices, the mobile application collects the sensorsÕ data in background, collecting 5 seconds of data every 5 minutes. Our results proved that is possible to recognize the ADL and environments performed with only 5 seconds of data.

3 Demonstration

The demonstration shows the mobile application for the recognition of ADL and their environments. In the main screen, the ADL, environment and location recognized are presented. In the first case is shown the values recognized in the last use of the mobile application or the last time that the sensors were available (with a delay higher than 30 minutes).

In the second case is shown the values recognized in almost real-time.

The mobile application also allows the definition of the daily plan, providing a daily feedback about the performance of ADL registered in the daily plan. Depending on the mobile device used, it reports an accuracy between 58.02% and 89.15% in the recognition of the 7 ADL and 9 environments proposed.

4 Conclusions and Future Work

This demonstration shows the implementation of a mobile application using an Android library for the recognition of ADL and their environments. It implements some methods that are able to be used with the mobile devices with good performance. Mobile devices has several sensors that can exploit the creation of local processing methods for the recognition of ADL and their environments everywhere, providing a rapid feedback without a constant network connection.

The Android library received the data acquired from the sensors available in the off-the-shelf mobile devices and implements the data processing, data fusion and classification methods. In data processing, the low pass filter and/or the FFT method should be applied to the data acquired for the correct extraction of the features needed and further fusion. Finally, the classification methods used are based in the DNN method with a reported accuracy between 58.02% and 89.15% in the recognition of 7 ADL and 9 environments.

In conclusion, with the use of only 5 seconds of sensorsÕ data, the recognition of ADL and their environments reports reliable results, helping in the minimization of the restrictions with the use of local processing methods in the mobile devices. As future work, the processing speed of previously created neural networks should be optimized, loading and storing them in the memory on the initialization of the library, speeding up the recognition process.

Acknowledgments

This work was supported by FCT project UID/EEA/50008/2013 (Este trabalho foi suportado pelo projecto FCT UID/EEA/50008/2013).

The authors would also like to acknowledge the contribution of the COST Action IC1303 Ð AAPELE Ð Architectures, Algorithms and Protocols for Enhanced Living Environments.