Multi-agent Attentional Activity Recognition

1 Introduction

The ability to identify human activities via on-body sensors has been of interest to the healthcare community [Anguita et al., 2013], the entertainment [Freedman and Zilberstein, 2018] and fitness [Guo et al., 2017] community. Some works of Human Activity Recognition (HAR) are based on handcrafted features for statistical machine learning models [Lara and Labrador, 2013]. Until recently, deep learning has experienced massive success in modeling high-level abstractions from complex data [Pouyanfar et al., 2018], and there is a growing interest in developing deep learning for HAR [Hammerla et al., 2016]. Despite this, these methods still lack sufficient justification when being applied to HAR. In this work, we consider two inherent characteristics of human activities and exploit them to improve the recognition performance.

The first characteristic of human activities is the spatiallytemporally varying salience of features. Human activities can be represented as a sequence of multi-modal sensory data. The modalities include acceleration, angular velocity and magnetism from different positions of testers' bodies, such as chests, arms and ankles. However, only a part of modalities from specific positions are informative for recognizing certain activities [Wang and Wang, 2017]. Irrelevant modalities often influence the recognition and undermine the performance. For instance, identifying lying mainly relies on people's orientations (magnetism), and going upstairs can be easily distinguished by upward acceleration from arms and ankles. In addition, the significance of modalities changes over time. Intuitively, the modalities are only important when the body parts are actively participating in the activities. Therefore, we propose a spatial-temporal attention method to select salient modalities and their active periods that are indicative of the true activity. Attention has been proposed as a sequential decision task in earlier works [Denil et al., 2012; Mnih et al., 2014]. This mechanism has been applied to sensor based HAR in recent years. [Chen et al., 2018] and [Zhang et al., 2018] transform the sensory sequences into 3D activity data by replicating and permuting the input data, and they propose to attentionally keep a focal zone for classification. However, these methods heavily rely on data preprocessing, and the replication increases the computation complexity unnecessarily. Also, these methods do not take the temporally-varying salience of modalities into account. In contrast, the proposed spatial-temporal attention approach directly selects informative modalities and their active time that are relevant to classification from raw data. The experiment results shows that our model makes HAR more explainable.

The second characteristic of human activities considered in this paper is activities are portrayed by motions on several body parts collectively. For instance, running can be seen as a combination of arm and ankle motions. Some works like [Radu et al., 2018; Yang et al., 2015] are committed to fusing multi-modal sensory data for time-series HAR, but they only fuse the information of local modalities from the same positions. These methods, as well as the existing attention based methods [Chen et al., 2018; Zhang et al., 2018], are limited in capturing the global interconnections across different body parts. To fill this gap, we propose a multi-agent reinforcement learning approach. We simplify activity recognition by dividing the activities into sub-motions with which an independent intelligent agent is associated and by coordinating the agents' actions. These agents select informative modalities independently based on both their local observations and the information shared by each other. Each agent can individually learn an efficient selection policy by trial-and-error. After a sequence of selections and information exchanges, a joint decision on recognition is made. The selection policies are incrementally coordinated during training since the agents share a common goal which is to minimize the loss caused by false recognition.

The key contributions of this research are as follows:

• We propose a spatial-temporal attention method for temporal sensory data, which considers the spatiallytemporally varying salience of features, and allows the model to focus on the informative modalities that are only collected in their active periods.
• We propose a multi-agent collaboration method. The agents represent activities with collective motions by independently selecting modalities associated with single motions and sharing observations. The whole model can be optimized by coordinating the agents' selection policies with the joint recognition goal.
• We evaluate the proposed model on four datasets. The comprehensive experiment results demonstrate the superiority of our model to the state-of-the-art approaches.

2 The ProposedMethod

2.1 Problem Statement

We now detail the human activity recognition problem on multi-modal sensory data. Each input sample (x, y) consists of a 2-d vector x and an activity label y. Let x = [x0 , x1 , ...xK ] where K denotes the time window length and xi denotes the sensory vector collected at the point i in time. xi is the combination of multi-modal sensory data collected from testers'

different body positions such as chests, arms and ankles. Suppose that xi = (mi1 , mi2 , ...miN ) = (xi0 , xi1 , ...xiP ), where m denotes data collected from each position, N denotes the number of positions (in our datasets, N = 3), and P denotes the number of values per vector. Therefore, x ∈ RK×P and y ∈ [1, ...,C]. C represents the number of activity classes. The goal of the proposed model is to predict the activity y.

2.2 Model Structure

The overview of the model structure is shown in Figure 1. At each step s, the agents select an active period together and individually select informative modalities from the input x. These agents share their information and independently decide where to "look at" at the next step. The locations are determined spatially and temporally in terms of modalities and time. After several steps, the final classification is jointly conducted by the agents based on a sequence of the observations. Each agent can incrementally learn an efficient decision policy over episodes. But by having the same goal, which is to jointly minimize the recognition loss, they collaborate with each other and learn to align their behaviors such that it achieves their common goal.

Multi-agent Collaboration

In this work we simplify activity recognition by dividing the activities into sub-motions and require each agent select informative modalities that are associated with one motion. Suppose that we employ H agents a1, a2, ...aH (we assume H = 3 in this paper for simplicity). The workflows of a1, a2, a3 are shown in red, green and blue in Figure 1.

At each step s, each agent locally observes a small patch of x, which includes information of a specific modality from a motion in its active period. Let the observations be es1 , es2 , es3 as Figure 1 shows. They are extracted from x at the locations (ts , ls1 ), (ts , ls2 ) and (ts , ls3 ), respectively, where t denotes the selected active period and l denote the location of a modality in the input x. The model encodes the region around (ts , ls i ) (i ∈ {1, 2, 3}) with high resolution but uses a progressively lower resolution for points further from (ts , ls i ) in order to remove noises and avoid information loss in [Zontak et al., 2013]. We then further encode the observations into higher level representations. With regard to each agent ai (i ∈ {1, 2, 3}), the observation esi and the location (ts , ls i ) are linear transformed independently, parameterized by θe and θtl, respectively. Next, the summation of these two parts is further transformed with another linear layer parameterized by θo The whole process can be summarized as the following equation:

where L(·) denotes a linear transformation and concat(ts , ls i ) represents the concatenation of ts and ls i . Each linear layer is followed by a rectified linear unit (ReLU) activation. Therefore, osi contains information from "what" (ρ(C f , lf t )), "where" (lf t ) and "when".

Making multiple observations not only avoids the system processing the whole data at a time but also maximally prevents the information loss from only selecting one region of data. Furthermore, multiple agents make observations individually so that they can represent activities with the collective modalities from different motions. The model can explore various combinations of modalities to recognize activities during learning.

Then we are interested in the collaborative setting where the agents communicate with each other and share the observations they make. So we get the shared observation osg by concatenate os1 , os2 , os3 together.

so that osg contains all the information observed by three agents. A convolutional network is further applied to process osg and extract the informative spatial relations. The output is then reshaped to be the representation rs g .

And rs g represents the activity to be identified with multiple modalities selected from motions on different body positions.

Attentive Selection

In this section, the details about how to select modalities and active period attentively are introduced. We first introduce the episodes in this work. The agents incrementally learn the attentive selection policies over episodes. In each episode, following the bottom-up processes, the model attentively selects data regions and integrates the observations over time to generate dynamic representations, in order to determine effective selections and maximize the rewards, i.e., minimize the loss. Based on this, LSTM is appropriate to build an episode as it incrementally combines information from time steps to obtain final results. As can be seen in Figure 1, at each step s, the LSTMmodule receives the representation rs g and the previous hidden state hs-1 as the inputs. Parameterized by θh, it outputs the current hidden state hs :

Now we introduce the selection module. The agents are supposed to select salient modalities and an active period at each step. To be specific, they need to select the locations where they make next observations. Three agents control ls+1 1 , ls+1 1 , ls+1 3 independently based on both the hidden state hs and their individual observations os1 , os2 , os3 so that the individual decisions are made from the overall observation as well. ts+1 is jointly decided based on hs only since it is a common selection. The decisions are made by the agents' selection policies which are defined by Gaussian distribution stochastic process:

and

The purpose of stochastic selections is to explore more kinds of selection combinations such that the model can learn the best selections during training.

To align the agents' selection policies, we assign the agents a common goal that correctly recognizing activities after a sequence of observations and selections. They together receive a positive reward if the recognition is correct. Therefore, at each step s, a prediction ˆ ys is made by:

Usually, agents receive a reward r after each step. But in our case, since only the classification in the last step S is representative, the agents receive a delayed reward R after each episode.

The target of optimization is to coordinate all the selection policies by maximizing the expected value of the reward ¯R after several episodes.

2.3 Training and Optimization

This model involves parameters that define the multi-agent collaboration and the attentive selection. The parameters Θ = {θe, θtl, θo, θc, θh, θli , θt, θy} (i ∈ {1, 2, 3}). The parameters for classification can be optimized by minimizing the crossentropy loss:

where Fy is the overall function that outputs ˆ y given x. C is the number of activity classes, and yn(c) = 1 if the n-th sample belongs to the c-th class and 0 otherwise.

However, selection policies that are mainly defined by θli (i ∈ {1, 2, 3}) and θt are expected to select a sequence of locations. The parameters are thus non-differentiable. In this view, we deploy a Partially Observable Markov Decision Process (POMDP) [Cai et al., 2009] to solve the optimization problem. Suppose es = (es1 , es2 , es3 ), lts = (ls1 , ls2 , ls3 , ts ), We consider each episode as a trajectory τ = {e1 , lt1 , y1 ; e2 , lt2 , y2 ; ..., eS , ltS , yS }. Each trajectory represents one order of the observations, the locations and the predictions our agents make. After agents repeat N episodes, we can obtain {τ 1 , τ 2 , ..., τ N }, and each τ has a probability p(τ;Θ) to be obtained. The probability depends on the selection policy Π = (π1, π2, π3) of the agents.

Our goal is to learn the best selection policy Π that maximizes ¯R. Specifically, Π is decided by Θ. Thus we need to find out the optimized Θ∗ = argmax Θ [ ¯R]. One common way

is gradient ascent.

Generally, given a sample x with reward f(x) and probability p(x), the gradient can be calculated as follows:

In our case, a trajectory τ can be seen as a sample, the probability of each sample is p(τ;Θ), and the reward function ¯R = Ep(τ ;Θ)[R]. We have the gradient:

By considering the training problem as a POMDP and following the REINFORCE rule [Williams, 1992]:

Since we need several samples τ for one input x to learn the best policy combination, we adopt Monte Carlo sampling which utilizes randomness to yield results that might be theoretically deterministic. Supposing M is the number of Monte Carlo sampling copies, we duplicate the same input for M times and average the prediction results. The M copies generate M subtly different results owing to the stochasticity, so we have:

where M denotes the number of Monte Carlo samples, i denotes the ith duplicated sample, and yi is the correct label for the ith sample. Therefore, the overall optimization can be summarized as maximizing ¯R and minimizing Eq. 9. The detailed procedure is shown in Algorithm 1.

3 Experiments

3.1 Experiment Setting

We now introduce the settings in our experiments. The time window of inputs is 20 with 50% overlap. The size of each

Algorithm 1 Training and Optimization

Require: sensory matrix x, label y, the length of episodes S, the number of Monte Carlo samples M. Ensure: parameters Θ. 1: Θ = RandomInitialize() 2: while training do 3: duplicate x for M times 4: for i from 1 to M do 5: l1(i) 1 , l1(i) 2 , l1(i) 3 , t1(i) = RandomInitialize() 6: for s from 1 to S do 7: extract es(i) 1 , es(i) 2 , es(i) 2 8: os(i) 1 , os(i) 2 , os(i) 3 ←Eq. 1 9: os(i) g , rs(i) g , hs(i) ←Eq. 2,Eq. 3,Eq. 4 10: ls(i) 1 , ls(i) 2 , ls(i) 3 , ts(i) ←Eq. 5,Eq. 6 11: ˆ ys(i) ←Eq. 7 12: record τ (i) 1:s 13: end for 14: R(i) ←Eq. 8 15: end for 16: ˆ y = 1M ∑Mi=1 ˆ yS(i) 17: Lc,∇Θ ¯R←Eq. 9,Eq. 13 18: Θ←Θ-∇ΘLc +∇Θ ¯R 19: end while 20: return Θ

observation patch is set to K8 × P8 , where K ×P is the size of the inputs. In the partial observation part, the sizes of θe, θtl, θo are 128, 128, 220, respectively. The filter size of the convolutional layer in the shared observation module is 1 × M and the number of feature maps is 40, where M denotes the width of osg . The size of LSTM cells is 220, and the length of episodes is 40. The Gaussian distribution that defines the selection policies is with a variance of 0.22.

To ensure the rigorousness, the experiments are performed by Leave-One-Subject-Out (LOSO) on four datasets, MHEALTH [Banos et al., 2014], PAMAP2 [Reiss and Stricker, 2012], UCI HAR [Anguita et al., 2013] and MARS. They contain 10, 9, 30, 8 subjects' data, respectively.

3.2 Comparison with State-of-the-Art

To verify the overall performance of the proposed model, we first compare our model with other state-of-the-art methods. The compared methods include a convolutional model on multichannel time series for HAR (MC-CNN) [Yang et al., 2015], a CNN-based multi-modal fusion model (C-Fusion) [Radu et al., 2018], a deep multimodal HAR model with classifier ensemble (MARCEL) [Guo et al., 2016], an ensemble of deep LSTM learners for activity recognition (ELSTM) [Guan and Pl¨otz, 2017], a parallel recurrent model with convolutional attentions (PRCA) [Chen et al., 2018] and a weighted average spatial LSTM with selective attention (WAS-LSTM) [Zhang et al., 2018].

As can be observed in Table 1, with respect to the datasets, MARCEL, E-LSTM, PRCA, WAS-LSTM and the proposed model perform better than MC-CNN and C-Fusion in MHEALTH and PAMAP2, as these models enjoy higher variance. They fit well when data contain numerous features and complex patterns. On the other hand, data in UCI HAR and MARS have fewer features, but MARCEL, E-LSTM, PRCA and our model still perform well while the performance of WAS-LSTM deteriorates. The reason is that WASLSTM is based on a complex structure and it requires more features as input. In contrast, MARCEL and E-LSTM adopt rather simple models like DNNs and LSTMs. Despite the ensembles, they are still suitable for fewer features. PRCA and the proposed model select salient features directly with intuitive rewards, so they do not necessarily need a large number of features as well. In addition, the attention based methods, PRCA and WAS-LSTM, are more unstable than the other methods since the selection is stochastic and they cannot guarantee the effectiveness of all the selected features. Overall, our model outperforms the compared state-of-the-art and eliminates the instability of regular selective attentions.

3.3 Ablation Study

We perform a detailed ablation study to examine the contributions of the proposed model components to the prediction performance in Table 2. Considering that there are five removable components in this model: (a) the modality selection module, (b) the transformation from esi to osi (i ∈ {1, 2, 3}), (c) the convolutional network for higher-level representations, (d) the temporal attentive selection (e) the multi-agent. We consider six structures: S1: We first remove the selection module including the observations, episodes, selections and rewards. For comparison, we set S1 to be a regular CNN as a baseline. S2: We employ one agent but remove (b), (c), (d) and (e), so the workflow is: inputs → esi → LSTM → selections and rewards. The performance decreases and is more unstable than other structures. Although S2 includes attentions, the model does not include the previous selections in their observations, which influences their next decisions significantly. S3: Based on S2, we add (b) to (a). (b) contributes considerably since the prediction results are improved by 5% to 7%, because it feeds back the history selections to the agents for learning. S4: We further consider (c) in the model. It can be observed that this setting also achieves better performance than S3 since it convolutionally merges the partial observations. S5: (d) is added. The workflow is the same as S3, but the agents make an additional action: selecting tS , which leads to another attention mechanism in time level. The performance is improved by 3% to 5%. S6: The proposed model. When combining all these benefits, our model achieves the best performance, higher than S5 by 5% to 7%.

3.4 Visualization and Explainability

The proposed method decomposes the activities into participating motions, from each of which the agents decide the most salient modalities individually, which makes the model explainable. We present the visualized process of recognizing standing, going upstairs, running and lying on MHEALTH. The available features include three 3-axis acceleration from chests, arms and ankles, two ECG signals, two 3-axis angular velocity and two 3-axis magnetism vectors from arms and ankles. Figure 2 shows the modality heatmaps of all agents. We observe that each agent does focus on only a part of modalities in a time period during recognition. Table 3 lists the most frequently selected modalities. We can observe that magnetism (orientation) in standing and lying is selected as one of the most active features, owing to the fact it is easy to distinguish between standing and lying with people's orientation. Another example is that the most distinguishing characteristic of going upstairs is "up". Therefore, Z-axis acceleration is specifically selected by agents for going upstairs. Also, identifying running involves acceleration, ECG, and arm swing, which conforms to the experiment evidence as well. The agents also select several other features with lower frequencies, which avoids losing effective information.

4 Conclusion

In this work, we first propose a selective attention method for spatially-temporally varying salience of features. Then, multi-agent is proposed to represent activities with collective motions. The agents' cooperate by aligning their actions to achieve their common recognition target. We experimentally evaluate our model on four real-world datasets, and the results validate the contributions of the proposed model.