Answering Mixed Type Questions about Daily Living Episodes

1 Introduction

Recording human activities in the physical-world as linguistic descriptions [Krishna et al., 2017; Miyanishi et al., 2018] can bring a better understanding of our everyday lives. Answering questions about such human daily episodes as in a personal digital store [Gemmell et al., 2002] is an essential skill for intelligent machines that support people in their everyday activities such as human-memory aid and healthcare monitoring.

In this paper, we tackle a physical-world questionanswering (QA) task where the QA method returns various answers in response to the content of a question when a sequence of sentences about physical-world events is given as in Figure 1. To achieve this scenario, the QA method requires complex reasoning referring to multiple contexts because everyday life consists of multiple events (e.g. drinking coffee in the living room after making it in the kitchen and moving to the living room.) Moreover, since people have a variety of information needs in physical-world situations, the methods of the physical-world QA tasks have to answer mixed-type responses (e.g. word sequence, word set, number, and time, as well as a single word) as to a user question. However, most imental results suggest that encoding physical-world events with memory components significantly improves QA performance. Moreover, we found that multi-task learning on different type QA problems is highly effective for solving physical-world QA problems and that self-paced learning of multiple decoders supports solving difficult QA tasks.

The remainder of this paper is organized as follows. First, we provide some background in question-answering for text stories. Then, we present the physical-world QA task and the proposed method to solve it. Finally, we show an experimental evaluation of daily-activity datasets collected in house-like settings and then present our conclusions.

2 RelatedWork

The physical-world QA can be seen as returning text response after reading a text describing a story of arbitrary length. There have been many QA tasks answering questions for stories, such as selecting an answer from multiple choices after reading materials [Richardson et al., 2013; Weston et al., 2016], selecting the span of text that answers a question from a large collection of documents [Yang et al., 2015; Rajpurkar et al., 2016], and predicting the missing named entities in a given passage [Hermann et al., 2015; Hill et al., 2016].

Among these tasks, the bAbI [Weston et al., 2016] 1 is closely related to our physical-world QA tasks. The dataset is commonly used for checking algorithms of text-story understanding and reasoning. In this dataset, a series of triplets (short stories, question, and answer) generated by a simple simulation similar to the approach of [Bordes et al., 2010] are given. For example, when the story and a question, "Mary moved to the bathroom. John went to the hallway. Where is Mary?" is given, QA methods have to give the correct answer "bathroom". To solve this task, many deep learning methods have been proposed such as Memory Networks [Weston et al., 2015; Sukhbaatar et al., 2015], Dynamic Memory Networks [Kumar et al., 2016], and Differentiable Neural Computers [Graves et al., 2016]. These neural-network models have an external memory for encoding a given sequence of sentences and access it using an attention mechanism [Bahdanau et al., 2015] for selecting a relevant answer. The results of these works indicate using external memory as encoders is highly effective for answering questions that require solving long-term sentence dependencies compared to RNN-based models [Hochreiter and Schmidhuber, 1997; Sukhbaatar et al., 2015] that output an answer after reading a sequence of words in a question and story as input. Even though we also use Memory Networks for encoding a sequence of sentences that represent daily episodes, the effectiveness of memory components is unknown when answering physical-world QA tasks. In addition, we use multiple decoders and simultaneously learn them for producing mixedtype answers that are required to respond to physical-world questions instead of selecting prepared candidates of answers or producing words or word sequences with a single decoder. Furthermore, the physical-world QA dataset we use is not artificial data created by simulation unlike the bAbI dataset. Our dataset is manually annotated by humans watching firstperson videos captured by wearable cameras attached to subjects who perform various activities in a real house.

1 https://github.com/facebook/bAbI-tasks

3 ProposedMethod

In this section, we present our method. First, we give an overview of the proposed physical-world QA framework along with QA examples. We then proceed to the details of our QA framework based on the encoder-decoder model.

3.1 Physical-world Question Answering

Our goal is to answer questions about daily episodes in response to a given language question and temporally ordered sentences that describe activities of daily living (i.e. event timeline). We use the encoder-decoder models that encode a given sequence of sentences with timestamps into internal representations as external-memory components and then decode mixed-type answers in response to a given question using multiple decoders and a decode selector.

Figure 2 shows an example of the proposed method with the event timeline when moving to the living room after preparing coffee and cookies in kitchen. Our framework answers various kinds of questions according to the content of the question. For example, the method has to answer the objects a person has when a question "What did the person hold?" is given. In addition, the method has to return a timestamp when a question "When did the person do some action".

These problems need to consider reasoning multiple facts in a sequence of sentences. To this end, we use Memory Networks as an encoder, which is a simple neural network with an external memory. Moreover, to generate mixed-type answers, we use several decoders that can solve multi-class and multi-label classification, regression, sequence generation, and time prediction problems. We introduce the encoder and the decoders in the following sections.

3.2 Memory Networks Encoder

We use Memory Networks [Weston et al., 2015; Sukhbaatar et al., 2015] for encoding multiple sentences that describe physical-world states. Our Memory Networks takes a discrete set of events e = [e1, . . . , e|e|] that is to be stored in the memory, a question q, and outputs an answer a, where q has timestamp qt when the question was issued, e has timestamp et , which is a start-time of an event when a subject does ADL (activity of daily living), and a is a tuple (a1, . . . , a|a|). Note that a becomes mixed-type data in response to the content of a question. For example, a denotes a number a = 3 when predicting how many objects he/she is holding. Each of the ei and q contains symbols coming from a dictionary with V words. Each answer a also comes from a dictionary with V words when predicting a word, word sequence and set.

Content-Based Memory Addressing

We convert all events ei into N real-valued vectors mi and m′ i for input and output of memory representations. Old events are discarded when |e| > M, where M is the memory size. We compute each mi and m′ i by embedding each ei in a continuous space using embedding matrices A,C ∈ RN×V , respectively. We also embed the question q with another embedding matrix B ∈ RN×V to obtain an internal state u. To read memory in response to question q, we use soft attention defined as follows:

where p ∈ RN is a vector of non-negative weights that sum to one. Each weight p is used for addressing content in memory related to a question. We define for the i-th read weight,

where Softmax(z)i = exp(zi)/ ∑j exp(zj). The weight pi can be seen as a probability vector over the inputs. We compute gradients in all rows of memory per time step.

Multi-Hop Addressing

We stack memory layers to handle l hop operations in Memory Networks. The input to layers above the first is the sum of the output ok and the input uk from layer k:

Each layer has its own embedding matrices Ak, Ck, used to embed the inputs. To ease training and reduce the number of parameters, we use adjacent weight. Thus, the output embedding for one layer is the input embedding for the one above, i.e. Ak+1 = Ck. In addition, we constrain the question embedding to match the input embedding of the first layer, i.e. B = A1. After memory addressing l times, we get the input embedding ul+1 that is a context vector c ∈ RN .

3.3 Mixed-Type Decoders

To respond to mixed-type answers, we use multiple decoders that transform the context vector c into mixed-type data such as word, word set, word sequence, number, and time (memory pointer). We also use a decode selector for selecting an optimal decoder suitable for each problem as in Figure 2. By using decode selectors, the framework selects decode outputs in response to the content of question. An index of the decoder for the final prediction is

where W d ∈ RD×N and D is the number of decoders. The final prediction is

where Decoderi(·) outputs i-th decoder. The parameter of the decode selector is updated by the cross-entropy between its outputs and the question-type label attached to each question-answer pair in the training data. We introduce each decoder in the following sections.

Word Decoder

To predict a single word as answer, we use the multi-class classification framework. The method predicts a label a = w from vocabulary V based on the context vector c. We use the weight matrix W c ∈ RV×N and a softmax function to produce the indicator that selects the word.

We update their parameters while minimizing a cross-entropy loss between the predicted probabilities and the true answers as Lword( ˆw, w) = - ∑i wi log( ˆwi).

Set Decoder

To predict a word set in response to a question, we use a multi-labeling approach. The method predicts a tuple a = (w1, . . . , wm), where m is the size of a word set and each w is from vocabulary V based on the context vector c. We use the weight matrix W m ∈ RV×N and a softmax to produce the output of the decoder as

where Sigmoid(zi) = 1/(1 + exp(-zi)). We select top m words with higher values in ˆw as an answer. To update parameters, we use the loss function that optimizes a multi-label one-versus-all loss defined as Lset( ˆw,w) = - ∑i{wi log(1 - ˆwi) + (1 - wi) log( ˆwi)}.

Sequence Decoder

For the word-sequence generation, we use a recurrent neural network language model (RNNLM) [Mikolov et al., 2010] that produces a word sequence a = [w1, w2, . . . , wn], where w from vocabulary V and n is the number of words in the answer. RNNLM updates the following equations:

where f(·) is a function of RNN, W s ∈ RV×O , O is the number of hidden units (N in our case), and b is a bias term. We initialize h0 with the context vector c. We use the cross entropy between wt+1 and ˆwt over {w1, . . . ,wt}, then Lseq({ ˆw1, . . . , ˆwn}, {w1, . . . ,wn}) = - ∑t wt+1 log( ˆwt).

Number Decoder

To predict a number, we solve the regression problem. The method predicts a label a = r ∈ R based on the context vector c. We use the weight vector wr ∈ RN as

where y ∈ R. When predicting an integral number, we cast the real number into the integer as an output. We use a squared loss Lnum(ˆ r, r) = (ˆ r - r)2 to update parameters.

Time Decoder

To solve the time prediction problem, we use a binary-class classification framework that indicate a specific index of memory (memory pointer) s ∈ [1, . . . ,M] relevant to the answer based on the context vector c. We use the weight matrix W s ∈ R(M+1)×N and a softmax function to produce the indicator that selects the output of the decoder:

where the index of M + 1 indicates out of memory. We update their parameters while minimizing a cross-entropy loss between the predicted probabilities and the true answers as Ltime(ˆ s, s) = - ∑i si log(ˆsi). This decoder outputs an timestamp of event a = ets when testing.

3.4 Self-Paced Learning for Multiple Decoders

We train the proposed encoder-decoders based on neural networks with multi-task learning, which is widely used for many natural language processing (NLP) tasks such as multilingual neural machine translation [Luong et al., 2016] and a joint learning of a wide variety of NLP tasks [Collobert and Weston, 2008; Hashimoto et al., 2017]. The advantage of multi-task learning of neural networks is that it can learn useful shared features for related tasks and robustly improve performance with limited data. However, each decoder in our proposed method has a different scale of loss values, since each decoder outputs a different type data.

To address this problem, we use self-paced learning for simultaneously learning our encoder-decoder model. We follow the self-paced learning for multi-task problems [Murugesan and Carbonell, 2017]. The objective function ˜ L of the self-paced learning for T tasks is given as:

where L(θt,Xt, yt) = 1 Nt ∑i∈Nt l(θi t ,Xit , yit ). Note that each L corresponds to Lword, Lset, Lseq, Lnum, and Ltime. l(θi t ,Xit , yit ) is each loss function of a i-th instance given that t-th task is associated with Nt training examples. Since our method of shared knowledge in the encoder and regularization term is included in each loss, we omit the regularization term over decoders. The learning algorithm considers a task easy if it has low training error. We define τt as

We set τt = δ for the hard tasks. We also use the parameter c that controls the learning pace of the self-paced procedure (i.e., λ ← cλ for each iteration). We set the c value as greater than 1 for relaxing threshold λ.

4 Experiments

In this section, we reports on the empirical evaluation for the proposed physical-world QA method on a daily activity dataset.

4.1 Dataset

We make a document collection describing various daily activities in a house-hold setting. To this end, we re-annotate a daily activity dataset that has been used in the egocentric video retrieval and the event timeline generation [Miyanishi et al., 2016; 2018] for physical-world QA problems. It contains first-person videos captured by a head-mounted wearable camera attached to eight subjects. They performed 20 continuous ADLs in six different places 10 times (i.e. a total of 10 sessions) under the semi-naturalistic collection protocol [Bao and Intille, 2004] for collecting variable activities. For example, the subject makes a cookie in the kitchen, moves to the living room with it, and eats it after sitting on the sofa. Total video length is about 17 hours. We define physical-world events in one session as a single story indicating that each subject has 10 stories. To evaluate physicalworld QA methods, we re-annotated this dataset with structured event descriptions of the ADLs that subjects performed in a house. We obtained 11,497 sentences of ADL events with 1,031 unique events. For QA tasks, we prepared 22 question and answer templates that ask about the state of the physicalworld. Each template is categorized by question types corresponding to each decoder output. We prepared questions and their answers on the collected stories using these templates. Table 1 shows the template of questions and answers we used for generating question-and-answer pairs of each task. We randomly insert 25 questions into each story. If there are no answers to questions, an "unknown" token is used for answers. In total, our QA dataset, which we call a daily living QA dataset, has 2,000 questions for each question answer template. Figure 3 shows examples of our QA tasks.

4.2 Baselines

We prepared the proposed method and several baselines based on the encoder-decoder model:

• RNN+RNN [Sutskever et al., 2014] is a standard sequence-to-sequence model that encodes the story until the point it reaches a question and then decodes word sequence as an answer.
• RNN+MixDecs encodes a story with RNN and then decodes mixed-type answers in response to a question using multiple decoders.
• MemNN+RNN [Sukhbaatar et al., 2015] is standard Memory Networks that encodes a story with multiple sentence embeddings and addresses them in response to a question embedding and then decodes word sequence as an answer.
• MemNN+MixDecs encodes a story and a question with Memory Networks into a context vector and decodes it to mixed-type answers using multiple decoders.
• MemNN+MixDecs+SelfPaced is equal to MemNN+MixDecs except that self-paced learning is used when learning this model.

RNN+RNN and RNN+MixDecs have a disadvantage with regard to encoding a long story compared to MemNN+RNN and MemNN+MixDecs, which use external memory. Moreover, they cannot answer questions about time since RNN simply reads the story as a sequence of words and does not distinguish each physical-world event. RNN+RNN and MemNN+RNN use RNN for generating an answer, so they cannot predict continuous numbers such as time. Thus, the use of multiple decoders enables us to use more training examples for multi-task learning. We also prepared MemNN+MixDecs+SelfPaced for whether the self-paced learning contributes our QA tasks.

4.3 Experimental Settings

We used the QA dataset about four subjects for a training set and two subjects for a validation set, which was used to tune model- and hyper-parameters. We trained all methods using a learning rate 0.01 and a batch size of 128 until 10 epochs were reached. The weights were initialized randomly from a Gaussian distribution N(0, 0.01). A null symbol was used to pad them all to a fixed size. The embedding of the null symbol was constrained to be zero. For all RNN encoders and decoders, we used a gated recurrent unit (GRU) [Cho et al., 2014] with two stacked hidden layers and a bidirectional RNN [Schuster and Paliwal, 1997]. For all methods, we make the dimension of the hidden states equal to the dimension of the word embeddings and select the the dimension of states from N = {128, 256, 512}. We used a dropout [Srivastava et al., 2014] rate of 0.5 for the outputs of each RNN layer, and the linear transformation be- Task 1:

02-17 10

02-17 10

02-17 10

Task 11:

02-17 10

02-17 10

02-17 10

02-17 10

02-17 10

Task 14:

02-24 14

02-24 14

02-24 14

02-24 14

02-24 14

02-24 14

Task 16:

03-02 14

03-02 14

03-02 14

03-02 14

03-02 14

03-02 14

Task 20:

03-03 10

03-03 10

03-03 10

03-03 10

03-03 10

03-03 10

fore the softmax for the answers except for the number decoder. For MemNN+RNN and MemNN+MixDecs, we set a fixed number (l = 3 hops) in the encoding of Memory Networks. For MemNN+MixDecs, we also used self-paced learning selecting parameter candidates among λ = {0.1, 1}, δ = {0.01, 0.1}, and c = {1.01, 1.1}. We evaluated all methods under the ParlAI framework [Miller et al., 2017] and used F1 measure as an evaluation metric. We separately trained and predicted when the question type is Time or not since the RNN encoder does not have explicit memory to point out a timestamp. Note that we simultaneously learn multiple decoders unless otherwise noted.

4.4 Results

In this section, we describe the experimental results over all methods when using the daily living QA dataset. Table 2 shows the QA performances across 19 tasks except for the Time question type using RNN+RNN, RNN+MixDecs, MemNN+RNN, MemNN+MixDecs. The overall results showed that MemNN+RNN and MemNN+MixDecs outperformed RNN+RNN and RNN+MixDecs respectively indicating that memory components mounted on Memory Networks significantly improve the QA performance on daily episodes, which requires short-term and longterm dependencies in QA compared to RNN. These results are similar to using the simulation dataset introduced in past work [Sukhbaatar et al., 2015]. With regard to decoder performance, MemNN+MixDecs outperformed MemNN+RNN on average when used with and without time questions. The results suggest that using multiple decoders with memory components is effective for a wide variety of questions compared to decodes with RNN that generate only word sequence. Table 3 shows the QA performances across all 22 tasks. The results show that MemNN+MixDecs+SelfPaced outperformed MemNN+MixDecs, suggesting that weighting decoders with self-paced learning is effective when learning multiple decoders for our QA tasks.

Table 2 also shows the QA performances on each question type except for the Time question type. Across all types of answers, MemNN+RNN and MemNN+MixDecs outperformed RNN+MixDecs and RNN+RNN. The results indicate that memory components are essential for improving QA performance for all types of question. From the results in table Sequence and Number, questions are relatively difficult. MemNN+MixDecs+SelfPaced improved these questions. We assumed that self-paced learning, which learns more difficult problems after first learning simple problems, improves this performance. The results of MemNN+MixDecs+SelfPaced outperforming MemNN+MixDecs support this idea.

Table 4 shows the results of learning the encoder-decoder model by each question type or joint learning of various questions. We assumed that jointly learning multiple problems despite using mixed-type decoders is good for solving various questions since this method can learn useful features in the shared encoder for related tasks augmenting training data and then improve its performance. Comparing MemNN+MixDecs (Each) to MemNN+MixDecs (Joint) and MemNN+MixDecs+SelfPaced (Joint), the results show that joint learning is effective for improving most question types. The results indicates that the multi-task learning on different type QA problems is highly effective for solving physical-world QA problems.

5 Conclusion

We have shown that physical-world QA tasks answer various questions by reading physical-world events in daily episodes. To solve this problem, we proposed an encoderdecoder model combining memory components and multitask learning for mixed-type decoders. The experimental results of using a daily living QA dataset indicate that multitask learning of several different QA problems is suitable for solving problems that are important for the physical-world QA tasks.

Acknowledgments

This work was supported by JST CREST Grant Number JPMJCR15E2 and JSPS KAKENHI Grant Number 16K21718.