Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling

1 Introduction

Equipping deep neural networks (DNN) with attention mechanisms provides an effective and parallelizable approach for context fusion and sequence compression. It achieves compelling time efficiency and state-of-the-art performance in a broad range of natural language processing (NLP) tasks, such as neural machine translation [Bahdanau et al., 2015; Luong et al., 2015], dialogue generation [Shang et al., 2015], machine reading/comprehension [Seo et al., 2017], natural language inference [Liu et al., 2016], sentiment classification [Li et al., 2017b], etc. Recently, some neural nets based solely on attention, especially self-attention, outperform traditional recurrent [Bowman et al., 2015] or convolutional [Dong et al., 2017] neural networks on NLP tasks, such as machine translation [Vaswani et al., 2017] and sentence embedding [Shen et al., 2018], which further demonstrates the power of attention mechanisms in capturing contextual dependencies.

Soft and hard attention are the two main types of attention mechanisms. In soft attention [Bahdanau et al., 2015], a categorical distribution is calculated over a sequence of elements. The resulting probabilities reflect the importance of each element and are used as weights to produce a context-aware encoding that is the weighted sum of all elements. Hence, soft attention only requires a small number of parameters and less computation time. Moreover, soft attention mechanism is fully differentiable and thus can be easily trained by endto-end back-propagation when attached to any existing neural net. However, the softmax function usually assigns small but non-zero probabilities to trivial elements, which will weaken the attention given to the few truly significant elements.

Unlike the widely-studied soft attention, in hard attention [Xu et al., 2015], a subset of elements is selected from an input sequence. Hard attention mechanism forces a model to concentrate solely on the important elements, entirely discarding the others. In fact, various NLP tasks solely rely on very sparse tokens from a long text input. Hard attention is well suited to these tasks, because it overcomes the weaknesses associated with soft attention in long sequences. However, hard attention mechanism is time-inefficient with sequential sampling and non-differentiable by virtue of their combinatorial nature. Thus, it cannot be optimized through back-propagation and more typically rely on policy gradient, e.g., REINFORCE [Williams, 1992]. As a result, training a hard attention model is usually an inefficient process - some even find convergence difficult - and combining them with other neural nets in an end-to-end manner is problematic.

However, soft and hard attention mechanisms might be integrated into a single model to benefit each other in overcoming their inherent disadvantages, and this notion motivates our study. Specifically, a hard attention mechanism is used to encode rich structural information about the contextual dependencies and trims a long sequence into a much shorter one for a soft attention mechanism to process. Conversely, the soft one is used to provide a stable environment and strong reward signals to help in training the hard one. Such method would improve both the prediction quality of the soft attention mechanism and the trainability of the hard attention mechanism, while boosting the ability to model contextual dependencies. To the best of our knowledge, the idea of combining hard and soft attention within a model has not yet been studied. Existing works focus on only one of the two types.

In this paper, we first propose a novel hard attention mechanism called "reinforced sequence sampling (RSS)", which selects tokens from an input sequence in parallel, and differs from existing ones in that it is highly parallelizable without any recurrent structure. We then develop a model,"reinforced self-attention (ReSA)", which naturally combines the RSS with a soft self-attention. In ReSA, two parameter-untied RSS are respectively applied to two copies of the input sequence, where the tokens from one and another are called dependent and head tokens, respectively. ReSA only models the sparse dependencies between the head and dependent tokens selected by the two RSS modules. Finally, we build an sentence-encoding model, "reinforced self-attention network (ReSAN)", based on ReSA without any CNN/RNN structure.

We test ReSAN on natural language inference and semantic relatedness tasks. The results show that ReSAN achieves the best test accuracy among all sentence-encoding models on the official leaderboard of the Stanford Natural Language Inference (SNLI) dataset, and state-of-the-art performance on the Sentences Involving Compositional Knowledge (SICK) dataset. Compared to the commonly-used models, ReSAN is more efficient and has better prediction quality than existing recurrent/convolutional neural networks, selfattention networks, and even well-designed models (e.g., semantic tree or external memory based models). All the experiments codes are released at https://github.com/ taoshen58/DiSAN/tree/master/ReSAN.

Notation: 1) lowercase denotes a vector; 2) bold lowercase denotes a sequence of vectors (stored as a matrix); and 3) uppercase denotes a matrix or a tensor.

2 Background

2.1 Attention

Given an input sequence x = [x1, . . . , xn] ∈ Rde×n (xi ∈ Rde denotes the embedded vector of i-th element), and the vector representation of a query q, an vanilla attention mechanism uses a parameterized compatibility function f(xi, q) to computes an alignment score between q and each token xi as the attention of q to xi [Bahdanau et al., 2015]. A softmax function is then applied to the alignment scores a ∈ Rn over all tokens to generate a categorical distribution p(v|x, q), where v = i implies that token xi is selected according to its relevance to query q. This can be formally written as

The output of attention, s, is the expectation of sampling a token according to the categorical distribution p(v|x, q), i.e.,

Multi-dimensional (multi-dim) attention mechanism [Shen et al., 2018] extends the vanilla one [Bahdanau et al., 2015] to a feature-wise level, i.e., each feature of every token has an alignment score. Hence, rather than a scalar, the output of f(xi, q) is a vector with the same dimensions as the input, and the resulting alignment scores compose a matrix a ∈ Rde×n . Such feature-level attention has been verified in terms of its ability to capture the subtle variances of different contexts.

2.2 Self-Attention

Self-attention is a special case of attention where the query q stems from the input sequence itself. Hence, self-attention mechanism can model the dependencies between tokens from the same sequence. Recently, a variety of self-attention mechanisms have been developed, each serving a distinct purpose, but most can be roughly categorized into two types, token2token self-attention and source2token self-attention.

Token2token self-attention mechanisms aim to produce a context-aware representation for each token in light of its dependencies on other tokens in the same sequence. The query q is replaced with the token xj, and the dependency of xj on another token xi is computed by f(xi, xj). There are two proposed self-attentions in this type, i.e., scaled dot-product attention which composes the multi-head attention [Vaswani et al., 2017] and masked self-attention which leads to directional self-attention [Shen et al., 2018]. Because the latter experimentally outperforms the former, we select the masked self-attention as our fundamental soft self-attention module.

Masked Self-Attention is more sophisticated than scaled dot-product attention in that, it uses multi-dim and multilayer perceptron with an additional position mask, rather than a scaled dot-product, as the compatibility function, i.e.,

where c is a scalar and M is the mask with each entry Mij ∈ {-∞, 0}. When Mij = -∞, applying the softmax function to a results in a zero probability, p(z = i|x, xj) = 0, which switches off the attention of xj to xi. An asymmetric mask where Mij ̸= Mji enforces directional attention between xi and xj, which can encode temporal order information. Two positional masks have been designed to encode the forward and backward temporal order, respectively, i.e.,

In forward and backward masks, Mii = -∞. Thus, the attention of a token to itself is blocked, so the output of masked self-attention mechanism comprises the features of the context around each token rather than context-aware features.

Directional self-attention uses a fusion gate to combine the embedding of each token with its context. Specifically, a fusion gate combines the input and output of a masked selfattention to produce context-aware representations. This idea is similar to the highway network [Srivastava et al., 2015].

Source2token self-attention mechanisms [Shen et al., 2018] remove the query q from the compatibility function in Eq.(1) and directly compresses a sequence into a vector representation calculated from the dependency between each token xi and the entire input sequence x. Hence, this form of self-attention is highly data- and task- driven.

3 ProposedModels

This section begins by introducing a hard attention mechanism called RSS in Section 3.1, followed by integrating the RSS with a soft self-attention mechanism into a context fusion model called ReSA in Section 3.2. Finally, a model named ReSAN, based on ReSA, is designed for sentence encoding tasks in Section 3.3

3.1 Reinforced Sequence Sampling (RSS)

The goal of hard attention mechanism is to select a subset of critical tokens that provides sufficient information to complete downstream tasks, so any further computations on the trivial tokens can be saved. In the following, we introduce a hard attention mechanism called RSS. Given an input sequence x = [x1, . . . , xn], RSS generates an equal-length sequence of binary random variables z = [z1, . . . , zn] where zi = 1 implies that xi is selected whereas zi = 0 indicates that xi is discarded. In RSS, the elements of z are sampled in parallel according to probabilities computed by a learned attention mechanism. This is more efficient than using MCMC with iterative sampling. The particular aim of RSS is to learn the following product distribution.

The function f(·; θf ) denotes a context fusion layer, e.g., BiLSTM, Bi-GRU, etc., producing context-aware representation for each xi. Then, g(·; θg) maps f(·; θf) to the probability of selecting the token. Note we can sample all zi for different i in parallel because the probability of zi (i.e., whether xi is selected) does not depends on zi-1. This is because the context features given by f(·; θf) already take the sequential information into account, so the conditionally independent sampling does not discard any useful information.

To fully explore the high parallelizability of attention, we avoid using recurrent models in this paper. Instead we apply a more efficient f (·; θf ) inspired by source2token self-attention and intra-attention [Liu et al., 2016], i.e.,

where ⊙ denotes the element-wise product, and the pooling(·) represents the mean-pooling operation along the sequential axis. RSS selects a subset of tokens by sampling zi according to the probability given by g(hi; θg) for all i = 1, 2, . . . , n in parallel.

For the training of RSS, there are no ground truth labels to indicate whether or not a token should be selected, and the discrete random variables in z lead to a non-differentiable objective function. Therefore, we formulate learning the RSS parameter θr as a reinforcement learning problem, and apply the policy gradient method. Further details on the model training are presented in Section 4.

3.2 Reinforced Self-Attention (ReSA)

The fundamental idea behind this paper is that the hard and soft attention mechanisms can mutually benefit each other to overcome their inherent disadvantages via interaction within an integrated model. Based on this idea, we develop a novel self-attention termed ReSA. On the one hand, the proposed RSS provides a sparse mask to a self-attention module that only needs to model the dependencies for the selected token pairs. Hence, heavy memory loads and computations associated with soft self-attention can be effectively relieved. On the other hand, ReSA uses the output of the soft self-attention module for prediction, whose correctness (as compared to the ground truth) is used as reward signal to train the RSS. This alleviates the difficulty of training hard attention module.

Figure 1 shows the detailed architecture of ReSA. Given the token embedding in an input sequence, x = [x1, . . . , xn], ReSA aims to produce token-wise context-aware representations, u = [u1, . . . , un]. Unlike previous self-attention mechanisms, ReSA only selects a subset of head tokens, and generates their context-aware representations by only relating each head token to a small subset of dependent tokens. This notion is based on the observation that for many NLP tasks, the final prediction only relies on a small set of key words and their contexts, and each key word only depends on a small set of other words. Namely, the dependencies between tokens from the same sequence are sparse.

In ReSA, we use two RSS modules, as outlined in Section 3.1, to generate two sequences of labels for the selections of head and dependent tokens, respectively, i.e.,

We use ˆ zh and ˆ zd sampled from the two independent (parameter untied) RSS to generate an n × n mask M rss , i.e.,

The resulting mask is then applied as an extra mask to the masked self-attention mechanism introduced in Section 2.2. Specifically, we add M rss to Eq.(4) and use

to generate the alignment scores. For each head token xj, a softmax function is applied to f rss (·, xj), which produces a categorical distribution over all dependent tokens, i.e.,

The context features of xj is computed by

where ⊙ denotes a broadcast product in the vanilla attention or an element-wise product in the multi-dim attention.

For a selected head token, as formulated in Eq.(10), the attention from a token to itself is disabled in M rss , so the sj for the selected head token encodes only the context features but not the desired context-ware embedding. For an unselected head token xj with ˆ zh j = 0, its alignment scores over all dependent tokens are equal to -∞, which leads to the equal probabilities in P j produced by the softmax function. Hence, sj for each unselected token xj can be regarded as the result of mean-pooling over all dependent tokens.

To merge the word embedding with its context feature for the selected heads, and distinguish the representations from others for the unselected heads, a fusion gate is used to combine s with the input embedding x in parallel and generate the final context-aware representations for all tokens, i.e.,

where W (f) , b(f) are the learnable parameters. The contextaware representations, u = [u1, . . . , un], are final output. One primary advantage of ReSA is that it generates better predictions using less time and memory than existing selfattention mechanisms. In particular, major computations of ReSA are 1) the inference of self-attention over a shorter subsequence, and 2) the mean-pooling over the remaining elements. This is much more time- and memory- efficient than computing the self-attention over the entire input sequence.

3.3 Applications of the ProposedModels

To adapt ReSA for sentence encoding tasks, we build an RNN/CNN-free network, called reinforced self-attention network (ReSAN), which is solely based on ReSA and source2token self-attention (Section 2.2). In particular, we pass the output sequence of ReSA into a source2token selfattention module to generate a compressed vector representation, e ∈ Rde , which encodes the semantic and syntactic knowledge of the input sentence and can be used for various downstream NLP tasks.

Further, we propose two simplified variants of ReSAN with a simpler structure or fewer parameters, i.e., 1) ReSAN w/o unselected heads which only applies the soft self-attention to the selected head and dependent tokens, and 2) ReSAN w/o dependency restricted which use only one RSS to select tokens for both heads and dependents. Both variants entirely discard the information of the unselected tokens and hence are more time-efficient. However, neither can be used for context fusion, because the input and output sequences are not equal in length.

4 Model Training

The parameters in ReSAN can be divided into two parts, θr for the RSS modules and θs for the rest parts which includes word embeddings, soft self-attention module, and classification/regression layers. Learning θs is straightforward and can be completed by back-propagation in an end-to-end manner. However, Optimizing θr is more challenging because the RSS modules contain discrete variables z and, thus, the objective function is non-differentiable w.r.t. θr.

In supervised classification settings, we use the crossentropy loss plus L2 regularization penalty as the loss, i.e.,

where (x∗ , y∗ ) denotes a sample from dataset D. The loss above is used for learning θs by back-propagation algorithm.

Optimizing θr is formulated as a reinforcement learning problem solved by the policy gradient method (i.e., REINFORCE algorithm). In particular, RSS plays as an agent and takes action of whether to select a token or not. After going through the entire sequence, it receives a loss value from the classification problem, which can be regarded as the negative delay reward to train the agent. Since the overall goal of RSS is to select a small subset of tokens for better efficiency and meanwhile retain useful information, a penalty limiting the number of selected tokens is included in the reward R, i.e.,

where λ is the penalty weight and is fine-tuned with values from {0.005, 0.01, 0.02} in all experiments. Then, the objective of learning θr is to maximize the expected reward, i.e.,

where the ˆ z = (ˆ zh, ˆ zd ) ∼ p(zh|x∗ ; θrh)p(zd |x∗ ; θrd) ≜ π(ˆ z; x∗ ; θr) and N is sample number in the dataset. Based on REINFORCE, the policy gradient of Jr(θr) w.r.t θr is

Although theoretically feasible, it is not practical to optimize θs and θr simultaneously, since the neural nets cannot provide accurate reward feedback to the hard attention at the beginning of the training phrase. Therefore, in early stage, the RSS modules are not updated, but rather forced to select all tokens (i.e., z = 1 ). And, θs is optimized for several beginning epochs until the loss over development set does not decrease significantly. The resulting ReSAN now can provide a solid environment for training RSS modules through reinforcement learning. θr and θs are then optimized simultaneously to pursue a better performance by selecting critical token pairs and exploring their dependencies.

Training Setup: All experiments are conducted in Python with Tensorflow and run on a Nvidia GTX 1080Ti. We use Adadelta as optimizer, which performs more stable than Adam on ReSAN. All weight matrices are initialized by Glorot Initialization [Glorot and Bengio, 2010] and the biases are initialized as zeros. We use 300D GloVe 6B pre-trained vectors [Pennington et al., 2014] to initialize the word embeddings [Liu et al., 2018]. The words which do not appear in GloVe from the training set are initialized by sampling from uniform distribution between [-0.05, 0.05]. We choose Dropout [Srivastava et al., 2014] keep probability from {0.65, 0.70, 0.75, 0.8} for all models and report the best result. The weight decay factor γ for L2 regularization is set to 5 × 10-5 . The number of hidden units is 300.

5 Experiments

We implement ReSAN, its variants and baselines on two NLP tasks, language inference in Section 5.1 and semantic relatedness in Section 5.2. A case study is then given to provide the insights into model.

The baselines are listed as follows: 1) Bi-LSTM: 600D bi-directional LSTM (300D forward LSTM + 300D backward LSTM) [Graves et al., 2013]; 2) Bi-GRU: 600D bidirectional GRU [Chung et al., 2014]; 3) Multi-window CNN: 600D CNN sentence embedding model (200D for each of 3, 4, 5-gram) [Kim, 2014]; 4) Hierarchical CNN: 3layer 300D CNN [Gehring et al., 2017] with kernel length 5. GLU [Dauphin et al., 2016] and residual connection [He et al., 2016b] are applied; 5) Multi-head: 600D multi-head attention (8 heads, each has 75 hidden units), where the positional encoding method is applied to the input [Vaswani et al., 2017]; 6) DiSAN: 600D directional self-attention network (forward+backward masked self-attn) [Shen et al., 2018].

5.1 Natural Language Inference

The goal of natural language inference is to infer the semantic relationship between a pair of sentences, i.e., a premise and the corresponding hypothesis. The possible relationships are entailment, neutral or contradiction. This experiment is conducted on the Stanford Natural Language Inference [Bowman et al., 2015] (SNLI) dataset which consists of 549,367/9,842/9,824 samples for training/dev/test.

In order to apply sentence encoding model to SNLI, we follow Bowman et al. [2016] and use two parameter-tied sentence encoding models to respectively produce the premise and the hypothesis encodings, i.e., sp , sh . Their semantic relationship is represented by the concatenation of sp , sh , sp-sh and sp⊙sh , which is passed to a classification module to generate a categorical distribution over the three classes.

The experimental results for different methods from leaderboard and our baselines are shown in Table 1. Compared to the methods from official leaderboard, ReSAN outperforms all the sentence encoding based methods and achieves the best test accuracy. Specifically, compared to the last best models, i.e., 600D Gumbel TreeLSTM encoders and 600D Residual stacked encoders, ReSAN uses far fewer parameters with better performance. Moreover, in contrast to the RNN/CNN based models with attention or memory module, ReSAN uses attention-only modules with equal or fewer parameters but outperforms them by a large margin, e.g., 600D Bi-LSTM + intra-attention (+3.0%), 300D NSE encoders (+1.7%) and 600D Deep Gated Attn (+0.8%). Furthermore, ReSAN even outperforms the 300D SPINN-PI encoders by 3.1%., which is a recursive model and uses the result of an external semantic parsing tree as an extra input.

In addition, we compare ReSAN with recurrent, convolutional, and attention-only baseline models in terms of the number of parameters, training/inference time and test accuracy. Compared to the recurrent models (e.g., Bi-LSTM and Bi-GRU), ReSAN shows better prediction quality and more compelling efficiency due to parallelizable computations. Compared to the convolutional models (i.e., Multiwindow CNN and Hierarchical CNN), ReSAN significantly outperforms them by 3.1% and 2.4% respectively due to the weakness of CNNs in modeling long-range dependencies. Compared to the attention-based models, multi-head attention and DiSAN, ReSAN uses a similar number of parameters with better test performance and less time cost.

Further, we conduct an ablation study of ReSAN, as shown in Table 2, to evaluate the contribution of each component. One by one, each component is removed and the changes in test accuracy are recorded. In addition to the two variants of ReSAN introduced in Section 3.3, we also remove 1) the hard attention module, 2) soft self-attention module and 3) both hard attention and soft self-attention modules. In terms of prediction quality, the results show that 1) the unselected head tokens do contribute to the prediction, bringing 0.2% improvement; 2) using separate RSS modules to select the head and dependent tokens improves accuracy by 0.5%; and 3) hard attention and soft self-attention modules improve the accuracy by 0.3% and 2.9% respectively. In terms of inference time, it shows that 1) the two variants are more timeefficient but have poorer performance; and 2) applying the RSS modules to self-attention or attention improves not only performance but also time efficiency.

5.2 Semantic Relatedness

Semantic relatedness aims to predict the similarity degree of a given pair of sentences, which is formulated as a regression problem. We use s1 and s2 to denote the encodings of the two sentences, and assume the similarity degree is a scalar between [1,K]. Following Tai et al. [2015], the relationship between the two sentences is represented as a concatenation of s1⊙s2 and |s1-s2 |. The representation is fed into a classification module with K-way categorical distribution output. We implement ReSAN and baselines on the Sentences Involving Compositional Knowledge [Marelli et al., 2014] (SICK) dataset, which provides the ground truth as similarity degree between [1, 5]. SICK come with a standard training/dev/test split of 4,500/500/4,927 samples.

The results in Table 3 show that the ReSAN achieves state-of-the-art or competitive performance for all three metrics. Particularly, ReSAN outperforms the feature engineering method by a large margin, e.g., Meaning Factory and ECNU. ReSAN also significantly outperforms the recursive models, which is widely used in semantic relatedness task, especially ones that demand external parsing results, e.g., DT/SDT-RNN and Tree-LSTMs. Further, ReSAN achieves the best results among all the recurrent, convolutional and self-attention models listed as baselines. This thoroughly demonstrates the capability of ReSAN in context fusion and sentence encoding.

5.3 Case Study

To gain an insights into how the hard/soft attention and fusion gate work within ReSA, we visualize their resulting values in this section. Note that only the values at token level are illustrated. If the attention probabilities and the gate values are feature-level, we average the probabilities over all features.

Two sentences from the SNLI test set serve as examples for this case study: 1) "The three men sit and talk about their lives." and 2) "A group of adults are waiting for an event.".

The head and dependent tokens selected by RSS modules are show in Figure 2 (a small square with color white denotes unselection and vice versa). It shows that more dependent tokens are selected than the head tokens, because all non-trivial dependents should be retained to adequately modify the corresponding heads, e.g., three, their in sentence 1 and group in sentence 2, whereas only the key heads should be kept to compose the trunk of a sentence. It also shows that most stop words (i.e., articles, conjunctions, prepositions, etc.) are selected as neither head tokens nor dependent tokens.

We also visualize the probability distributions of the soft self-attention module in Figure 2 (the depth of color blue). From the figure, we observe that 1) the semantically important words (e.g., noun and verb) usually receive great attention from all the other tokens, e.g., sit, talk, lives in sentence 1 and adults, waiting, event in sentence 2; and 2) the attention score increases if the token pair can be constituted to a sense-group, e.g., (sit, talk) in sentence 1 and (adults, waiting), (waiting, event) in sentence 2.

6 RelatedWork

Applying reinforcement learning (RL) to natural language processing (NLP) tasks recently attracts enormous interests for two main purposes, i.e., optimizing the model according to non-differentiable objectives and accelerating the model speed. Lei et al. [2016] propose a method to select a subset of a review passage for sentiment analysis from a specific aspect. He et al. [2016a] use RL method to fine-tune a bilingual machine translation model by well-trained monolingual language models. Yogatama et al. [2016] use builtin transition-based parsing module to generate semantic constituency parsing tree for downstream NLP tasks by using RL. Yu et al. [2017] propose a RL-based skim reading method, which is implemented on recurrent models, to skim the insignificant time slots to achieve higher time efficiency. Choi et al. [2017a] separately implement a hard attention or a soft attention on a question answering task to generate the document summary. Shen et al. [2017] use dynamic episode number determined by RL rather than fixed one to attend memory for efficient machine comprehension. Hu et al. [2017] employ policy gradient method to optimize the model for nondifferentiable objectives of machine comprehension, i.e., F1 score of matching the prediction with the ground truth. Li et al. [2017a] propose a service dialog system to sell movie tickets, where the agent in RL is used to select which user's information should be obtained in next round for minimum number of dialog rounds to sell the ticket. Zhang and Lapata [2017] simplify a sentence with objectives of maximum simplicity, relevance and fluency, where all three objectives are all non-differentiable w.r.t the parameters of model.

7 Conclusions

This study presents a context fusion model, reinforced selfattention (ReSA), which naturally integrates a novel form of highly-parallelizable hard attention based on reinforced sequence sampling (RSS) and soft self-attention mechanism for the mutual benefit of overcoming the intrinsic weaknesses associated with hard and soft attention mechanisms. The hard attention modules could be used to trim a long sequence into a much shorter one and encode rich dependencies information for a soft self-attention mechanism to process. Conversely, the soft self-attention mechanism could be used to provide a stable environment and strong reward signals, which improves the feasibility of training the hard attention modules. Based solely on ReSA and a source2token self-attention mechanism, we then propose an RNN/CNN-free attention model, reinforced self-attention network (ReSAN), for sentence encoding. Experiments on two NLP tasks - natural language inference and semantic relatedness - demonstrate that ReSAN deliver a new best test accuracy for the SNLI dataset among all sentence-encoding models and state-of-theart performance on the SICK dataset. Further, these results are achieved with equal or fewer parameters and in less time.

Acknowledgments

This research was funded by the Australian Government through the Australian Research Council (ARC) under grants 1) LP160100630 partnership with Australia Government Department of Health and 2) LP150100671 partnership with Australia Research Alliance for Children and Youth (ARACY) and Global Business College Australia (GBCA). We also acknowledge the support of NVIDIA Corporation and MakeMagic Australia with the donation of GPU.