Scanpath Prediction for Visual Attention using IOR-ROI LSTM

1 Introduction

Selective visual attention mechanism of human vision system automatically extracts information from a subset of visual input by shifting our eyes to bring region of interest (ROI) onto the fovea where fine-grained analysis can be carried out. Most studies concerning selective visual attention are dealing with overt attention involving eye movements [Zhang and Lin, 2013]. Successive eye movements during scene exploration, called the visual scanpaths, are comprised of a series of saccades and fixations. Fixations indicate ROIs of observers and saccades represent rapid changes of gaze. Unlike saliency models that aim to measure the probability distribution of fixations with respect to the spatial layout of a stimulus, visual scanpath prediction models can record not only attention locations but also the order among fixations. Understanding and predicting scanpaths is a challenge yet important task, potentially contributing to advancements of vision research, visual art design, robot vision, virtual reality, which has received great research interests recently [Boccignone, 2016; Le Meur et al., 2017].

Visual inhibition of return (IOR) facilitates foraging by discouraging re-examination of recently fixated locations and objects [Samuel and Kat, 2003; Wang and Klein, 2010]. Due to the limited capacity of the visual working memory (VWM), effects of IOR gradually fade over time, which allows the possibility to re-fixate previously attended regions. Such a process is fundamental to computational models of selective visual attention [Bays and Husain, 2012]. [Itti et al., 1998] firstly implemented a bottom-up saliency model that outputs a sequence of fixations using a winner-take-all neural network and IOR. [Wang et al., 2011] introduced a saccadic scanpath prediction model based on information maximization criteria, which incorporates the VWM to integrate fixated regions and to forget earlier ones at a constant rate. [Le Meur and Liu, 2015] proposed a new framework to predict visual scanpaths by modeling bottom-up saliency, oculomotor biases, and IOR. [Jiang et al., 2016] presented a reinforcement learning approach, also employed IOR, with leastsquares policy iteration to predict a sequence of human fixations, simulating the recorded eye-fixation examples.

This work was supported in part by the National Natural Science Foundation of China under Grant 61771348 and 61471273.

A few studies have been conducted to integrate deep learning frameworks into scanpath generation. [Assens Reina et al., 2017] introduced the saliency volume to generate scanpath on 360◦ images by sampling fixations from each temporal saliency slice in the saliency volume. But the accuracy of the generated scanpath heavily depends on the quality of saliency volume and sampling strategies. [Ngo and Manjunath, 2017] proposed a model that generates scanpath using both convolutional neural network (CNN) and recurrent neural network (RNN). High-level image features extracted by a pre-trained CNN are sent to RNN which predicts fixation transition probability from the current position to possible locations. This model utilized the standard LSTM [Hochreiter and Schmidhuber, 1997] to simulate the VWM, but the input to the LSTM is essentially a one-dimensional vector discarding relative spatial information which is indispensable to simulate human vision.

In this paper, we propose a bio-inspired and interpretable scanpath prediction architecture based on convolutional LSTM (ConvLSTM) [Shi et al., 2015] that incorporates a novel IOR-LSTM along with ROI-LSTM to capture IOR dynamics and gaze shift behavior simultaneously. Results on the OSIE dataset [Xu et al., 2014] and MIT dataset [Judd et al., 2009] have verified the effectiveness of our model.

2 Model Architecture

Figure 1 shows the overview of the proposed scanpath prediction architecture. There are three major components presented in this model: an image feature extractor, the IOR-ROI LSTMmodule and a saliency guidance network. The image feature extractor is responsible for synthesizing the foveated image and extracting visual feature maps using CNN. The IOR-ROI LSTM plays a critical role in predicting the next eminent ROI to which subjects tend to pay attention. The saliency guidance network is only employed during the training phase, forcing the feature extractor to encode more global saliency information. A new prediction loop is started by updating the foveated image using the latest fixation selected from the predicted ROI. In order to ensure that the IOR-ROI LSTM module can effectively learn visual IOR pattern, predicted IOR map is used to compute distance against its corresponding ground truth as an auxiliary training loss which will be jointly optimized with the ROI prediction loss.

2.1 Foveated Image and Image Feature Extractor

Thanks to the foveal system of human eyes, human can concentrate on a specific region with significant detail [Wandell, 1995]. The most area of the retina outside the foveal region is called peripheral vision which only occupies limited resources of the brain by perceiving visual information in a low-resolution manner. In our work, we employ the computational foveation model described in [Wang et al., 2017] and implement it on the modern GPU to efficiently simulate the foveation process of the human vision system.

The image feature extractor can be any form of CNNs, e.g., VGG-Net [Simonyan and Zisserman, 2014] and ResNet [He et al., 2016]. We employ all convolutional and pooling layers of a pre-trained VGG-19 network to extract convolutional features. However, original VGG-19 is composed of five blocks where each of them is followed by a max pooling layer with the stride of 2×2, which considerably reduces spatial resolution of the resulting feature maps by a factor of 32. This could significantly degrade the quality of the feature maps. To circumvent this issue, we modify the original VGG19 structure to produce feature maps with a downscale factor of 8, while maintaining the same receptive field of convolutional kernels. Last two max-pooling layers of the original VGG-19 is discarded and convolution operation in block 5 is replaced with dilated convolution [Yu and Koltun, 2016] which has kernel size of 3 × 3 and dilation rate of 2.

2.2 IOR-ROI LSTMModule

IOR behavior is individual dependent and reacts differently to various visual features [Hotta et al., 2010]. Thus, it is not bioplausible to mimic IOR using linear models which are usually integrated into computational visual attention and gaze shift models [Itti et al., 1998; Wang et al., 2011; Le Meur and Liu, 2015; Wang et al., 2017]. To this end, we devise the IOR-ROI LSTMmodule that is capable of capturing IOR dynamics and gaze shift behavior of human observers simultaneously. As shown in Figure 2, the IOR-ROI LSTMmodule is designed to incorporate inhibited attentive mechanism in predicting ROI from which the next fixation is generated.

To be self-contained, basic knowledge of ConvLSTM is first introduced. In ConvLSTM, fully connected structures in standard LSTM are substituted with convolutional layers which can process spatial features while reducing parameters. Like standard LSTM, ConvLSTM is also driven by three selfparameterized controlling gates, i.e., an input gate it, a forget gate ft, and an output gate ot. Gates controlling and states transition of ConvLSTM can be described as follows:

where xt is the input at time step t; ht-1 and ht are the hidden states of the ConvLSTM at time step t-1 and t, respectively; gt is candidate content to be written into the cell state ct; all Ws, Us, and bs are learned weights and biases. The ∗ and ⊙ represent convolution operation and Hadamard product, respectively.

1) IOR-LSTM: IOR-LSTM is in charge of learning to capture IOR dynamics when generating scanpaths. It should be fed with information from currently and previously attended regions. Specifically, its input is combined by:

where Xt ∈ RW ′×H′×C represents image features, containing hints of currently fixated location due to the foveation process. W ′ , H′ and C denotes width, height and dimension of the extracted feature maps, respectively. hROI t-1 ∈ RW ′×H′×D is the hidden state of the ROI-LSTM which encodes attention history. D is the dimension of state maps. The fused representation of current and previous attention ˜Xt ∈ RW ′×H′×D is obtained by convolving the concatenation of Xt and hROI t-1 through a non-linear activation function f(·).

The IOR-LSTM is modified from the original ConvLSTM to better simulate operating mechanism of the VWM where recently attended objects are retained, and earlier attended regions are gradually decayed. As shown in Figure 3, the learned forget gate selectively removes earlier information from previous cell state ct-1 by multiplying it with an adaptive forgetting factor in (0, 1)W ′×H′×D . Currently observed information is first transformed into gt and then modulated by the input gate which determines what and how much information should be passed in. Different from the original ConvLSTM that updates memory through an addition operation, we simulate the iterative process of the VWM in the mental representation by updating cell state with the element-wise maximal values between decayed memory and information to be written as suggested by [Wang et al., 2011], which can be described mathematically as:

where ⌈·⌉ is used to truncate values to the range [0, 1], which essentially simulates the saturation of memory neurons. Besides, the original non-linear activation of candidate memory gt is replaced with ReLU(·) to only retain positive activation.

In the IOR-LSTM, each channel of ct indicates the spatial VWM usage of one type of features. Hence, values of the cell state ct can be considered as the inhibition strength of each feature map. The more memory is occupied, the intenser that spatial location should be inhibited. To guide the ROI-LSTM to explore regions where its corresponding VWM has no or shallow footprint, hidden state of the IOR-LSTM is updated as:

As expressed by Eqn. 4, unexplored regions will have higher values than attended areas. The feature-wise inhibition masks hIOR t produced by IOR-LSTM are multiplied with feature maps of the foveated image Xt element-wisely, which forms the inhibited feature maps XI that can be considered as the context describing where should not be focused during the next attention.

2) ROI-LSTM: In determining the next ROI, we use the soft attention model described in [Cornia et al., 2017] that convolves XI along with hidden state of the ROI-LSTM hROI t-1 to output a 3D tensor summarizing the hidden representations focusing on the input XI . The hidden representations are then projected into a single channel output using 1 × 1 convolution filters. This is followed by a softmax(·) function to normalize the input across spatial pixels. The whole process can be expressed as:

The attention map A is then applied to each channel of the inhibited image feature maps XI through the element-wise product to produce the inhibited attentive image features:

Finally, the ROI-LSTM, with original ConvLSTM structure driven by Eqn. 1, takes XIA as input and updates its hidden state hROI t whichwill be used to generate the ROI distribution map from which the point with the maximum value is selected as the next fixation.

2.3 Saliency Guidance Network

Visual saliency is one type of visual attention representations and saliency map demonstrates rich information about to what extent a specific region grabs observers' visual attention. Therefore, encoding saliency information into image features provides the IOR-ROI LSTM with better cues on where it should fixate. This is achieved by partially adopting readout network from the DeepGaze II [K¨ummerer et al., 2017] which takes the original image features as input and outputs the predicted saliency map. Thus, the ground truth saliency map of the input image can be used to guide the training of the high-level image feature extractor. The readout network, consisting of four 1 × 1 convolution layers followed by ReLU activation functions, transforms image features across various channels into the final saliency map. We train the saliency guidance network to encourage last convolutional features to output higher activation corresponding to more salient regions of the original image.

2.4 Loss Function

During the training phase, outputs produced by the IOR-ROI LSTM at time t are the IOR distribution IORt ∈ [0, 1]W×H and the next ROI distribution ROIt ∈ [0, 1]W×H . W and H represent width and height of the input image. These two outputs are functions of the hidden state hIOR t and hROI t , respectively. They are modeled as one 1×1 convolutional layer followed by Sigmoid non-linearity. Besides, another output

• (a) Raw fixations

(b) Refined fixations

Figure 4: Example scanpath from the OSIE dataset.

during training phase is the predicted saliency map St from the saliency guidance network. To jointly optimize the entire model, we define the loss function at time step t as:

where IOR∗ t , ROI ∗ t and S∗ t denote ground truth of the IOR, ROI and saliency map, respectively; L(·) is the binary cross entropy function, while α, β and γ are three scalar factors setting to balance the three loss functions.

3 Experiments

3.1 Datasets

We trained and evaluated our proposed model on the widely used public OSIE and MIT eye-tracking datasets consisting of natural images along with eye-tracking data from different participants. In the OSIE dataset, there are 700 images with 800 × 600 pixels. Eye-tracking data was acquired from 15 participants for each image. The MIT dataset is composed of 1003 images with resolution ranging from 405 to 1024. Eye-tracking data was also recorded from 15 subjects for each image.

We observed a common phenomenon in both OSIE and MIT datasets that most scanpaths have consecutive fixations located within very close distance, i.e., those fixations fall into the same ROI. Assuming that ROI corresponds to valid receptive field of the fovea, i.e., 5◦ of central field of vision [Wandell, 1995], 94.85% scanpaths in the OSIE dataset and 81.36% scanpaths in the MIT dataset have successive fixations belonging to the same ROI. To be consistent with the IOR theory, it is appropriate to combine successive fixations within the same ROI into one fixation, the merge process is iteratively executed until no consecutive fixations are within a given ROI. According to the experiment setup collecting the OSIE and MIT dataset, 1◦ of visual angle contains 24 pixels in the OSIE images [Xu et al., 2014] and 35 pixels in the MIT images [Judd et al., 2009], thus radius of ROI for the OSIE and MIT dataset are 60 pixels and 88 pixels, respectively. Figure 4(a) shows an example scanpath in the OSIE dataset. All (n+1)-th fixation located within a radius of 60 pixels around the n-th fixation should be merged into one fixation. Figure 4(b) shows the refined scanpath in which the fourth and fifth fixation are merging results of (4, 5) fixations and (6, 7, 8, 9) fixations shown in Figure 4(a).

3.2 Inhibition and ROI Map Generation

It is important to retrieve ground truth IOR to supervise our model to learn reasonable and interpretable IOR behavior.

However, datasets we used do not provide any IOR information. Therefore, we need to define what re-fixation is and how IOR attenuates only with the given fixations' location and order information. We consider re-fixation as an action that the n-th fixation falls into ROI centered at the (n - t)-th fixation. The (n - t)-th fixation is called re-fixated point of the n-th fixation. When re-fixation happened, inhibition effects exerted around the re-fixated point should fade completely. Hence, inhibition decay rate of the re-fixated point depends on the number of time steps that re-fixation happened since the re-fixated point has been visited for the first time. For other locations, inhibition effects decline to a given threshold until the end of the scanpath. Initial inhibition strength around the i-th attended location pi is computed as:

where x represents location in the inhibition map and σ controls the spread of the inhibition which is set to 2.5◦ of visual angle in our experiments. The ground truth inhibition map is a combination of previous decayed inhibition map and the latest inhibition map via the max operation:

Figure 5 illustrates the synthesized ground truth inhibition maps with respect to the scanpath shown in Figure 4(b). When predicting the first fixation, Figure 5(a), there is no inhibition effect presented. But when predicting the second fixation, Figure 5(b), regions around the first fixation is inhibited. It is worth noting that the second fixation is the re-fixated point of the fifth fixation, thus, inhibition effects around the second fixation disappeared after two time steps when the fifth fixation shown in Figure 5(e) is going to be predicted. Inhibition effects applied on other area decay stepby-step but will not die out.

Each ground truth fixation associates with one unique ground truth ROI map. ROI map is modeled as Gaussian blob, computed with the similar formula as Eqn. 8, centered at the ground truth fixation with the σ setting to 2.5◦ of visual angle which is the common valid fovea radius.

3.3 Implementation Details

We randomly splitted the OSIE and MIT datasets into 80% training data and 20% test data. We fully trained the IORROI LSTM and saliency guidance part while only fine-tuning dilated layers of the pre-trained VGG-19 network. Furthermore, training images and fixation coordinates were randomly flipped horizontally to generate more training data. The model was trained using the Adam optimizer along with step learning rate decay strategy which decays learning rate at a constant rate of 0.9. Initial learning rate for the IORROI LSTM and saliency guidance part is 1 × 10-4 while fine-tuning the dilated VGG-19 with an initial learning rate of 1 × 10-5 .

3.4 EvaluationMetric

There are a few scanpath evaluation metrics quantifying different aspects of predicted scanpath performance [Anderson et al., 2015]. The experiments employed three evaluation metrics. We first adopted the Dynamic Time Warp algorithm (DTW) used in [Le Meur and Liu, 2015], which can measure the similarity between two sequences with different lengths. The second evaluation metric, MultiMatch, introduced in [Dewhurst et al., 2012] consists of five separate measures that assess the similarity between two scanpaths with respect to shape, direction, length, position, and duration. We also evaluated scanpath similarity using ScanMatch [Cristino et al., 2010] which is able to take spatial, temporal and sequential similarities into account simultaneously. Similar to the compared benchmark [Le Meur and Liu, 2015], we only evaluate the performance on the spatial distribution aspects, where the fixation duration prediction is out of the scope of our paper.

3.5 Experimental Results

We compared our results against the state-of-the-art scanpath prediction models, i.e., Itti's model [Itti et al., 1998], SGC [Sun et al., 2012], SaltiNet [Assens Reina et al., 2017], Wang's model [Wang et al., 2011] and Le Meur's model [Le Meur and Liu, 2015]. To generate scanpath for a specific image using Itti's and Le Meur's model, we need to provide them with a saliency map corresponding to the image and, hence, quality of the generated scanpath can be significantly affected by the accuracy of the input saliency map. Thus, in comparison experiments, we fed Itti's and Le Meur's model with the test image and its ground truth saliency map. Moreover, to validate that the superiority of our model does come from incorporating learned IOR dynamics, a performance comparison between scanpath prediction model using just single ConvLSTM and the model with the IOR-ROI LSTM was also conducted. To better demonstrate the effectiveness of the designed IOR-LSTM, we compared the performance of the dual ConvLSTM model, by replacing the IOR-LSTM with original ConvLSTM, with that of the IORROI LSTM. Furthermore, since inter-subject scanpaths similarities evaluate different natural and real human scanpaths. Thus, inter-subject scanpath similarities can be used as a reference quantifying the quality of generated scanpaths. The less performance gap between the generated scanpaths and that of the real human scanpaths, the more realistic the generated scanpaths are.

When comparing scanpaths from various subjects on the same stimulus, it can be observed that they can be either similar or very different. Due to the explicit supervision of IOR behavior during the training phase, the IOR-ROI LSTM tends to be confused if we provide it with scanpaths of the same image from different observers with very different IOR behavior. Therefore, we trained the IOR-ROI LSTM for each subject separately to model a consistent IOR dynamics and ROI shift behavior. We randomly selected 10 subjects from the OSIE and MIT dataset for training, and evaluation scores of each subject-specific model were averaged as the final result.

For each model, 10 scanpaths were generated for a given image. The length of each scanpath was determined as mean scanpath length of the training dataset, i.e., 8 fixations for scanpaths in both the OSIE and MIT datasets. All three evaluation metrics were computed by comparing each predicted scanpath to all subjects' ground truth scanpath and final results were the average of the 10 scores of each evaluation metric. To measure inter-subject performance, we first evaluated the performance of each subject by treating scanpaths of other subjects as the ground truth. Then, the average value of all subjects was used as inter-subject performance.

Tables 1 and 2 report performance of different models on the OSIE and MIT datasets in terms of the mentioned evaluation metrics, respectively. From Table 1, we can see that the model combined the IOR-ROI LSTM with saliency guidance network outperforms all the other models, except the inter-subject reference model, in all three evaluation metrics. However, as presented in Table 2, MultiMatch vector measure of the IOR-ROI LSTM based model does not perform very well when compared with that of the single ConvLSTM model. This may be caused by the inconsistency of image content type and image scale in the MIT dataset, which is also confirmed by [Wang et al., 2017]. The overall performance of scanpath prediction using just a single ConvLSTM is higher than that of Le Meur's saccadic scanpath prediction model, while the IOR-ROI LSTM model can significantly boost scanpath generation performance, which clearly indicates that performance improvement does come from the advantage of explicitly modeling of IOR dynamics and attention shift. Furthermore, the incorporating of saliency information brings another performance gain, especially in the ScanMatch. The dual ConvLSTM can achieve similar performance in separate measures of the MultiMatch when compared with that of the IOR-ROI LSTM model. But it can be obviously observed that the IOR-ROI LSTM model outperforms the dual ConvLSTM model in the ScanMatch, which suggests that our proposed IOR-LSTM, combined with the ConvLSTM for ROI prediction, can better capture sequential characteristics of scanpaths in the test dataset. It is also worth noting that our proposed model can achieve comparable results in the MultiMatch vector, direction, and length similarity metrics when compared with that of the inter-subject, which means that the proposed model is able to well capture the overall shape, saccade direction and saccade amplitude characteristics of scanpaths in the test dataset.

In addition to qualitative evaluation, Figure 6 illustrates the visual comparison of scanpaths generated by our proposed model with the ground truth human eye-tracking scanpaths. As is shown in the figure, fixations of the generated scanpath almost cover the same locations as fixations of the ground truth scanpath do. We can also observe that the generated scanpaths have high shape similarity with the ground truth scanpaths, which manifests the effectiveness of our proposed scanpath prediction model and also gives a well visual explain on why the proposed model can obtain comparable performance in MultiMatch vector, direction and length measures shown in Tables 1 and 2.

We also visualized the IOR dynamics and attention shift behavior for each fixation prediction step in Figure 7. As shown in the first row of Figure 7(a), when generating the second fixation, surrounding region of the first fixation is strongly inhibited and its corresponding region in the ROI map, shown in the second row of Figure 7(a), has relatively low value, then another salient object is going to be attended. Another thing should be noted is that as the scanpath proceeds, IOR effects on previously fixated location decay adaptively and ROI value of that region can gradually recover to a high level, which finally results in the re-fixation. This process is illustrated in Figure 7(c). When predicting the fourth fixation, inhibition in the first fixation declines below a threshold and ROI of that area becomes the highest. Figure 7(d)(f)(g) also demonstrate the occurrence of re-fixation.

4 Conclusion

In this paper, we present a model to predict scanpath. In the proposed model, a novel IOR-ROI LSTM module can simultaneously model IOR dynamics and attention shift behavior. Compared with existing computational scanpath estimation approaches which model IOR in a linear manner, our IOR-LSTM, simulating the VWM, learns IOR dynamics based on visual features extracted by CNNs. Experimental results show that our proposed scanpath prediction model outperforms state-of-the-art computational saccadic scanpath generation approach. Explicitly modeling IOR in scanpath estimation can effectively boost fidelity of the generated scanpath.