Recurrent Relational Memory Network for Unsupervised Image Captioning

1 Introduction

Traditional image captioning [Yao et al., 2019; Huang et al., 2019] requires full supervision of image-caption pairs annotated by humans. However, such full supervision is ridiculously expensive to acquire in cross-modal datasets. Recently, substantial researches tend to flexible constrained caption tasks, such as unpaired captioning [Gu et al., 2019; Guo et al., 2019] and unsupervised captioning [Feng et al., 2019] with weak or no supervised cues. It is challenging to leverage the independent image set and sentence corpus to train a reliable image captioning model; worse still, image captions usually cover specified or insufficient topics, e.g., the well-known benchmark MSCOCO images [Lin et al., 2014] just cover 80 object categories, raising up the challenges to generate rich semantical and grammatical sentences.

There are merely two unsupervised methods test on the disjointed image and text corpus data. [Feng et al., 2019]

˚Corresponding authors.

proposed an architecture comprising of an image encoder, a sentence generator, a discriminator with adversarial loss and concept reward. [Laina et al., 2019] learned a joint semantic embedding space, where either images or sentences were transformed. A discriminator was then designed to judge where the embedding feature came from, image or sentence domain. Both of them resolved the task with adversarial training, while obeying the usage of GAN (Generative Adversarial Networks) in unsupervised mode [Lample et al., 2018; Donahue and Simonyan, 2019; Yang et al., 2018]. As it is widely known, current GAN methods based on ordinary recurrent models (e.g., LSTM) always employ RL heuristics and are quite sensitive to parameter initializations and hyperparameter settings [Nie et al., 2019].

Orthogonal to above GAN-based models, in this paper, we propose a novel memory-based solution, named Recurrent Relational Memory Network (R2M). The novelty of R2M lies in its exploitation on visual concepts and describing image via memory, serving as a concepts-to-sentence memory translator to learn the textual knowledge from discrete common concepts in diverse sentences, meanwhile being capable of making sentences with correctly semantic and grammar syntax rules.

As illustrated in Fig.1, R2M explores the latent relevant semantic learning with the memory network, so as to enjoy the flexible and augmented memory capacity for both vision and natural language processing tasks [Pei et al., 2019; Fan et al., 2019]. Our intuition is that memory is proficient at storing and retrieving relational contexts to correlate input information [Huang and Wang, 2019], while inhibits vanishing gradients [Santoro et al., 2018; Fu et al., 2019]. As illustrated in Fig. 2, R2M consists of two-stage memories, fusion memory (FM) and recurrent memory (RM). The relational reasoning based on FM and RM in our captioning process, not only considers the dependencies between words and common concepts, but also distills the useful context into the memory, retaining for long periods. Technically, R2M performs the recurrent relational reasoning through multi-head self-attention and a relational gate (detailed in Section 2.2).

So far, FM and RM memories resolve the relational reasoning for text generation. As inspired, as shown in Figs.1 and 2, we develop a joint exploitation of supervised learning (SPL) and unsupervised learning (UPL) on the disjoint datasets. In particular, the SPL strategy is to learn the memories and make sentences from several salient words that separately appeared in text corpus, while the UPL training is to fine-tune the memories and make much more appropriate sentences about the visual context in the image. The supervised training on text corpus incorporates inductive semantic bias into the language model training. Turning to visual concepts in images without supervision cues, we explore a semantic matching (hinge-based triplet ranking) loss LIM and reconstruction loss. These two losses encourage the cross-modal similarity score of image, along with the generated sentence to be larger than that of the other sentences. For LIM , we distinguish the negatives pI 1 , CIq and pI, C1 q from a positive image pair pI, CIq, which is discussed in Section 2.4.

The major contributions are summarized as follows:

• Orthogonal to GAN-based architectures for unsupervised image captioning, we propose a novel light Recurrent Relational Memory Network (R2M), which merely
• utilizes the attention-based memory (detailed in Section 2.2) to perform the relational semantics reasoning and reconstruction.
• A joint exploitation of Supervised learning on text corpus and Unsupervised learning on images is proposed. We optimize the cross-modal semantic alignment and reconstruction via an unsupervised manner to achieve a novel concepts-to-sentence translation.
• The proposed R2M achieves better performances than state-of-the-arts on all the current unsupervised datasets: MSCOCO paired Shutterstock captions, Flickr30k paired MSCOCO captions and MSCOCO paired GCC (Google's Conceptual Captions).

2 ProposedMethod

In this section, we formally discuss our proposed R2M. The overall architecture of R2M is depicted in Fig.2, which consists of three modules: encoder, decoder and reconstructor.

2.1 R2M. Encoder

We first discuss the encoder. A visual dictionary D is learned ahead by using Faster R-CNN [Huang et al., 2017] trained on a public dataset OpenImages-v4 [Krasin et al., 2017; Kuznetsova et al., 2018] to cover the majority of common visual concepts in daily conversations, which is used to filter out visual concepts V " tvi|ki"1 u of image I or sentence S. After that, visual concepts V are randomly and sequentially incorporated into LSTMwith their word embeddings, leading to the encoded vector v " vI or vS from I or S.

2.2 R2M. Decoder

Details of decoder are illustrated in Fig.3. The effect of R2M.Decoder is to generate grammatical and semantical sentences from a few discrete words, e.g., translating "man" and "motorcycle" to "a man riding on the back of a motorcycle". The set of visual concepts has no available grammar and syntax contexts. Based on that, we train the model to think, infer and talk about as human beings. To address this issue, we propose a memory-based decoder, which not only considers the correlation between visual concepts and current generated word, but also captures the temporal dependencies and distills the underlying memory information.

Relation Learning I: Fusion Memory (FM)

As shown in Fig.3, the fusion memory (FM) in the decoder phase is used to learn the relationship between visual concepts and generated words, while recurrent memory (RM) in both decoder and reconstructor recurrently updates the memory to deliver useful semantics. At time step t, FM learns the implicit relationship between the encoded concept vector v P Rd and previous generated word wt´1 P Rd . We adopt a row-wise concatenation to acquire a joint feature matrix xt " rv;wt´1s P R2ˆd , upon which multi-head selfattention [Vaswani et al., 2017] is performed. The intuition is to explore the correlation between v and wt´1. We consider the influences: v Ñ v, v Ñ wt´1, wt´1 Ñ wt´1 and wt´1 Ñ v. They are performed by the dot-product of query and key transformers of xt as follows:

where Uq,Uk P Rdˆdk are parameters of linear transformations of xt (query and key); λ1 is a scaling factor to balance the fusion attention distribution.

The cross interaction between v and wt´1 is calculated based on both attended weights and values as follows:

where Uv P Rdˆdv is another learnable parameter of linear transformations of xt (value).

To ensure diverse and comprehensive attention guidance, we fuse v and wt´1 from H perspectives. There are H heads of independent attention executions. The outputs are concatenated into a new matrix x1 t as follows:

where || denotes column-wise concatenation. Finally, we use a fully-connection (linear) layer to convert the matrix x1 t into a fusion-aware feature ft below:

Relation Learning II: Recurrent Memory (RM)

Observing ft at the t-th time step, RM recurrently learns a decoded memory variable M d t as shown in Fig.4. To distill the information worthy to retain in memory, we apply a relational gate for the recurrent memory updating among sequential learning. First, the multi-head self-attention is recycled to model latent transformers of previous memory state M d t´1 and fusion-aware feature ft , where M d 0 is initialized with zero-padding. Note that we merely focus on the memory variation itself. The query is related toM d t´1 , key and value refer to rM d t´1 ; fts, implying that the joint effect of rM d t´1 ; fts is learned under the guidance of M d t´1 . In this part, the detailed dimensions of parameters are shown in Fig.4.

where W h q ,W h k P RdˆdK andW h v P RdˆdV are learnable parameters, and λ2 is the scaling factor to balance the attention distribution in RM.

Module ψ M 1dt is then fed into two residual connection layers and one row-wise multi-layer perception (MLP) with layer normalization. Thus, we achieve a memory gain ˜M d t .

Relational Gate To model the temporal dependencies along the adjacent memories, we update the memory state in a gated recurrent manner. Specifically, we apply input gate gi,t and forget gate gf,t to balance the memory updating from the current memory gain ˜M d t and original memoryM d t´1 , respectively. Both gi,t and gf,t are affected by ft and M d t´1 .

where d and σ denote dot product and sigmoid functions. Based on the updated memory M d t , RMoutputs the word wt:

where Wd is a learnable matrix that maps M d t to a vector with the dictionary size.

2.3 R2M. Reconstructor

So far, the decoder yields a pipeline to translate discrete visual concepts into a formal sentence. Here, we attempt to ensure that R2M can talk about correct contents. As inspired, we reversely reconstruct the concept semantics, i.e., rebuilding the crucial concept semantics from the generated sentence. We adopt the memory unit RM to compose the R2M.Reconstructor. Note that learnable parameters of RM in R2M.Decoder and Reconstructor are completely different.

If we define the RM operation in R2M.Decoder as a function M d t " RMpM d t´1 , ftq involving Eqs. 5"7, the R2M.Reconstructor operation is formulated as follows:

where M r t indicates a reconstructed memory at time t, M r 0 is initialized with zero-padding, and len is the length of the generated caption C. The last output of R2M.Reconstructor is treated as the reconstructed vector of concepts, denoted as ˜ vI or ˜ vS corresponding to image I or sentence S.

2.4 Training

Supervision Learning on Text Corpus

We train the concepts-to-sentence decoder R2M.Decoder by maximizing log-likelihood of the generated sentences with original corpus sentences:

For R2M.Reconstructor, there is the reconstructed vector ˜ vS corresponding to sentence S. We align it in an unsupervised mode. The full objective on text corpus is:

where LSrec " ||vS ´ ˜ vS ||2L2 , β is a hyper-parameter, and ||.||2L2 denotes the L2-norm loss.

Unsupervised Visual Alignment on Images

The remaining question is how to achieve a better generalization ability with no supervision cues for image captioning? To answer this question, we adopt a hinge-based triplet ranking loss LIM , which encourages the semantic relevance of pI, CIq to be much larger than other negative examples. We choose the hardest negatives I 1 and C1 for a positive pair pI, CIq, and perform LIM as follows:

where rxs` " maxpx, 0q, Sp¨q is the similarity function calculated by inner product, and m serves as a margin parameter. SpI, CIq " Spˆ vI , ˜ vI q, where ˆ vI is the visual feature of image I extracted by Inception-V4 [Szegedy et al., 2017] and ˜ vI is the reconstructed vector by the RM unit. For computational efficiency, we search the negatives I 1 and C1 within each mini-batch instead of the entire training set.

Besides, the image reconstruction loss LIrec is utilized to train the model. The full objective on images is:

3 Experiments

3.1 Dataset andMetrics

We test all the existing unsupervised image captioning datasets, including (1) MSCOCO images [Lin et al., 2014] paired with Shutterstock captions [Feng et al., 2019]; and (2) Flickr30k images [Young et al., 2014] paired with MSCOCO captions and (3) MSCOCO images paired with Google's Conceptual Captions (GCC) [Sharma et al., 2018; Laina et al., 2019]. In the test splits of datasets, each image has five ground-truth captions.

3.2 Implementation Details

We split each image set and filter captions as [Feng et al., 2019; Laina et al., 2019]. The visual dictionary D in Fig.2 is collected by a pre-trained Faster R-CNN [Huang et al., 2017] OpenImages-v4 [Krasin et al., 2017; Kuznetsova et al., 2018]. We merge the visual concepts in D and words in training captions into a large vocabulary, to cover the majority of the to-be-generated words. The vocabulary sizes of the three datasets are 18,679/11,335/10,652, respectively, including tokens ă#startą, ă#endą, and ăUNKą. For experimental setting, we filter out visual concepts form images with the detected score ě 0.3. Both the sizes of LSTM and RM memory are set to N " 1 and d " 512. The parameters of multi-head self-attention are H " 2, dk " dK " 256, and dv " dV " 256. The margin in Eq. 12 is m " 0.2. Adam optimizer is adopted with batch size of 256. For three datasets, hyper-parameters (β, γ) are set to (1, 1), (1, 1), (0.2, 0.2). We train the model with a loss LXE under learning rate 10´4 , while fine-tune it with the joint loss LS. After that, LIM is used to train with a learning rate 10´5 . Finally, we jointly train the model with LI . In the test process, we use the beam search tactic [Anderson et al., 2017] with width of 3.

3.3 Experimental Results and Analysis

Comparison with the State-of-the-arts

R2M exhibits large improvements across all the metrics. Both UC-GAN [Feng et al., 2019] and SME-GAN [Laina et al., 2019] rely on complicated GAN training strategies, whereas ours R2M is a memory solution. As shown in Table 1, R2M upgrades BLEU-4 (B-4) by 14.3%, 48.1% and 27.7% on three datasets, where BLEU-4 involves 4gram phrases. It implies the stronger capacity of R2M to learn long-range dependencies than others. R2M also raises CIDEr/SPICE, from 28.6/8.1 to (29.0/9.1), 9.9/7.5 (18.1/8.3) and 22.7/7.4 (29.3/9.6). The promising improvements demonstrate the consistency of superior performances. With the released code of UC-GAN [Feng et al., 2019] on the MSCOCOØShutterstock dataset, here is an efficiency comparison: R2M vs. UC-GAN « 35 min vs. 34 hours. R2M also enjoys higher computational efficiency.

Ablation Study of R2M

To verify each component in R2M, we propose the ablation study. (1) Effect of FM. Compared to the entire R2M, the performance of "D w/o FM" drops significantly, e.g., with 18.3%, 17.7% and 18.4% reduction of CIDEr (C) on three datasets in Table 2. FM effectively implements the implicit correlation between visual concept vector and word embedding. (2) Effect of memory in RM. For Table 2, either "D, R or D&R w/o Memory in RM" suffers from worse performance, e.g., on dataset MSCOCOØGCC, dropping the CIDEr from 29.3 to 27.1, 28.1 and 27.8. RM excels at storing and retrieving information across time than classical LSTM, to effectively handle sequential learning. (3) Effect of Loss. In each block diagram of Table 3, the first line records the result of model trained with only LXE on text corpus. Note that this baseline is competitive and outperforms the existing methods. The SPICE (S) is increased by around 11.1%, 9.3% and 25.7% compared on three datasets. By gradually incorporating LSrec , LIM and LIrec , the model performs much better. The CIDEr gradually raises from 25.4 to 27.0, 28.9 and 29.0 on MSCOCOØShutterstock. Especially after the assistance of semantic matching loss LM, the CIDEr is significantly improved, nearly 7.0%, 6.6% and 3.2% on all datasets.

Qualitative Results

Visualization of Attention Weights in FM & RM. Fig.5 illustrates an example of memory learning in R2M.Decoder, which is interpretable. FM displays the average weight of multi-head attention, while RM offers H " 2 heads attention. With the beginning token ă#startątt"0u and the encoded concept vector v, FM pays more attention to the richer semantics v. And at time t " 2, FM focuses much more on the previous word portraittt"1u as portrait is the first generated concept and deserves more attention. Then, we discuss the interpretation of RM. Taking previous word wearingtt"9u as an example, it affects the generation of sunglassestt"10u more influentially than the memory M d t . However, at t " 11, under the previous cue sunglassestt"10u, the model infers a relational conjunction andtt"11u by mainly recalling M d t . The same situation holds at the last time umbrellatt"14u, there is no relevance cues to be found from M d t , the model decides to terminate the entire generation process.

7LPH

��� ��� ��� ��� ��� ���

model pays to each input variable.

,PDJHBLG� ����

&RQFHSWV� >�FDUURW�� �SKRQH�@� >����� ����@

*7� &DUURWV DUH EHLQJ FXW LQWR SLHFHV ZLWK D ODUJH NQLIH�

63/ RQ WH[WV� FDUURW DQG PRELOH SKRQH RQ ZKLWH EDFNJURXQG�

83/ RQ LPDJHV� IUHVK FDUURW RQ D ZRRGHQ EDFNJURXQG�

� D � $ H[DPSOH LQ 06&2&2 ↔6KXWWHUVWRFN GDWDVHW

,PDJHBLG� ������

&RQFHSWV� >�VXUIERDUG�� �PDQ�� �FORWKLQJ�@� >����� ����� ����@

*7� $ SHUVRQ DW WKH EHDFK ZLWK D VXUIERDUG�

63/ RQ WH[WV� PDQ ZLWK D VXUIERDUG RQ WKH EHDFK�

83/ RQ LPDJHV� D \RXQJPDQ ZLWK D VXUIERDUG RQ WKH EHDFK�

� E � $Q H[DPSOH LQ 06&2&2 ↔*&& GDWDVHW

,PDJHBLG� ����������

&RQFHSWV� >�SHUVRQ�� �KRUVH�@� >����� ����@

*7� &RZER\ DW D URGHR ULGLQJ D KRUVH�

63/ RQ WH[WV� D SHUVRQ ULGLQJ D KRUVH RQ D EHDFK�

83/ RQ LPDJHV� D SHUVRQ ULGLQJ D KRUVH LQ D ILHOG�

� F � $Q H[DPSOH LQ )OLFNU��N ↔06&2&2 GDWDVHW

Figure 6: Qualitative examples of R2M. Visual concepts are detected by Faster R-CNN. Words in tred, brown, blue, greenu fonts mark the incorrectly recognized, irrelevant, and new detected concepts, and the adjective, respectively.

Visualization of Generated Captions. We detect visual concepts and their scores by Faster R-CNN. As shown in Fig.6 (a), phone is an incorrectly detected object with a high score 0.75. While performing training on text corpus with LXE and LSrec , R2M translates discrete concepts to a sentence, still containing phone. With further unsupervised training on images over LIM and LIrec , R2M automatically eliminates the wrong concept. By contrast, exemplified in Fig.6 (b), clothing is a correctly identified concept, but irrelevant to salient visual regions of the image. R2M eliminates the redundant visual concepts yet. Moreover, there are new learned concepts beach and an adjective young from all the joint SPL and UPL semantic learning. To strengthen the intuition that R2M can extrapolate beyond the concepts in the images, we offer another example in Fig.6 (c). Both the new words beach and field are undetected visual concepts. Following the textual cues learning from text corpus, R2M acquires the knowledge to infer a new context-independent concept beach; however, it is irrelevant. After unsupervised visual alignment learning, the caption finally outputs a new word f ield instead of beach. R2M is effective to infer promising descriptions about images without annotated captions.

&RQFHSWV� >�VKLUW�� �WLH�� �VXLW�� �PDQ�� �FORWKLQJ�� �IDFH�@

*7� $ PDQ ZLWK D WLH DQG D VXLW�

5 � 0� KDQGVRPH PDQ LQ VKLUW DQG WLH�

5 � 0 Z 94$�Y�� ZKDW FRORU LV WKH VKLUW RI WKH PDQ

ZLWK WKH WLH DQG VXLW KDYH RQ"

5 � 0 Z 6HQWL&DS� D PDQ LV ZHDULQJ D VXLW � WLH DQG QLFH VKLUW�

&RQFHSWV� >�WHQW�� �FDW�� �XPEUHOOD�@

*7� $ FDW WKDW LV VLWWLQJ XQGHUQHDWK DQ XPEUHOOD�

5 � 0� WKH FDW LV VLWWLQJ XQGHU WKH XPEUHOOD�

5 � 0 Z 94$�Y� � ZK\ LV WKH FDW XQGHU WKH XPEUHOOD QHDU

WKH FDW"

5 � 0 Z 6HQWL&DS� D IULHQGO\ FDW LQ D EHDXWLIXO XPEUHOOD�

&RQFHSWV� >�NH\ERDUG�� �FDW�@

*7� D FDW OD\LQJ GRZQ E\ D NH\ERDUG

5 � 0� FDW RQ WKH NH\ERDUG�

5 � 0 Z 94$�Y�� LV WKH FDW W\SLQJ RQ WKH NH\ERDUG"

5 � 0 Z 6HQWL&DS� WKH EHDXWLIXO FDW LV VOHHSLQJ RQ WKH NH\ERDUG�

Figure 7: Extensive captions of MSCOCO images in different language styles. Words in tbrown, pinku fonts highlight the incorrect detected concepts and the sentiment adjectives, respectively.

We also extend the experiments with new corpora with different language styles, such as VQA-v2 [Antol et al., 2015] and SentiCap [Mathews et al., 2016], involving the questions about the visual content and sentiment captions. For our experiments, 1,105,904 questions provided by VQA-v2 and 4,892 positive captions of SentiCap are respectively trained as text corpus. As shown in Fig.7, R2M also excels at questioning and describing images with positive emotion.

4 Conclusion

This paper proposes a novel recurrent relational memory network (R2M) for unsupervised image captioning with low cost of supervision. R2M is a lightweight network, characterizing self-attention and a relational gate to design the fusion and recurrent memory for long-term semantic generation. Experimental results show that the R2M surpasses the state-of-the-arts on three benchmark datasets.

Acknowledgments

This work is supported by the National Key Research and Development Program of China under grant 2018YFB0804205, and the National Natural Science Foundation of China (NSFC) under grants 61806035, U1936217, 61725203, 61732008, and 61876058.