GM-MLIC: Graph Matching based Multi-Label Image Classification

1 Introduction

Multi-label image classification (MLIC) is an essential computer vision task, aiming to assign multiple labels to one image based on its content. Compared with single-label image classification, MLIC is more general and practical since an arbitrary image is likely to contain multiple objects in the physical world. Thus, it widely exists in many applications such as image retrieval [Wei et al., 2019] and medical diagnosis recognition [Ge et al., 2018]. However, it is also more challenging because of the rich semantic information and complex dependency of an image and its labels.

∗The authors contributed equally to this work.

†Corresponding Author

The key to accomplish the task of MLIC is how to effectively explore the valuable semantic information from the image context, and further obtain the correct assignments between images and their labels. A simple and straightforward way is to treat each image as a bag of instances/proposals coping with the instances in isolation, and convert the multi-label problem into a set of binary classification problems. However, its performance is essentially limited due to ignoring the complex topology structure among labels. This stimulates research for approaches to capture and mine the label correlations in various ways. For example, Recurrent Neural Networks (RNNs) [Wang et al., 2016] [Wang et al., 2017] and Graph Convolution Network (GCN) [Chen et al., 2019b] [Wang et al., 2020b] are widely used in many MLIC frameworks owing to their competitive performance on explicitly modeling label dependencies. However, most of these methods ignore the associations between semantic labels and image local features, and the spatial contexts of images are not sufficiently exploited. Some other works [Zhu et al., 2017] [Chen et al., 2019a] introduce attention mechanisms to adaptively search semantic-aware instance regions and aggregate features from these regions to identify multiple labels. However, due to the lack of fine-grained supervision information, these methods could merely locate instance regions roughly, which neither consider the interactions among instances nor explicitly describe the instance-label assignment relationship.

To address the above mentioned issues, in this paper, we fully explore the correspondence (matching) between each instance and label, and reformulate the task of MLIC as an instance-label matching selection problem. Accordingly, we propose a novel Graph Matching based Multi-Label Image Classification (GM-MLIC) deep learning model, which simultaneously incorporates instance spatial relationship, label semantic correlation and co-occurrence possibility of varying instance-label assignments into a unified framework. Specifically, inspired by Graph Matching scheme (GM) suitable for structured data, we first design an instance spatial graph by representing each instance feature as the node attribute and the relative location relationship of adjacent instances as the edge attribute. Meanwhile, a label semantic graph is employed to capture the overall semantic correlations, which takes the word embedding of each label as node attribute, and concatenates the attributes of two nodes (labels) associated with the same edge to form the edge attribute. Then each instance is connected to all labels to form the final assignment graph, as shown in Figure 1, which aims to explicitly model the instance-label matching possibility. Furthermore, the Graph Network Block (GNB) is introduced to our framework to perform computation on the constructed assignment graph, which forms structured representations for each instance and label by a graph propagation mechanism. Finally, we design a weighted cross-entropy loss to optimize our network output, which indicates the prediction score of each instance-label correspondence. Extensive experiments demonstrate that our proposed method can achieve superior performance against state-of-the-art methods.

2 RelatedWork

The task of MLIC has attracted an increasing interest recently. A straightforward way to address this problem is to train independent binary classifiers for each label. However, such method does not consider the relationship among labels, and the number of predicted labels will grow exponentially as the number of categories increase. To overcome the challenge of such an enormous output space, some works convert multilabel problem into a set of multi-class problems over region proposals. For example, [Wei et al., 2015] extracted an arbitrary number of object proposals, then aggregated the label confidences of these proposals with max-pooling to obtain the final multi-label predictions. [Yang et al., 2016] treated each image as a bag of instances/proposals, and solved the MLIC task in a multi-instance learning manner. However, the above methods ignore the label correlation in multi-label images when converting MLIC to the multi-class task.

Recently, researchers focus on exploiting the label correlation to facilitate the learning process. [Gong et al., 2013] leveraged a ranking-based learning strategy to train deep convolutional neural networks for MLIC and found that the weighted approximated-ranking loss can implicitly model label correlation and work best. [Wang et al., 2016] utilized recurrent neural networks (RNNs) to transform labels into embedded label vectors, so that the correlation between labels can be employed. [Chen et al., 2019b] used Graph Convolutional Network to map a group of label semantic embeddings borrowed from natural language processing into interdependent classifiers. [Wang et al., 2020b] proposed to model label correlation by superimposing label graph built from statistical co-occurrence information into the graph constructed from knowledge priors of labels. However, none of the aforementioned methods consider the associations between semantic labels and image local features, and the spatial contexts of images have not been sufficiently exploited.

To solve the above issues, recent progress on MLIC attempt to model label correlation with region-based multilabel approaches. For example, [Wang et al., 2017] introduced a spatial transformer to locate semantic-aware instance regions and then captured the spatial dependencies of these regions by Long Short-Term Memory. [Chen et al., 2019a] incorporated category semantics to better learn instance features and explored their interactions under the guidance of statistical label dependencies. Although the above methods have achieved competitive performance, they neither consider the spatial location relationships among instances nor explicitly describe the instance-label assignment relationships, which may make these methods lose the ability to effectively represent the categories visual features. Different from all these methods, we utilize the GM scheme and propose a novel multi-label image classification learning framework called GM-MLIC, where the instance spatial relationship, label semantic correlation and instance-label assignment possibility are simultaneously incorporated into the framework to improve the classification performance. The details of the framework are introduced in the following section.

3 The ProposedMethod

Given a multi-label image dataset D = {(Xi, Yi)}Ni=1 , we denote Xi = {x1i , x2i , ..., xMi } as the i-th image that consists of M instances, Yi = [y1 i , y2 i , ..., yC i ]T as the ground-truth label vector for Xi, where each instance xji is a d-dimensional feature vector and C is the number of all possible label in the dataset. yc i = 1 indicates that image Xi is annotated with label c, and yc i = 0 otherwise. GM-MLIC aims to learn a multi-label classification model from the instance-level feature vector together with image-level ground-truth label vector, and further assign the predictive labels for a test image.

3.1 Overview

The overview architecture of the proposed GM-MLIC is illustrated in Figure 2, which consists of two components: the ASsignment Graph Construction (ASGC) and the InstanceLabel Matching Selection (ILMS). The constructed assignment graph takes instance spatial graph and label semantic graph as input, and considers each instance-label connection as candidate matching edge. The matching selection module is another core component of our learning framework, which introduces Graph Network Block (GNB) to convolve the information of neighborhoods of each instance and label to form a structured representation through several convolution operators. Finally, our model derives a prediction score for each instance-label correspondence and optimizes such correspondence with a weighted cross-entropy loss.

3.2 The Assignment Graph Construction

As depicted in Figure 1, we first construct an instance spatial graph to explore the relationship of the spatially adjacent instances. Specifically, we feed an image to a pre-trained Faster R-CNN network [Ren et al., 2016] to generate a set of semantic-aware instances, where each instance contains a bounding box B(x, y, w, h). Each of these instances is taken as a node in the instance spatial graph Go in which the edges between each pair of nodes are produced through k-Nearest Neighbor criteria. Note that our Go is a directed graph, where the attributes of nodes are denoted by the feature f of corresponding instances, and the attributes of edges are represented by the concatenations of bounding box coordinates of its source instance and receive instance

where fi and ˆBi(xi, yi, xi +wi, yi +hi) denote the attributes and location coordinates of i-th instance respectively and [·, ·] the concatenations of its input.

Similar to [Chen et al., 2019b] [You et al., 2020], we construct a label semantic graph Gl to capture the topological structure in the label space, where each node of the graph is represented as word embeddings of the label. Different from above these methods, our Gl does not require pre-defined label co-occurrence statistics. It instead combines the word embeddings of the connected two nodes (labels) to form the initial edges attributes of Gl as

where wi denotes the word embedding of i-th label.

To explicitly establish the instance-label matching relationship, we connect each instance in Go to all labels in Gl to form the instance-label assignment graph GA. In GA, the attributes of the matching edge that connecting instance and label are represented asemij = [vo i , vlj ]. (3)

In this way, we successfully convert the problem of building the complex correspondence between an image and its labels to the issue of selecting reliable edges from a constructed assignment graph. Accordingly, the goal of the MLIC problem is transformed into how to solve the matching selection problem and obtain the optimal instance-label assignment.

3.3 Modeling Instance-Label Correspondence

As illustrated in Figure 2, our matching selection module is designed on the top of Graph Network Block presented in [Wang et al., 2020a], which defines a class of functions for relation reasoning over graph-structured representations. Specifically, the matching selection module consists of three main components: Encoder, Graph Convolution and Decoder.

Encoder. The encoder takes the constructed assignment graph GA as input, and transforms its attributes into a latent feature space by two parametric update functions ϕvenc and ϕeenc . In our framework, ϕvenc and ϕeenc are designed as multi-layer perceptrons (MLPs), each of which takes respectively a node attribute vector and an edge attribute vector as input and transforms them into latent spaces.

Formally speaking, we denote ϕvenc (V A) and ϕeenc (E A) as the updated node attributes and edge attributes by applying ϕvenc and ϕeenc to each node and each edge respectively. Then the encoder module can be briefly described as

The updated graph GA is then passed to the subsequent convolution modules as input.

Graph Convolution Module. This module consists of a node convolution layer and an edge convolution layer. The node convolution layer collects the attributes of all the nodes and edges adjacent to each node to compute per-node updates. It is followed by the edge convolution layer that assembles the attributes of the two nodes associated with each edge to generate a new attribute of this edge.

Specifically, for the i-th node in Go , the aggregation function gathers the information from its adjacent nodes and associated edges, and the update function outputs the updated attributes according to the gathered information. Given that for each instance node vo i , other instance nodes and label nodes are connected with instance edges and matching edges respectively, we design two types of aggregation functions as

where No and Nl are two sets of instance nodes and label nodes adjacent with vo i . ˆ ρon and ˜ ρon gather the information from object nodes and label nodes respectively. Then an update function is employed to update the attributes for vo i

where φon takes the concatenation of the current attributes of vo i and gathered information ˆ vo i and ˜ vo i , and outputs the updated attributes for vo i .

Similar to instance nodes, the label nodes are connected with two types of nodes and associated with two types of edges. Therefore, we also design two aggregation functions and an update function for label node vl i . Specifically, the aggregation functions are formulated as

and the update function is represented as

For an instance edge eoij , both of its source node and receive node are in Go . Therefore, we design an aggregation function and the update function as follows

where ρoe aggregates the information from source instance node vo i and receive node vo j , φoe updates the attributes of eoij according to the gathered information.

Similar to the instance edge convolution operator, the label edge convolution layer consists of an aggregation function and an update function, which are designed as

Different from instance edges and label edges, the matching edges connect instance nodes and label nodes. Therefore, the aggregation function gathers information from different type of nodes

and the update function also takes the combination of aggregated features and its current features as input and output the updated attributes

All the above aggregation functions and update functions are designed as MLPs, but their structure and parameters are different from each other.

Decoder. The decoder module reads out the final output from the updated graph state. Since only the attributes of instance-label matching edges are required for final evaluation, the decoder module contains only one update function ϕedec that transforms the edges attributes into the desired space

where S ∈ [0, 1]M×C denotes the prediction score that each instance is matched with the corresponding label. Similarly, ϕeenc is parameterized by an MLP.

3.4 OptimizingMulti-Label Prediction

In order to interpret each ground truth label of the input image, there should be at least one instance that best matches it. With consideration of the possibly noisy instances, a cross-instances max-pooling is carried out to fuse the output of our framework into an integrative prediction. Suppose sj(j = 1, 2, ...,M) is the prediction score vector of the jth instance from the decoder and scj (c = 1, 2, ...,C) is the c-th category matching score of sj . The cross-instances maxpooling can be formulated as

where pci can be considered as the prediction score for the c-th category of the given image i. Finally, the training process of our network is guided by a weighted cross-entropy loss with the ground-truth labels Yi as supervision,

where wc is used to alleviate the class imbalance, β is a hyperparameter and rc is the ratio of label c in the training set.

4 Experiments

4.1 EvaluationMetrics

We adopt six widely used multi-label metrics to evaluate each comparing method, including the average per-class precision (CP), recall (CR), F1 (CF1) and the average overall precision (OP), recall (OR), F1 (OF1), whose detailed definitions can be found in [Chen et al., 2019a]. In this paper, we present the above metrics under the setting that a label is predicted as positive if its estimated probability is greater than 0.5. To fairly compare with the state-of-the-art methods, we also report the results of top-3 labels. Besides, we compute and report the average precision (AP) and mean average precision (mAP).

4.2 Implementation Details

In ASGC module, we apply Faster R-CNN (resnet50-fpn) [Ren et al., 2016] to generate a set of instances for per image, where each of these instances, in addition to the visual feature f and bounding box B, also contains a preliminary class label c with a confidence score s. To save computational cost, we only select the instances with the top-m confidence score for each image. For label representations, we adopt 300-dim GloVe [Pennington et al., 2014] trained on the Wikipedia dataset. In ILMS module, the Graph Convolution Layer consists of k convolution modules, where they are stacked to aggregate the information of kth -order neighborhoods. In our experiments, k is 2 and the output dimension of the corresponding convolution module is 512 and 256, respectively. During training, the input images are randomly cropped and resized into 448× 448 with random horizontal flips for data augmentation. All modules are implemented in PyTorch and the optimizer is SGD with momentum 0.9. Weight decay is 10-4 . The initial learning rate is 0.01, which decays by a factor of 10 for every 30 epochs. And the hyperparameter β in the Eq. (15) is set to 0 in VOC 2007 dataset and 0.4 in both MS-COCO and NUS-WIDE datasets.

4.3 Comparisons with State-of-the-Arts

Results on VOC 2007. Pascal VOC 2007 [Everingham et al., 2010] is the most widely used dataset to evaluate the MLIC task, which covers 20 common categories and contains a trainval set of 5,011 images and a test set of 4,952 images. We present the AP of each category and mAP over all categories on the VOC 2007 dataset in Table 1. Considering that the dataset is less complicated and its size is relatively small, our proposed model still achieves 94.7% mAP, which is respectively 0.4%, 1.3% and 1.6% superior over TSGCN, SSGRL and ML-GCN on VOC 2007 dataset. Particularly, our GM-MLIC obtains significant improvements in the categories of small objects, including bird, plant, sheep and cat with 98.5%, 87.0%, 98.3% and 99.4% in terms of AP.

Results onMS-COCO. Microsoft COCO [Lin et al., 2014] contains a training set of 82,081 images and a validation set of 40,137 images, and covers 80 common categories with 2.9

instance labels per image. The number of labels of different images also varies considerably, which makes MS-COCO more challenging. As shown in Table 2, GM-MLIC outperforms all baselines in terms of all evaluation metrics in most cases. Specifically, GM-MLIC is superior to other comparing methods with 84.3% mAP, 78.3% (74.9%) CF1 (Top-3), 80.6% OF1, 67.3% Top-3 CR and 69.8% Top-3 OR, respectively. To visually understand the effectiveness of our model, we randomly select some images from different scenes and exhibit the top-3 returned labels by GM-MLIC and SSGRL in Figure 3.

Results on NUS-WIDE. The NUS-WIDE dataset [Chua et al., 2009] contains 161,789 images for training and 107,859 images for testing. The dataset is manually annotated by 81 concepts, with 2.4 concept labels per image on average. Experimental results on this dataset are shown in Table 3, which is clearly to observe that GM-MLIC can not only effectively learn from such large-scale data, but also achieve superior performance on most evaluation metrics with 62.2% mAP, 61.0% CF1, 74.1% (72.5%) OF1 (Top-3), respectively.

4.4 Further Analysis

Ablation Studies. To evaluate the effectiveness of each component in our proposed framework, we conduct ablation studies on the MS-COCO and VOC 2007 datasets, as shown in Table 4. Specifically, using label semantic graph Gl would result in 1.3% higher precision but 0.7% lower recall score than using instance spatial graph Go solely. In other words, introducing Go into our framework would reduce the possibility of missing instances in images. Besides, the instancelabel matching edge that incorporating category semantics can guide the model to better learn semantic-specific features and further improve the classification precision.

Hyper-parameter and Visualization. Furthermore, we study the performance of our proposed method given different parameter settings. We first vary the number m of the extracted instances per image, and show the results in Figure 4 (a). Note that, if we keep all instances, the model will contain a lot of noise instances, which is difficult to converge. However, when too many instances are filtered out, the accuracy drops since some positive instances in images may be removed incorrectly. Empirically, the optimal value of m is set to 10 in VOC 2007 dataset, and 35 in both MS-COCO and NUS-WIDE datasets. Then, we show the performance results with different numbers of convolution modules k for our model in Figure 4 (b). With the number of convolution modules increases, the accuracy drops on all the three datasets. Thus, we set k = 2 in our work. In Figure 5, we visualize the classifiers learned by our proposed method, which demonstrates that the meaningful semantic topology is maintained.

5 Conclusion

In this paper, we propose a novel graph matching based multilabel image classification learning framework GM-MLIC, which reformulates the MLIC problem into a graph matching structure. By incorporating instance spatial graph and label semantic graph, and establishing instance-label assignment, the proposed GM-MLIC utilizes graph network block to form structured representations for each instance and label by convolving its neighborhoods, which can effectively contribute the label dependencies and semantic-aware features to the learning model. Extensive experiments on various image datasets demonstrate the superiority of our proposed method.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 61872032, 61972030, 62072027, 62076021), the Beijing Natural Science Foundation (Nos. 4202058, 4202057, 4202060), and in part by the Fundamental Research Funds for the Central universities (Nos. 2020YJS036, 2019YJS044, 2020YJS026).