One-Shot Affordance Detection

1 Introduction

The concept of affordance was proposed by the ecological psychologist Gibson [Gibson, 1977]. It describes how the inherent "value" and "meanings" of objects in an environment are directly perceived, and explains how this information can be linked to the action possibilities offered to an organism by the environment [Hassanin et al., 2018]. After Gibson put forward the definition of affordance, a lot of cognitive psychologists made profound studies on the relationship between object affordance and perceiver intention in recent decades [Heft, 1989]. In particular, perceiving object affordance in unseen scenarios is a valuable capability and has a wide range of applications in scene understanding, action recognition, robot manipulation, and Human-Computer Interaction [Zhang and Tao, 2020].

∗Wei Zhai is an intern at JD Explore Academy.

†Corresponding Author

To learn such capability of perceiving affordance, we consider the challenging one-shot affordance detection 1 problem in this paper, i.e., given a support image that depicts the action purpose, all objects in a scene with the common affordance should be detected (see Figure 1). Unlike the object detection/segmentation problem [Shaban et al., 2017], affordance and semantic categories of objects are highly intercorrelated but do not imply each other. An object may have multiple affordances (see Figure 2), e.g., the sofa can be used to sit or lie down. Actually, the possible affordance depends on the person's purpose in real-world application scenarios. Directly learning the affordance from a single image without the guidance of purpose makes the model tend to focus on the statistically dominant affordances while ignoring other visual affordances that are possibly suitable for completing the task.

To address this problem: 1) We try to find clear hints about the action purpose (i.e., via the subject and object locations [Chen et al., 2020]) from a single support image, which implicitly defines the object affordance and is a reasonable setting in real-world unseen scenarios. 2) We adopt collaboration learning to capture the inherent relationship between different objects to counteract the interference caused by visual appearance differences and to improve generalization. Specifically, we devise a novel One-Shot Affordance Detection (OS-AD) network to solve the problem. Taking an image as support and a set of images (5 images in this paper) as a query, the network first captures the human-object interactions from the support image using a purpose learning module (PLM) to encode the action purpose. Then, a purpose transfer module (PTM) is devised to use the encoding of the action purpose to activate the features in query images that have the common affordance. Finally, a collaboration enhancement module (CEM) is devised to capture the intrinsic relationships between objects with the same affordance and suppress backgrounds that are irrelevant to the action purpose. In this way, OS-AD can learn a good adaptation capability for perceiving unseen affordances.

1 "Detection" follows the term pixel-wise detection task, which has also been used in the area of salient object detection.

Moreover, the existing datasets still have gaps relative to real application scenarios, due to the limitation of its diversity. The affordance detection for scene understanding and general applications should be able to learn from the humanobject interaction when the robot arrives at a new environment and retrieves the objects in the environment, rather than just finding objects with the same categories or similar appearance. To address the limitations of the datasets, we propose the Purpose-driven Affordance Dataset (PAD), which contains 4,002 diverse images covering 31 affordance categories as well as 72 object categories from different scenes. Moreover, we trained several representative models to address the problem and compare them with our OS-AD model comprehensively in terms of both objective evaluation metrics and visual quality.

Contributions (1) We introduce a new one-shot affordance detection problem along with a benchmark to facilitate the research for empowering robots with the ability to perceive unseen affordances in real-world scenarios. (2) We propose a novel OS-AD network that can efficiently learn the action purpose and use it to detect the common affordance of all objects in a scene via collaboration learning, resulting in a good adaptation capability that can deal with unseen affordances. (3) Experiments on the proposed PAD benchmark demonstrate that OS-AD outperforms state-of-the-art models and can serve as a strong baseline for future research.

2 RelatedWork

2.1 Affordance Detection

The problem of visual affordance detection has been investigated for decades [Hassanin et al., 2018], it is a widely used strategy in AI community to perceive action intentions and thereby to infer visual affordance from the image/video of human and objects. Early works mainly attempted to establish and learn an association between the apparent characteristics of objects and their affordance for perceiving affordance. [Myers et al., 2015] proposed a framework for jointly locating and identifying the affordance of object parts and presented the RGB-D Part Affordance dataset, which is the first pixel-wise labeled affordance dataset. However, the affordances of objects do not simply correspond to representational characteristics, which are shifted in response to the state of interactions between objects and humans. Therefore, [Chuang et al., 2018] considered the problem of affordance reasoning in the real world by taking into account both the physical world and the social norms imposed by the society, and constructed the ADE-Affordance dataset based on ADE20k [Zhou et al., 2017]. Since the change of an object's affordance state is usually due to the interaction between the human and the object, further research has begun to consider human action as a cue for learning affordance. [Fang et al., 2018] used human-object interactions in demonstration videos to predict the affordance regions of static objects via linking human actions to object affordance, and proposed the OPRA dataset for affordance reasoning. Different from the above existing works, our proposed method aims to explore the multiplicity of affordance by a collaborative learning strategy. To a certain extent, our work conforms to Gibson's definition of affordance that "it implies the complementarity of the animal and the environment". Since there exist multiple potential complementarities between animal and environment, it leads to multiple possibilities of particular affordance. To address this issue, we present a novel task of one-shot affordance detection, in which action intension is introduced through support image to alleviate the multiplicity of affordance.

2.2 One-Shot Learning

Recently, one-shot learning has received a lot of attention and substantial progress has been made based on metric learning using the siamese neural network [He et al., 2021; Koch et al., 2015; Snell et al., 2017]. Besides, some works build upon meta-learning and generation models to achieve one-shot learning. Specifically, [Michaelis et al., 2018] proposed the problem of one-shot segmentation in clutter, finding and segmenting a previously unseen object in a cluttered scene based on a single instruction example. [Zhu et al., 2019] proposed a one-shot texture retrieval, given an example of a new reference texture, detecting and segmenting all the pixels of the same texture category within an arbitrary image.

Different from these studies, we aim to identify the potential action possibilities (affordance) of objects in unseen scenarios, given a single support image that implicitly defines the action purpose without an explicit mask to denote the object affordance, which is very challenging but of practical value.

3 Method

Our One-Shot Affordance Detection (OS-AD) network is shown in Figure 3. During training, it receives a single support image that implicitly defines the affordance via humanobject interactions, e.g., in the form of bounding boxes of the person and the object interacting with the person, which are weaker signals and easier to acquire than a pixel-wise affordance mask. Meanwhile, a set of query images (5 in this paper) containing objects with the same affordance. Our goal is to estimate the action purpose from the support image, transfer it to the query images, and learn to segment all the objects having the same affordance. To this end, three modules named PLM, PTM, and CEM are devised. During testing, OS-AD can recognize the common affordance from any number of query images given a single support image.

3.1 Framework

Given a set of query images I = {I1, ..., In} and a support image Isup containing human-object interactions, we first extract the features of I and Isup using resnet50 [He et al., 2016] to obtain their feature representations X = {X1, ..., Xn} and Xsup, respectively. We then feed Xsup and the bounding boxes of the human and object into PLM to extract information about the human-object interaction and encode the action purpose. As shown in Figure 3, our network needs to estimate the purpose that the person wants to swing. Subsequently, we feed the feature representation of the action purpose and X into PTM, which transfers the action purpose to X, enabling the network to learn to attend those objects with that affordance. Finally, the encoded features are fed into CEM, which captures the intrinsic connections between objects with the same affordance and suppresses irrelevant object regions, predicting the common affordance masks.

3.2 Purpose LearningModule

As shown in Figure 4 (a), we feed the Xsup and the bounding boxes of the person and object into the purpose learning module to estimate the action purpose of the person. We first use the bounding boxes to extract the features of the person and object fromXsup as XH and XO. Subsequently, inspired by human-object interaction and visual relationship detection [Zhan et al., 2019; Zhan et al., 2020], the features of the instance (person or object) can provide guidance information about where the network should focus, we use the features of the person and object to activate Xsup respectively to obtain MH and MO:MO = Softmax(fO ⊗ Xsup) ⊗ Xsup, (1)

where fO and fH are the representations of XO and XH respectively after global maximum pooling (GMP). ⊗ denotes element-wise product. We then use fO to guide the network to focus on the area of human-object interaction MHO:

Then, MHO is multiplied with MH and MO respectively to activate the relevant features of human-object interaction on the global features, which are added together and go through a GMP layer to get the encoding of the action purpose Fsup:

where ⊙ denotes position-wise dot product.

3.3 Purpose Transfer Module

After obtaining Fsup, we transfer it to each query image to segment the objects that can fulfill that purpose, i.e., having the common affordance. As shown in Figure 4 (b), we perform correlation calculations between Fsup and each position of the query image. After normalization, we get the attention probability map that may contain objects with the common affordance. It is used to enhance the feature that may contain the affordance object and suppress the area that does not contain the affordance object. Finally, the attended feature is added back to the original feature to get XT:

3.4 Collaboration Enhancement Module

There are some inherent characteristics between objects with the same affordance. For example, both cups and bowls have concave areas in the middle so they can be used to hold water. Thus, by discovering intrinsic features from the query image collection [Ma et al., 2020], objects with the same affordance can be activated and unrelated regions can be suppressed, resulting in better segmentation results (see Figure 4 (c)).

Inspired by the Expectation-Maximization (E-M) [Dempster, 1977] algorithm and EMANet [Li et al., 2019], we run the "E-step" and "M-step" alternately to obtain a compact set of bases, which are used to reconstruct the feature map of query images. During the iterative process, this set of bases can learn the common properties between the input set of features and thus can segment the common feature attributes. For a set of input feature maps XT = {XT1 , ..., XTn } (XTi ∈ RN×C , N = W ×H), they firstly go through a convolution layer to get F = {F1, ..., Fn}. Then, a set of base µ ∈ RK×C is initialized. The E-step estimates the latent variable Z = {Z1, ..., Zn}, Z ∈ RN×K . The weight of the k-th basis for the j-th pixel of the i-th image is calculated as:

where fij is the feature of the j-th position of the i-th image, κ denotes the exponential kernel function, i.e., exp(, ). Thus, we have Zi = Softmax(FiµT ). The M-step updates the bases and calculates µ as the weighted average of F:

After convergence of the E-M iterations, we use µ and Z to reconstruct X to obtain ˜ F: ˜ Fi = Ziµ. Finally, ˜ F is mapped to the residue space of X using convolution and added to X to obtain ˆ X, i.e., ˆXi =Xi +Conv( ˜ Fi), ˆ X = { ˆX1, ..., ˆXn}.

3.5 Decoder

For the i-th image, the output of the m- th layer is P m i = Conv(Unsample(Conv(Xm i ) + P m+1 i ), m ∈ [1, 4], where P 5 i = Conv( ˆ X5 i ). A convolutional prediction layer is used to get the final output, i.e., Dm i = Conv(P m i ), m ∈ [1, 5].

The cross-entropy loss is used as the training objective of our network. For the prediction Dm i from the m-th layer of the i-th image, we calculate the loss Lmi as:

where Pr(yij = 1|Dm ij ) is the prediction map in which each pixel denotes the affordance confidence. Y+ and Y- denote the affordance region pixels set and non-affordance pixels set, N = H ×W. The final training objective is defined as:

4 Experiments

4.1 Dataset

Data collection. We construct the Purpose-driven Affordance Dataset (PAD) with images mainly from ILSVRC [Russakovsky et al., 2015], COCO [Lin et al., 2014], etc. The affordance categories of the dataset is shown in Figure 5. Different images are collected according to the class keywords of the objects to make sure that the images in these datasets cover different scenes and appearance features of the objects, which constitute a challenging benchmark. Additionally, to make the dataset richer in categories, we obtain a part of the images from the Internet to expand the dataset. The description of each affordance and the object categories it contains are provided in the supplementary material. To benchmark different models comprehensively, we follow the k-fold evaluation protocol, where k is 3 in this paper. To this end, the dataset is divided into three parts with non-overlapped categories, where any two of them are used for training while the left part is used for testing. See the supplementary material for more details about the setting.

Data annotation. 1) Category labeling: We create a hierarchy for PAD dataset by selecting 72 common object categories (e.g. "cups", "bowls", "basketballs", etc.) and assigning affordance labels to each object category. An affordance category may contain multiple subcategories, e.g. objects with "Swing" affordance include "tennis rackets", "table tennis rackets", "golf clubs", "baseball bats", etc., and the appearance feature of these objects varies considerably. An object may contain more than one affordance category, e.g. "sofa" and "bench" have both "Sit" and "Lie" affordance labels. 2) Mask labeling: For the images from COCO etc., we select a portion of the mask labels from the original dataset. Since the above dataset may not annotate all the objects belonging to the same affordance, we select these images and annotate them manually. The images downloaded from the web are also manually annotated with the objects containing the affordance category. 3) Purpose image labeling: 656 purpose images are downloaded from the Internet which contains diverse human-object interactions. We annotate the bounding boxes of human and object accordingly.

4.2 Benchmark Setting

To provide a comprehensive evaluation, four widely used metrics are used to evaluate the performance of affordance segmentation. The IoU metric for segmentation task [Long et al., 2015] is adopted in our task. Mean Absolute Error (MAE) [Perazzi et al., 2012] is used to measure the absolute error between the prediction and ground truth (GT). Emeasure (Eφ) [Fan et al., 2018] is a metric that combines local pixels and image-level average values to jointly capture image-level statistics and local pixel matching information. Pearson Correlation Coefficient (CC) [Le Meur et al., 2007] is used to evaluate the correlation between the prediction and GT. We report the average metric score for all test images.

Our method is implemented in Pytorch and trained with the Adam optimizer [Kingma and Ba, 2014]. The backbone is resnet50 [He et al., 2016]. The input is randomly clipped from 360×360 to 320×320 with random horizontal flipping.

We train the model for 40 epochs on a single NVIDIA 1080ti GPU with an initial learning rate 1e-4. The number of bases in the collaboration enhancement module is set to K=256. The number of E-M iteration steps is 3. Besides, two segmentation models (UNet [Ronneberger et al., 2015], PSPNet [Zhao et al., 2017]), three saliency detection models (CPD [Wu et al., 2019], BASNet [Qin et al., 2019], CSNet [Gao et al., 2020]) and one co-saliency detection models (CoEGNet [Fan et al., 2021]) are chosen for comparison.

4.3 Quantitative and Qualitative Comparisons

The results are shown in Table 1, our results outperform all methods on all metrics in the three one-shot learning settings. Measured by the mean IoU value of the three-fold test, our model improves by 42.2% compared to the co-saliency detection approach. Compared to the best saliency detection model and the best segmentation model, our method improves by 42.2% and 47.6% respectively. This indicates that our method can effectively use the action purpose extracted from the support image to guide the segmentation of query images, rather than simply constructing a link between apparent features and affordance. The prediction results generated by each model are shown in Figure 6. It can be seen that our method can detect all the objects with common affordance according to the human purpose. In the 1st and 3rd rows of "Kick", the soccer ball and the punching bag are completely different in terms of apparent features, but both are kickable objects. This demonstrates that our method can detect object affordance in the unseen scenarios by extracting the action purpose and collaborative learning strategy. Meanwhile, we calculate the IoU for each affordance class, and Table 2 shows the results for several of them. The results from the first three columns show that our method can detect object affordance with different apparent features but belonging to the same affordance, indicating that our model can capture the common relation of the objects in the unseen scenarios well and detect them more completely. The two categories with the worst results out of all the results are "Jump" and "Brush", respectively. The possible reasons for the poor performance are as follows: while in the "Brush" category, its accompanying actions are not obvious and the size of the objects are relatively small, making it more difficult for the network to extract the features of the objects. In summary, our approach can learn object affordances well by transferring human purposes to new objects and capturing common features between objects using collaborative learning strategies, with good generalization ability on the task of detecting object affordance in unseen scenarios.

5 Conclusion

In this paper, we make the first attempt to deal with a challenging task named one-shot affordance detection, which has practical meaning for real-world applications, such as empowering robots with the ability to perceive unseen affordance. Specifically, we devise a novel one-shot affordance detection (OS-AD) network and construct a benchmark named purpose-driven affordance dataset (PAD). OS-AD exceeds representative models adapted from related areas such as segmentation and saliency detection, which can serve as a strong baseline for this task. PAD contains 4k images with diverse affordance and object categories from different scenes, which can serve as a pioneer test bed for this task. In the future, we plan to deploy the model in a real-world robot to complete some well-defined tasks and evaluate its performance. Going a step further, we attempt to make such a test suite available to the community via a web-based interface.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2020AAA0105701, the National Natural Science Foundation of China (NSFC) under Grants 61872327, the Fundamental Research Funds for the Central Universities under Grant WK2380000001.