Autonomous Object Manipulation: A Semantic-Driven Approach

1 Introduction

Research on grasping and manipulation is nowadays focusing on how to make this task as autonomous as possible: recent advances in the fields of computer vision, knowledge engineering and neurophysiology show that a variety of techniques can be used to get closer to this goal. One of the major issues is given by the high dimensionality of the problem, as many variables occur both in the object recognition and in the grasp decision processes.

Different data representations have been studied to deal with the grasping problem [Rubio et al., 2010; Huebner et al., 2009], the main aim being to find a suitable representation of object and grasp types, in order to reduce the complexity of the problem. Most of the proposed representations focus on the visual characteristics of the objects more than on their "conceptual" representations. In our project we are exploring a semantic representation of objects using a suitable description logic (DL) formalism, which should take into account the qualitative and quantitative characteristics that can be useful for grasping and manipulation tasks.

The decomposition of an object into parts has been shown to be both close to the way human beings reason about objects [Biederman, 1987] and effective in reducing the complexity of computing tasks like object recognition (e.g., [Agarwal et al., 2004]). The semantic description of the parts composing an object and their mutual relationships is a difficult problem: apart from the problems related to the image analysis phase, the vagueness affecting real world data makes formal representation of an object structure quite inconvenient. Novel tools such as fuzzy DLs [Lukasiewicz and Straccia, 2008], which extend crisp DLs by adding imprecision and vagueness in the reasoning process, may thus be used to bridge qualitative and quantitative reasoning in a unified framework.

∗Research project supervised by Prof. Giuseppina Gini.

The problem of semantic part decomposition is still an open problem and, to the best of our knowledge, there are no tools available to automatically create a fuzzy ontology from raw data taken from an image. The use of fuzzy DLs for object recognition has been investigated in some works such as [Hudelot et al., 2008], in which little advantage is taken from the (partial) fuzzy extension and from the expressivity of the used logic (i.e., no cardinality restrictions are used); furthermore, a preliminary phase of semantic annotation of the images by domain experts has to be performed.

Some recent works make a thorough use of fuzzy DLs to reason about multimedia information (e.g., [Simou et al., 2007]) but, again, little advantage is taken from the expressivity given by cardinality restrictions (when available). Generally speaking this is due to the fact that, for scene understanding purposes, it is sufficient to know whether an object is present or not, while, for object recognition purposes, it is often necessary to be able to count the number of similar instances of the same class.

2 Current work

As shown in the previous section, the first problem arising in an autonomous manipulation task is given by object localization and recognition. The input image has to be segmented in a meaningful way, in order to distinguish each object from the background and from other objects; we are now focusing only on the shape of the objects (thus using synthetic models), as the first step is to make the system able to build its own representation of each object. The problems due to occlusions and cluttering have to be faced after some knowledge about the object to recognize has been gathered, so to use it for trying to solve ambiguities.

The most delicate part of the process is given by the decomposition of an object in its composing parts, firstly as an image or mesh processing task. We are currently taking advantage of some concepts pertaining to the Gestalt theory and its application to computer vision [Desolneux et al., 2004], such as proximity, closure and similarity, which can be described in a formal and efficient way. Some studies in the field of developmental robotics have been performed on how to generalize such relations by using perceptual functions [Modayil and Kuipers, 2007], but this is at the moment out of our scope.

The formalization is built on top of fuzzy DLs, which give a chance both to ground qualitative concepts as "close to" and "connected to" to quantitative real-world data and to define a degree of membership of an object to a certain class. The available tools (as the fuzzy reasoners FuzzyDL 1 and FiRE 2 ) are still under development, so there are several limitations to take into account in the formalization process.

Fuzzy DLs are used to represent the structure of the objects, e.g., "a fork is composed of a long and thin part (the handle) and four short and thin parts (the tines)", while the ontology of grasping types currently contains axioms like

GRASPABLEBYPINCH ≡ MEDIUMLENGTHOBJECT ⊓ HIGHLYECCENTRICALOBJECT

which are related to the geometrical characteristics of the objects (and object parts) involved. The role of the latter ontology is to determine (even offline) whether a specific kind of grasp can be performed or not on a selected object or on one of its composing parts.

From the semantic point of view, the most difficult problem is to define what is to be called a part; this is a well known problem in logic faced in the mereotopology framework [Varzi, 2007; Keet and Artale, 2008], in which formal properties of the part-whole relation are extended to take into account the notion of connection. The work on fuzzy spatial relations which has been conducted in [Hudelot et al., 2008] makes use of structural information for helping the recognition process; fuzzy extensions of the standard RCC properties are used for dealing with medical images, in order to focus on conceptual representations more then on concrete data. In the object recognition phase we use semantic information in a similar fashion, the main differences being that such information is automatically produced (i.e., not provided by a domain expert) and the reasoning is performed via a fuzzy reasoner.

Some results of our research are shown in [Vitucci et al., 2010], in which a common household object is used as an example of the building of a semantic representation.

3 Conclusions and future work

The images and models we are currently using represent synthetic household objects and toys, but we aim to realize a system capable of dealing with real life objects. There are many challenges to face, the most important being an efficient segmentation process and an unambiguous definition of part, in order to build reasonable yet effective representations; moreover, the problem of occlusion, i.e., of distinguishing objects which are superimposed on other objects, is critical. To address both problems, we are planning to take advantage of a stereo camera (such as a Kinect) in order to exploit more spatial information and obtain suitable object models after a reconstruction phase. A big challenge will be to effectively implement such a reasoning system on a robot and to perform experiments in a real environment (e.g., a kitchen).

Finally, we are working on how to assess the plausibility and efficacy of a grasping act and include such information in the grasp ontology.

1 http://gaia.isti.cnr.it/ straccia/software/fuzzyDL/fuzzyDL.html

2 http://www.image.ece.ntua.gr/ nsimou/FiRE/