Object Recognition Based on Visual Grammars and Bayesian Networks

I Introduction

Most current object recognition systems are centered in recognizing certain type of objects, and do not consider their structure. This implies several limitations: (i) the systems are difficult to generalize to any type of object, (ii) they are not robust to noise and occlusions, (iii) the model is difficult to interpret. This paper proposes a model that achieves a hierarchical representation of a visual object in order to perform object recognition tasks, based on a visual grammar [Ferrucci et al., 1996] and Bayesian Networks (BN's). Thus, we propose the incorporation of a visual grammar in order to develop an understandable hierarchical model so that from basic elements (obtained by any image segmentation algorithm) it will construct more complex forms by certain rules of composition defined in the grammar, in order to achieve object recognition in certain context (e.g. images of objects indoors). The importance of using a hierarchical approach is that it can build a more robust model to noise and occlusions, also the BN model can work with incomplete evidence. In addition, the model expresses the grammar in an understandable way to a human, in order to interpret the model and even modify the structure.

There are several works using a hierarchical approach [Chang et al., 2011; Felzenszwalb, 2011; Melendez et al., 2010; Zhu and Mumford, 2006]. We propose SymbolRelation grammars (SR-Grammars) because they can represent relationships between elements using predicate logic; the transformation into a BN can deal with uncertainty and we can do inference in order to perform object recognition tasks.

II Methods

The proposed method compromises two phases: (i) model construction and transformation to a BN (Fig. 1); and (ii) image pre-processing and object recognition using probabilistic inference (Fig. 2). Next we briefly describe the main steps of our method.

A Segmentation and Lexicon

The segmentation is performed with simple RGB quantization (32 levels) and edge extraction using Gabor Filters. Small regions are fused with other regions. The idea is to use a simple segmentation algorithm. These regions define a visual dictionary. Every region is described using several shape and color features. Similar regions by their features are considered as candidates to terminal elements in our grammar. All the terminal elements are described in a "Lexicon".

B SR-Grammars and Spatial Relationships

The object representation is based on SR-grammars [Ferrucci et al., 1996] which can incorporate spatial relationships between elements. In our work, these relationships determine the position of one object element with respect to another object element, at different levels in the hierarchy. Although there are different types of spatial relationships, in our model we use topological and order relationships, such as inside of or above.

C Transformation of the grammar

We transform the grammar into a BN: for every production rule Y 0 → 〈M, R〉, we produce the node Y 0 in the network and connect this node with all x ∈ M. For every relationship r(a, b) ∈ R we produce the node r connected with its parents a, b ∈ M.

D Parameter learning and object recognition

Once the BN is obtained, its parameters are learned from examples using EM, as the intermediate elements (nodes) in the BN are hidden nodes. Then, for recognition, the low level features (color segments and edges) are detected in the image in order to instantiate the low level nodes in the BN; this information is propagated to determine the probability of observing certain object.

III Results

We have applied this method for face detection [Ruiz et al., 2011]. Here we present a simple example of how an object is represented using a grammar, and its transformation to a BN.

We describe the object pumpkin, by the following grammar:

GPUMPKIN=(VN,VT,VR,S,P, /O);

VN={PUMPKIN,Stem,Fruit}; VT={Bh,Bv,Hg1,Hg2,Hg3}; VR={above,ady}; S=PUMPKIN; P:

• 1: PUMPKIN0 →<{Stem2 , Fruit2 }, {above(Stem, Fruit)}>

2: Stem0 → <{Bh2 ,Hg12 ,Bv2 ,Bv3 }, {above(Bh2 ,Hg12 ), ady(Bv2 ,Hg12 ), ady(Hg12 ,Bv3 ), ady(Bv2 ,Bh2 ), ady(Bh2 ,Bv3 )}>

3: Fruit0 → <{Bh3 ,Bh4 ,Hg22 ,Hg32 ,Bv4 ,Bv5 }, {above(Bh3 ,Hg22 ), ady(Bv4 ,Hg22 ), ady(Hg22 ,Hg32 ), above(Hg32 ,Bh4 ), ady(Hg32 ,Bv5 )}>

where the Lexicon is given by: i) Hg as homogeneous region, ii) Bh as horizontal edge and iii) Bv as vertical edge. Numbers associated to these elements represent minimal variations in its features (color). The BN generated from this grammar is depicted in the Fig. 3.

In this example the probabilistic inference detects the regions which probably represent a pumpkin. Results of detected objects are illustrated in Fig. 4. The configurations presented have maximum likelihood given by inference in the BN. 1

1 Additional information of our model can be found on the website: http://srmodel.erzh.org

(a) pumpkin1

(b) pumpkin2

• (c) pumpkin3

Figure 4: Regions detected for a pumpkin example. In spite of the simple segmentation algorithm, the grammar helps to detect parts of the object.

IV Conclusion

A first stage in the design of a visual grammar to detect objects was described. This approach combines Symbol Relation grammars and Bayesian networks to describe an object in an image. This model was tested for natural objects (like pumpkins) with low level features as terminal elements with promising results. The next stages in our research are to learn the SR grammars from examples and apply this formalism to other classes of objects, such as those in service robotics scenarios.