Learning Visual Symbols for Parsing Human Poses in Images

1 Introduction

Parsing human poses in images has been a classical topic in artificial intelligence for decades [Marr, 1982]. This research facilitates a number of fundamental studies ranging from visual perception [D'Ausilio et al., 2012] to computer vision [Felzenszwalb and Huttenlocher, 2005], to particularly cognitive robotics in recent years [Jenkins et al., 2007].

We focus on learning visual representations of body parts in this parsing process, which we call visual symbols. [Marr, 1982] has already argued that any meaningful representation of the human body should be self-contained in a semantic hierarchy. In his work, the main ingredients of "selfcontainedness" are (i) self-contained unit must have a limited complexity, such that (ii) information appears in geometric contexts appropriate for recognition, and (iii) the representation can be processed flexibly.

However, recent research is contrary to this intuitive philosophy. The body parts are frequently represented by plain cardboard models (e.g., joints or limbs only). Since each part is not distinctive, the visual units are considered as an approach for computing probability of the body part locations, and the geometric contexts are coarsely defined as simple distributions between parts. As such, the inference models may have to go beyond tree structures to model long range interactions in this coarse structure (e.g., [Sun et al., 2012]). This essentially makes the problem less tractable, and approximate inference has to be adopted.

Can we still use exact and fast inference to remedy the problems caused by the deformable nature of human beings? We propose to use compositional parts and exploit symbolwise geometric contexts map for effective pose parsing in still images. Our part representation is compositional, i.e., each part may contain one or more physical joints of the human body, and the relationship between two parts can be either hierarchical (parent-child, e.g., leg and upper leg, Fig. 1) or flat. This allows us to categorize distinctive visual features for body parts, which are the descriptors that characterize the properties of image patches, and eventually model pairwise interactions in fine granularity.

A popular view of the geometric context is that pairwise relationship between two parts can be characterized by a distribution. Fig. 1b shows a distribution of relative locations between upper/lower leg (denoted by green/blue colors) and "leg" (denoted by cyan), respectively.

It is seemingly legitimate to assume both point sets satisfy normal distributions, but let us take a further look at the data. Assuming we have learned the symbols of the "leg" as depicted in Fig. 1c. We can group all the instances in each point set to a few categories (Fig. 1c and 1d). Further, we redraw the relative locations of upper/lower leg that are only associated to the corresponding symbols of the leg (Fig. 1e). Clearly, we have two observations: 1) relative locations may have different distributions and exhibit different characteristics, and 2) some categories may have similar distributions, but they are compactly distributed and have much smaller variances compared to Fig. 1b. Therefore, it is much easier to model these subsets separately.

The concept of symbols, although it is not new in vision and artificial intelligence, has not been used explicitly in many state of the art pose estimation methods. Such a set of learned symbols enables encoding the symbol-wise geometric information in a finer scale, and provides more information in inference. Therefore, it is critical to learn symbols for compositional parts.

With the help of geometric information, we categorize visual features from part instances, and use cross validation to select the best categories. We used Histogram of Oriented Gradient (HOG) [Dalal and Triggs, 2005] in this paper. HOG is frequently used as the feature for the appearance model of body parts. In this descriptor, an image is divided into smaller regions called cells, and the histogram of gradient directions of the pixels within each cell is calculated as the descriptor.

Following human kinematic structure, we derive an approach to effectively estimate human poses. We demonstrate the performance of our method in two large datasets, and our method outperforms the state of art methods.

Our approach has the following contributions:

• We explore an effective procedure for learning selfcontained symbols of body parts in parsing poses.
• Our symbol-wise context map naturally encodes the legitimate combinations of human poses from images.
• Our representation is very flexible, therefore, it is compatible with the majority of the popular inference algorithm in human pose estimation.

2 Related work

Marr is among the first to propose a hierarchical organization of body for parsing human poses [Marr, 1982]. Each model in this hierarchy is a self-contained unit, and the geometric contexts among these units are designed for recognition. His structure motivates a number of approaches in artificial intelligence, and evolves to deformable models in recent years, where pose estimation has been formulated as a part based inference problem.

Pictorial Structure Model (PSM) [Felzenszwalb and Huttenlocher, 2005] is one of the most successful deformable models. A tree structure graphical model is used, and pairwise terms are based on the relative distances between corresponding body parts. [Yang and Ramanan, 2011] proposed a mixtures-of-parts model for articulated pose estimation. Instead of modeling both location and orientation of body limbs as rigid parts (e.g., [Andriluka et al., 2009]), they used nonoriented pictorial structures with co-occurrence constraints.

Their work relies on the geometry to define clusters (called "types" in their paper). Therefore, the representation is less self-contained fromMarr's point of view.

Following this research direction, [Sun and Savarese, 2011] used predefined symbols for a simultaneous detection of body parts and estimation of human poses. [Tian et al., 2012] used compatibility maps in a tree structure. Latent nodes encode compatibility between parts, and accuracy is improved because incompatible poses are pruned. However, their "type" variables are solely based on the geometry, and do not encode visual information.

Beyond tree structures, graphical models were proposed in pose estimation. [Tran and Forsyth, 2010] evaluated the performance of human parsing in full relational model. A recent work by [Sun et al., 2012] showed the solution of a loopy star model using Branch-and-Bound strategy. These structures usually lead to better performance, but all require efficient approximate inference methods.

All above methods used various definitions of card-board style parts, such as limbs [Andriluka et al., 2009] and joints [Yang and Ramanan, 2011]. This may appear easier to model, but essentially the features are less distinctive and the performance is limited by coarse geometric context. Beyond these plain structures, researchers seek to compositional parts in recent years to characterize higher level visual representations.

[Wang et al., 2011] proposed to use compositional parts to provide more precise results, but his method has a higher computation cost due to the loopy graph structure. [Rothrock and Zhu, 2011] cast human pose into AND/OR graphs, and performed human parsing using top-down scheme. Rich appearance models were adopted to estimate human parts. [Bourdev et al., 2010] proposed to use poselets for human recognition. Each poselet refers to a combined part that is distinctive in training images. Please note that these poselets do not characterise geometric contexts in modeling pairwise distributions, which makes it less effective to fully capture the body dynamical structures.

3 Approach

Define the set of M parts as P = {Pi}, i ∈ [1, ...,M]. One of our goals is to learn a symbol set S such that an instance of a body part can be labelled by an entry in S.

In many state of the art datasets, the locations of primitive parts (e.g., joints) are manually annotated. Therefore, the goal of our approach is to learn and assign symbols for compositional parts in the training set, and to detect compositional parts in test images by inferring their symbols.

We first present our approach to learning visual symbols for compositional parts, then we derive the compatibility map for fine scale modeling of geometric context. Finally, we adopt an efficient learning and inference method when the structure of graphical models is a tree.

3.1 Learning symbols for compositional parts

Let G = (P,E) denote the relationship graph, where P denotes the body parts, and E is the set of edges that denote the pairwise relationship between parts. An instance of a compositional part pi = (loci, si), where loci is the image coordinate, and si denotes visual appearance.

We exploit the advantages of both geometric and visual information. We first use geometric information to coarsely group image patches for pi to different clusters, then we categorize the visual features in each cluster to visual symbols. This is more computational efficient than first categorizing visual features then grouping locations to symbols.

Geometric grouping

Given a pair of connected parts (pi, pj) in G, we use the first part pi as reference, and calculate the relative locations of pj with respect to pi. As a result, all samples of pj are projected into 2D geometry space.

For efficiency consideration, we run k-mean to group samples in kj geometry clusters, each of which denotes a geometry type of pj.

Discovering visual categories

Instances for compositional parts still have large variations in appearance within a geometric type. Therefore, the geometry alone is not powerful to characterize symbols, and we need to further learn visual categories in each geometric group for generating more discriminative visual symbols.

Fig. 2 illustrates the learning process. Given a set of instances of a part (Fig. 2a), our approach categorizes these instances to a number of subsets that are meaningful both in geometric context and visual appearance (Fig. 2b), and summarize it to symbols by a set of linear weights (Fig. 2c)

Let φ(I, pi) denotes the visual feature of pi in the image I. For instances of pi within the same geometry context, we aim at learning linear classifiers to categorize visual features.

We followed [Divvala et al., 2012] and built a Latent Support Vector Machine model for learning visual categories. Given N1 positive instances of a compositional part, and N2 negative instances, we learn K categories of this positive set. This allows us to generate the labels L = l1, l2, · · · , lN1+N2 , li ∈ [1,K] for each instance. Our objective function is as follows

where yi = {1,-1} denotes whether yi is from positive or negative sample sets, and wk are the weights of the feature map for each part. Similar as other clustering methods, we use k-mean for initializing categorization.

Cross validation

To achieve effective training when number of visual samples is small, we turn to a cross-validation learning paradigm to discover the best classifiers and fine tune the performance. The main idea of cross validation is to perform a training step followed by a validation step [Singh et al., 2012]. During the validation stage, each classifier is evaluated on the validation set, and weak classifiers with few detected samples will be removed from the classifier set.

The whole training process is conducted iteratively. Algorithm 1 illustrates the whole training process. After cross training the "survived" classifiers can be regarded as a mixture of visual symbols.

Algorithm 1 Cross validation.

Input:

H : training set for a compositional part Pi;

Kin : the number of classifiers.

Output:

Kout : the number of visual categories; wk: linear classifiers k = [1, ...,Kout].

Procedure:

1. Divide the training set equally to H1 and H2;
2. Train the classifiers w[1,...,Kin] on H1 using Eq. 1;
3. Evaluate the classification result on H2;
4. If the number of detected samples for wi is small: Remove wi from classifiers and Kin = Kin - 1 ;
5. Swap H1 and H2;
6. Repeat step 1-5 for t times (t = 10 in our experiments);
7. Kout =Kin and output w[1,...,Kout] ;

Discussion

The visual categorization process of a compositional part characterizes the appearance models in a way that they can be regarded as "templates". When HOG feature is used, the set of learned weights is also considered as "HOG filters".

In this way, our symbols encode both geometric and visual appearance information. This makes our descriptors different from other work, because they are more discriminative and representative.

3.2 Defining symbol-wise geometric context

Assigning each part a symbol allows us to build a compatibility map for any pair of symbols. Assume we have two compositional parts pi and pj, each of which has symbols si and sj, respectively, we create the pairwise compatibility term between parts as follows.

where ψij(pi, pj) = [dx, dy, dx2 , dy2 ] denotes the relative distance between pi and pj , ωsisj ij denotes the symbolspecific weights, and bsisj ij denotes the bias of the compatibility. If two symbols are not "compatible", i.e., they never exist in any training image together, this bias term is -∞. Both terms are learned in Sec. 3.3.

In graphical models, we frequently model the energy minimization problem by passing messages from one node to the other. This message passing step is practical in both exact inference or inexact inference. We can utilize this compatibility term, which result in fine scale message passing. For node Pj , the incoming message mk→j (pj , sj ) from other nodes Pk and the outgoing message mj (pj , sj ) are computed as:

where n(j) denotes the neighbors of pj in G.

3.3 Inference

Given a set of visual symbols, compatibility map, and their graph G, one can learn the parameters and perform inference. When the structure of G is a tree, the inference is exact.

In our experiment, we define the following compositional parts (Fig. 2d). Semantically, our structure has three levels: "upper body" and "lower body" as a coarse modeling of the human body, head, upper and lower limbs used in midlevel description, and joints in the fine level.

Appearance term We use HOG templates to represent each visual subcategory. For each part pi in an image I, the appearance score of a local patch can be written as [Yang and Ramanan, 2011]

where ωsi i is the weight for symbol si in the ith part. This term can be initialized by the results wk in Eq. 1.

Deformable term Pairwise term between pi, pj is defined as the symbol-wise context (Eq. 2). This term can be computed effectively by distance transform in inference.

Objective function Our objective function is as follows

Since our model is a tree, standard message passing algorithm (Eq. 3 and Eq. 4) and exact inference are applicable.

Learning model parameters The objective function (Eq. 6) can be rewritten as f = 〈θ,Φ〉 , where 〈·, ·〉 is the inner product, θ consists of image filters for single parts (ωsi i ), pairwise deformable weights (ωsisj ij ) and biases bsi,sj ij , and Φ(In, p) denotes the concatenated features from appearance and deformable components. The learning of θ amounts to the quadratic optimization:

This is a standard quadratic programming procedure, and can be solved effectively.

4 Experiments

We present our experiments in this section. First, we describe the datasets we used for evaluation. Then, we demonstrate the visual symbols learned by our method, and compare our approach against four other methods.

In our experiments, we extract HOG features on grid image with 4 × 4 pixels from image patches, and learn visual symbols and geometric context map. The number of geometric cluster kj is 8 for large compositional parts, and 6 for small parts, and the number of visual symbols for each geometric types is set to 2 and 4, respectively. The number of geometric clusters is consistent with that in [Yang and Ramanan, 2011]. The final number of the appearance clusters depends on the cross validation. As a result, we learn 8 to 20 visual categories for each visual symbol after cross validation.

4.1 Dataset

We evaluate our performance on two large datasets, namely, Image Parse dataset [Ramanan, 2007] and Leeds Sport dataset [Johnson and Everingham, 2010]. In all experiments, we used 500 images in the negative set of INRIA person dataset as negative samples ([Dalal and Triggs, 2005])

Image Parse dataset

Image Parse dataset (PARSE) contains 305 images with annotated poses. This dataset has images from various human activities, background and different illuminations. All images are resized such that human in images have roughly the same scale (150 pixels in height). We used 100 images for training, and the rest for testing.

LSP dataset

The recent Leeds Sport Dataset (LSP) contains 2000 images. This collection has a larger variation of pose changes. Humans in each image were cropped and scaled to 100 to 150 pixels in height. The dataset was split evenly into training set and testing set.

4.2 Demonstration

We demonstrate the effectiveness of learning procedure in this section. Fig. 3 shows localization results for the eleven higher level compositional parts, as well as examples of their filters learned by our method. In this visualization, we use different colors and line types to denote different parts.

Filters for visual symbols Each filter in Fig. 3 exhibits a few characteristics for the corresponding compositional part. These filters are related, in the sense that they model the human body at different levels. Each filter is also selfcontained,i.e., stand alone part detectors for different tasks.

This intrinsic constraint facilitates the inference. For instance, the torso (yellow) filters indicate the body inclination in a coarse level, which limits the search of head position (red) both in geometric context and appearance.

Interpretation The localization results can be regarded as multi level symbolic annotations for the human body. Therefore, this can be used for a number of applications ranging from high level understanding to low level description. We can further assign semantic meaning to these symbols. For instance, the torso and lower body locations can be used for analyzing the actions of the human beings ("stretching", "standing", "squat", etc), and the midlevel limb detection results are used for motion analysis ("extending arm", "fetching", etc.).

4.3 Comparison

We compared our approach against four state of the art methods for human poses parsing on the PARSE and the LSP dataset in this section. In our comparison, we used the criterion in [Ferrari et al., 2008] for performance evaluation. A part is correctly detected if both its endpoints are within 50% of the length of corresponding ground truth segments. Then we used the probability of a correct pose (PCP) to measure the percentage of correctly localized body parts.

Table 1 summarizes the evaluation results, with highest scores being highlighted. We compared the parsing accuracy of our method against [Andriluka et al., 2009], [Yang and Ramanan, 2011], [Johnson and Everingham, 2010], and [Tian et al., 2012], respectively. We re-run the method of [Andriluka et al., 2009] and [Yang and Ramanan, 2011] on the datasets and report the results, and other data entries in the table are from the original papers, respectively. Please note that [Tian et al., 2012] tried two different settings in their methods, but they did not report the result using all the 1000 images in the training set.

Overall our method achieved the best performance. In the PARSE dataset, our method is marginally better than the original algorithm (75.7% vs 74.9% ). This is possibly because the power of our visual category training may not be fully explored due to small number of training samples.

The problem caused by limited training samples is relieved in the LSP dataset. When 1000 images are used for training, our method outperforms other methods. Compared to four methods, our total detection accuracy (65.2%) is consistently higher. Our performance is also superior to [Johnson and Everingham, 2011], where 11000 samples were used for training 1 . This suggests that our training is effective.

Fig. 4 shows some examples of parsing results for three methods. Each triplet contains results for [Andriluka et al., 2009], [Yang and Ramanan, 2011] and ours. The parsing results show that our method produces visually pleasing poses. The interaction between high level and low level compositional parts makes our results more "balanced". For instance, detection results for two legs are well separated if the corresponding symbols of the lower body part are detected, because the symbol-wise geometric context naturally guides the maximization (Eq. 6) to this optimal solution. Our method also tries to make the best guess of self-occluded parts.

1 Due to a large number of missing labels in this dataset, we do not perform our evaluation on it. In their method, the training samples were automatically relabeled during optimization.

The training takes approximately 8 hours on a 2.8GHz Quad Core CPU with 6GB memory. Test takes 1.5s for an 320 × 240 image. Compared to models where loopy BP is used, our tree structure essentially speeds up the training. The running time of our method is in the same order of magnitude of [Yang and Ramanan, 2011]. Therefore, our method strikes a balance between accuracy from long range interaction and the efficiency from exact inference.

5 Conclusion

This paper presents a novel approach to learning selfcontained representations for parsing human poses in images. The main contribution is the visual symbols that facilitate geometric context modeling. Our method can be used for many graphical models. When the model is a tree, we demonstrate that our method outperforms four current methods.