Predicting dominance in multi-person videos

1 Introduction

The problem of identifying dominant people in a group setting is important for many applications. Businessmen in meetings with external partners or customers might wish to

identify the key decision maker. Government delegations may be interested in identifying the most dominant person from the other side in a negotiation.

In this paper, we study two problems: identifying the most dominant person (MDP problem) in a group-interaction video and identifying the more dominant person when looking at pairs of people in a group interaction (pairwise dominance prediction or PDP). Although the MDP problem has been previously studied in pioneering works by [Jayagopi et al., 2009] and [Aran and Gatica-Perez, 2010], we are the first to study the PDP problem. We look at two variants of each of these problems (MDP-All and MDP-Distinct, PDP-All and PDP-Distinct). The paper makes three novel contributions. First, we propose a family of Dominance Rank (DR) features, which captures the dynamics of interactions between participants in a group-interaction video. Second, we propose the Dominance Ensemble Late Fusion (DELF) algorithm that uses Dominance Rank in combination with several other features to solve all four problems. Third, we propose the Group Dominance Prediction (GDP) algorithm to solve MDP-All and MDP-Distinct.

We test the DELF and GDP algorithms on two datasets. Our first setting consists of audio-visual data of groups of people playing a variation of The Resistance game. We collected data for 33 games involving 233 players with ground truth involving surveys on who is the most dominant. Each game involves 5-8 players. The data was collected from six sites (three in the US, one each in Israel, Zambia, and Singapore). The second dataset is the widely used ELEA dataset [Sanchez-Cortes et al., 2012], which shows small groups (3-4 people) involved in a winter survival task. The Resistance and the ELEA datasets further differ in the nature of social interaction present in them. The former involves an adversarial situation and models a conflict between two groups (an informed group of spies and an uninformed group of resistance). In contrast, the ELEA dataset involves a cooperative element as players wish to solve a common task. We test DELF and GDP both against each other and against several baselines and show that DELF beats out strong baselines from past work, and GDP beats out DELF. We should note that DELF is an improvement on past work, and hence all the excellent body of past work contributes to this algorithm.

Figure 1 depicts our approach to the four problems we study in this paper. We first divide each game G's videos into equal time slices of length ∆t seconds. For each player p, we then create a basic short-term feature vector bst(p, t,G) showing the values of basic features (defined in Section 4) for player p during time slice t. The basic features fall into four categories: speech-related features [Escalera et al., 2012], facial action unit features [Baltrusaitis et al., 2018], emotion-related features from Amazon's Rekognition, and Mel-Frequency Cepstral Coefficient (MFCC) features [Davis and Mermelstein, 1980]. We note that the emotion and MFCC features have never been used in prior work on dominance prediction. Based on bst(p, t, G), we develop a novel set of Dominance Rank features, inspired by the PageRank algorithm, on top of basic features. We thus have five types of short-term features, all applicable to short video segments.

The ground truth dominance labels in both datasets are provided for an entire game. Therefore, we need to predict whether a player is the most dominant in a game as a whole (or more dominant than another player in the game as a whole) rather than in a short time segment ∆t. For this, we associate a basic long-term feature vector blt(p,G) that aggregates the features for the short-time slices into features for the game as a whole using Fisher vector encodings [Perronnin et al., 2010] and histograms. A similar aggregation is also applied to the Dominance Rank features to get a vector ltf (p, G) of the long-term features for player p in game G.

We then develop predictive models based on each type of long-term feature and develop an ensemble late fusion approach that merges the five predictive models to make a final prediction. We investigate the importance of each type of feature in the ensemble predictor and show that Dominance Rank features play an important role. We re-emphasize that DR features build on top of basic features including ones proposed by others.

Finally, we also develop a Group Dominance Prediction (GDP) algorithm, which relies on the intuition that considering all players in the game at once is preferable to treating them independently. This naturally sets up a classification problem where each player's ltf feature vectors are fed into the classifier for training, together with the one-hot encoding for players (most dominant player in that game or not). Because games can have 5-8 players, we associate with each game G, and each possible subset of 5 players in that game, the concatenation of the feature vectors of those 5 players, along with an indication of which player was the most dominant. We then learn a classifier on the resulting data.

2 RelatedWork

Social scientists have studied dominance for years. [Dillard and Tusing, 2006] found that dominance is correlated with speaking rates and voice characteristics (frequency, amplitude, etc.). Visual cues like people looking at each other, body movements, gestures and facial expressions are also indicators of dominance in social interactions [Hall et al., 2005; Dunbar and Burgoon, 2005]. Moreover, [Dovidio and Ellyson, 1982] studied the relationship between dominance and the combination of looking-while-speaking and lookingwhile-listening periods. As a result, most work on predicting dominance use features based on these social science findings. Some early prediction papers use discrete features based on binary speaking variables (for a given time segment, does the person speak in it or not). These features include statistics such as total speaking length, total speaking turns, and total successful interruptions [Jayagopi et al., 2009; Aran and Gatica-Perez, 2010]. In addition, [Sanchez-Cortes et al., 2012] use prosodic features such as energy and pitch variation. We differ from these features by not converting continuous speaking variables into binary ones. Additionally, we differ from these efforts because we use Mel-Frequency Cepstral Coefficient (MFCC) [Davis and Mermelstein, 1980] features, which are a richer representation of audio features than prosodic features.

Other works extensively use visual features in the form of discrete variables. [Aran and Gatica-Perez, 2010] and [Jayagopi et al., 2009] use statistics on overall visual activity (binary variable - person either moves or not). [SanchezCortes et al., 2012] and [Beyan et al., 2018] analyze more fine-grained activity such as head and body movements, and gestures. In addition to these, a set of proposed methods use gaze-related features [Aran and Gatica-Perez, 2013; Okada et al., 2015; Okada et al., 2018].

Some past works attempt to capture dyadic or grouplevel information that might be relevant for the task. [Aran and Gatica-Perez, 2013] and [Okada et al., 2018] mine cooccurring events in the sequence of visual and audio features of individual players. [Sanchez-Cortes et al., 2012] use collective classification using speaking turns as weights on edges in the graph of the group.

We build upon these prior important works. The principal innovative class of representations that we introduce are Dominance Rank features that encode aspects of dynamic group interaction, building on basic audio-visual features. We validate our methods on two group-interaction datasets, and in ablation testing, we further show that Dominance Rank features turn out to be important for predicting dominance.

3 Dataset and Task Descriptions

Resistance dataset. Our team developed a Resistance dataset of videos of people participating in a variation of the roleplaying party game The Resistance. Each game has 5-8 players secretly divided into two teams: spies and resistance members (approximately 40% of players are spies). Spies have full information about other players' roles, whereas members of resistance do not know other players' roles. Games proceed in rounds (called "missions") involving a nomination and election of a mission leader, nomination and approval of a mission team, and the mission itself. Spies and resistance members have opposite incentives in the game: spies want to get elected on as many missions as possible in order to fail missions and thus earn points. Resistance members want to discover spies as early as possible and prevent spies from getting on mission teams. To achieve their goals, players on both teams need to be assertive and dominant, while spies also need to hide their true intentions. The data was collected in a variety of locations with different cultures (three locations in the USA, and one each in Israel, Zambia, and Singapore) and consisted of 33 games involving 233 players, each player appearing in exactly one game. Games typically last for 2-8 rounds. After every two rounds of the game, all players fill in a questionnaire, which asks them to rate other players' dominance in the past two rounds (we will refer to this period of time as "round" for simplicity). Therefore, every player is rated by every other player on an integer 1-5 scale (1 is not dominant at all, 5 is very dominant). We find the median score for each player and call it the ground truth perceived dominance score of the player in that round. Our dataset contains 79 rounds in all.

ELEA dataset. In addition to the Resistance dataset, we used the ELEA dataset developed by [Sanchez-Cortes et al., 2012]. This dataset contains videos of groups of people (3-4 persons in a group, 27 groups) participating in a winter survival task: the participants were given 12 items and asked to rank their importance for survival in the hypothetical scenario of a plane crash in a winter forest. Participants needed to have a discussion and come up with a consensus. Each video lasts 15 minutes. Videos are accompanied by survey results measuring participants' dominance: perceived dominance (PDom), ranked dominance (RDom).

Most Dominant Player (MDP). The MDP problem is to find the most dominant player in a given round. This is a binary classification task with label 1 if the player has the highest perceived dominance score among all the players in this round, and 0 otherwise. In our setting, more than one player in the group can have the highest dominance score. We therefore consider two instances of the problem: finding the most dominant players in all rounds (MDP-All), and finding the most dominant player in every distinct round (MDPDistinct). A round is distinct if there is a single player with the highest dominance score.

Pairwise Dominance Prediction (PDP). We also consider the more fine-grained problem of pairwise comparison. The PDP-All problem takes two players in a game as input and predicts which one has the higher dominance score. To pose this as a binary classification problem, we discard pairs with equal scores. The PDP-Distinct problem predicts which player in a pair is more dominant when the dominance scores of the players differ by 1 or more. We call such pairs of players distinct pairs.

4 DELF and GDP Algorithms

We have already provided a brief overview of our approach in Section 1. We first describe our short term traditional features and then our Dominance Rank features (both denoted further as stf ), followed by the extension of the short term features to the video as a whole.

4.1 Basic Short-termFeatures

Past work has shown that speech-related cues [Dillard and Tusing, 2006; Beyan et al., 2018] and gaze [Hall et al., 2005; Sanchez-Cortes et al., 2012] are closely related to perceived dominance of a person. We also use facial expressions and emotions as additional signals for visual dominance [Dunbar and Burgoon, 2005]. Our basic short-term features use audiovisual features from the frontal videos of players. While the use of these features is not novel, we note that emotion scores, facial action units, and MFCC features have never been used before for dominance prediction.

• Speaking probability st(pi) is an estimate of a probability that player pi is speaking during time interval t. This probability is estimated from the person's lip movement [Escalera et al., 2012].
• Gazing probability gt(pi, pj) is an estimate of the probability that player pi looks at player pj for every ∆t = 0.33 seconds [Ba and Odobez, 2011; Rayner, 2009].
• Facial Action Units scores (FAUs) capture the intensity of 17 action units in the given frame. These values are produced using OpenFace [Baltrusaitis et al., 2018].
• Emotion scores are the estimates of intensity of eight emotions and two facial traits (smile, open eyes) produced by Amazon Rekognition.
• Audio features are represented by Mel-Frequency Cepstral Coefficients, which are widely used in audio analysis [Davis and Mermelstein, 1980].

Dominance Rank Features

Previous research on dominance and leadership analysis shows that dyadic statistics are correlated with dominance ([Aran and Gatica-Perez, 2013; Okada et al., 2018; SanchezCortes et al., 2012]). We propose a family of short-term Dominance Rank (DR) features capturing the mutual interactions between players in the game. Suppose I(pi, pj) is a function that returns a value capturing the interaction between players pi and pj (we will show several such functions shortly). The short-term Dominance Rank Rdom(pi) of a player pi w.r.t.

function I in a given time period t is defined as:

where N is the number of players in the game, I(pi, pj) is an interaction function, and d ∈ [0, 1] is a damping factor. Damping factor d regulates the importance of the interaction function for the values of the Dominance Rank (the larger the d the more important role plays the interaction function). Although we note that Dominance Rank builds upon the idea of PageRank, unlike PageRank, Rdom is not one function, but a family of functions one for each possible interaction function I. Like PageRank, we set d = 0.85. We compute the Dominance Rank the same way as the PageRank: by building a system of N equations with N unknowns and iteratively solving it.

Interaction Functions. We define a family of interaction functions, each of which yields a different dominance rank function Rdom when used in Equation 1. We consider combinations of basic values defined on a slightly larger time period than basic features, representing interactions between players: S (speaking rate), G (gazing rate), LS (looking while speaking) and LL (looking while listening) defined as follows:

where k is the number of time slices of length ∆t, on which we define speaking and gazing probability, that fit into the time period (t1, t2) for the Dominance Rank. In our experiments we use time periods of 1 and 5 seconds for Dominance Rank features, thus k is 3 or 15. Based on these values, we define a set of interaction functions (Table 1) representing how interaction between players may be connected to distribution of dominance in the group, e.g., if less dominant players look at more dominant players more often than the other way around (in rows 1-2). To compare Dominance Rank (Equation 1) for players from different games, we normalize these values to be in [0, 1]. Table 1 lists some of the interaction functions we explored and the Pearson/Spearman Correlation Coefficients (r/ρ) of resulting Dominance Ranks with ground truth dominance scores. We recall that correlation coefficients lie in the [-1,+1] interval. We see that some of the Dominance Rank Functions such as those associated with interaction functions LL - LL and LS/LL (rows 3 and 11 respectively) demonstrate strong correlation with ground truth dominance scores.

4.2 Long-termFeatures

Since players in the game are instructed to score each other's dominance for only the round before the survey, to train models for our four classification tasks, we need to produce features representing whole rounds, which last 15 minutes on average. The features above, however, are extracted over a short-term period of time from 0.33 to 5 seconds. To aggregate these features we use two methods described below.

Fisher-Vector features. Fisher vector (FV) is a bag-ofwords model heavily used for object recognition in images [Perronnin et al., 2010]. Note that each round may have a different duration and hence the number of stf features can vary from round to round. Fisher Vectors aggregate the features of an arbitrarily long video into a fixed length encoding -we use 256-dimensional features for our experiments.

Histogram features. We compute a histogram feature for every stf feature (both bst features and normalized shortterm Dominance Rank features). For a player pi in a game round G and a short-term feature stf , we have a set {stf (pi, t1,G), . . . , stf (pi, tT ,G)} of all feature values for all short intervals over the round. We build a histogram of short-term features Vstf = 〈v1, v2, . . . , vb〉, where vl are frequencies of values stf (pi, tj,G) falling into the lth bin; b is the number of bins determined through cross-validation (on the training set alone).

4.3 Dominance Ensemble Late Fusion (DELF)

The best classifier for feature type i returns a score Si denoting the probability of a subject being the most dominant player in the corresponding round. DELF then fuses the scores S1, . . . , S5 by late fusion as

where ∑5i=1 αi = 1. The values of late fusion weights αi are obtained by grid-search and cross-validation on the training set alone. The best classifier for each of the five types of features is determined by exhaustive search through all possible combinations of classifiers.

4.4 Group Dominance Prediction (GDP)

We propose the Group Dominance Prediction (GDP) algorithm for solving MDP-All and MDP-Distinct. GDP's pseudo-code is shown as Algorithm 1 and also on Figure 2.

We reason that to determine the most dominant player in a game we need to compare players within that game to each other, therefore it should be beneficial to provide a classifier with features of all players in that game at once. But in the Resistance dataset, the numbers of players in games vary, which prevents us from building a single model with fixed feature length. GDP's goal is to develop a modified training set. The algorithm considers each game in turn and looks at all possible subsets G5 of 5 players in that game (the smallest possible number of players in any game). For each subset in G5, GDP considers the maximal ground truth dominance score (Step 8). It then generates a new feature vector by concatenating the long-term feature vectors of the five players (Step 9) and then assigning a label of 1 to the most dominant players in the subset and a label of 0 to the others (Step 10). Furthermore, GDP considers all permutations of players to augment the dataset (Steps 6-7). This creates a new training set with feature vectors 5 times as long as before. GDP then trains a classifier (multilabel for MDP-All, multiclass for MDP-Distinct).

At test time, GDP performs the same procedure (forming subsets of 5 players and all permutations) with the validation set. Once all the binary predictions are made, to obtain the final probability of a player being most dominant in the game round, we average the predictions for this player for all groups and permutations where this player is present.

5 Experimental Results with Resistance Data

Setup. We split the Resistance dataset into 10 folds by games. As each player appears in only one game, we always make predictions about the dominance of players in games that we have not seen before. Our classifier suite for binary prediction tasks consists of the 5 classifiers: k-Nearest Neighbors, Logistic Regression, Gaussian Naive Bayes, Linear SVM, and Random Forest. The tables below report the best results among these classifiers. Since our Resistance dataset is inherently imbalanced, we report the mean AUC over 10 folds and use it to compare models. But we also report False Positive rate (FPR) and Accuracy (Acc.) as reported in past works [Beyan et al., 2018; Sanchez-Cortes et al., 2012; Okada et al., 2015; Okada et al., 2018; Aran and GaticaPerez, 2013].

5.1 Binary Prediction with DELF

Table 2 shows the result of applying DELF and single ltf - classifiers to the four problems. We compare our models with two baselines adapted from the recent paper by Beyan et al. [Beyan et al., 2018]: one model uses speaking features such as total number of speaking turns or number of times a player gets interrupted, the other model combines speaking features with gazing features such as number of times a player looks at other players.

DELF produces the best AUCs in all four tasks outperforming both baselines and our single-ltf classifiers. For each task, a single-ltf classifier (Dominance Rank or speakingbased feature) outperforms the baselines. In most cases, the improvement in AUC comes with reduced FPR and better accuracy than the baselines. We see that Dominance Rank features prove to be more useful in the MDP task, while speaking-based features produce the highest AUCs on PDP among single-ltf features. We believe this happens because speaking-based features capture individual behavior of the player thus making it easier to compare two players, while Dominance Rank features capture the overall dynamics of the interaction of a player with all other players, which is useful for the most dominant player detection but introduces noise for pairwise comparison. Additionally, we found that features exclusively based on gaze information produce poor results (not reported in the Table 2), which holds both for our features and baseline features.

We further note that "nice" instances of problems (MDPDistinct and PDP-Distinct) are easier and get higher results, because the difference in dominance between players is more prominent.

Ablation study. To assess the importance of each group of features used in DELF, we exclude features one at a time and perform another late fusion on the reduced group of features We see from Table 3 that DR features prove to be important for identifying the most dominant player - both for MDPAll and MDP-Distinct. For PDP-All and PDP-Distinct most value is provided by speaking-based features and MFCC respectively.

Predictions of our models depend on the size of the game portion considered and what part of the game is considered. Figure 3 shows how the LS/LL Dominance Rank feature (best performing feature in Table 2) performs on MDP-All task when we process only a portion of each video (varied from 20% to 100%). We found that considering only 20% of the video drops the predictive performance of our models up to 0.2. Performance grows with increased video length reaching the highest result for the entire video. For the same length, however, it is usually advantageous to consider parts closer to the end of the game. The last 20% of the video sometimes can yield performance very close to the classifiers trained on the entire video. We attribute this finding to the fact that ground truth labels used in our work are based on players' assessment of each other, which is collected after every two rounds of the game, and people tend to remember recent events better. The analysis shows, however, that for the best performance it's important to consider the entire video.

Interaction functions. In addition to Dominance Rank features w.r.t. the interaction function LS/LL, we examined two more interaction functions: LL(pj , pi) - LL(pi, pj ) and LS(pi, pj) - LS(pj, pi). These functions show relatively high correlation with ground truth dominance scores (Table 1). For MDP-All these features yield the AUC of 0.755 and 0.748 respectively, showing the results close to the best single-ltf classifier in Table 2. For MDP-Distinct the AUCs are 0.795 and 0.847 respectively, which is higher than the corresponding baselines and on-par with the best single-ltf classifiers.

5.2 GDP AlgorithmPerformance

We tested GDP algorithm on the Resistance dataset. We used two classifiers: Multilayer Perceptron (MLP) with two layers, and Random Forest (RF) with 50 estimators. As shown in Table 4, GDP outperforms all the baselines as well as the various strong settings of DELF.

6 ELEACorpus Experiments

We conducted further tests on the ELEA dataset [SanchezCortes et al., 2012] which is a widely used benchmark for modeling and detecting personal traits such as leadership and dominance. We use speaking and gazing labels provided with the dataset to produce Dominance Rank features. Every participant in the dataset has two dominance scores: perceived dominance (PDom) and ranked dominance (Rdom). We followed two protocols: (1) as in [Okada et al., 2018; Aran and Gatica-Perez, 2013; Okada et al., 2015] we assign every participant a binary label by thresholding her dominance score by the median value, and (2) as in [Sanchez- Cortes et al., 2012] we solve the MDP-All task, i.e., finding the most dominant participant in every group. As in these works we use the leave-one-game-out method for training and testing classifiers. Table 5 shows that our proposed Dominance Rank feature outperforms strong baselines in existing work.

We used the average dominance scores assigned to participants by the independent viewers not participating in the task as human scores. In Table 5 we can see that our proposed features outperform humans on the task of detecting participants who are more dominant than others. Humans, however, are better at detecting the most dominant participant in a group, although our model achieves comparable accuracy.

7 Conclusion

We study two major problems: predicting the most dominant person in a group setting, as well as the more dominant of a pair of people. We develop a novel family of Dominance Rank features and develop two algorithms for these problems. The DELF algorithm uses past features (plus facial action unit, emotion, and MFCC features not previously used in dominance prediction), as well as dominance ranks combined with a late fusion approach and beats out past work in predictive accuracy - an ablation study additionally shows the Dominance Rank features to be the most important ones. The GDP algorithm proposes a way to expand and augment the dataset while retaining the group information. It beats out both past work and DELF on two tasks. But we note that both DELF and GDP use many well-known features from the past literature to achieve these high AUCs.

Acknowledgements

This work was funded by ARO Grant W911NF1610342. Role of Authors. Authors Burgoon and Dunbar designed the Resistance-style game, designed how the game would be run face to face, and collected the Resistance data. The remaining authors designed the feature extraction and machine learning algorithms and software, and designed/ran all experiments.