Hierarchical Multi-channel Hidden Semi Markov Models

Pradeep Natarajan, Ramakant Nevatia

Many interesting human actions involve multiple interacting agents and also have typical durations. Further, there is an inherent hierarchical organization of these activities. In order to model these we introduce a new family of hidden Markov models (HMMs) that provide compositional state representations in both space and time and also a recursive hierarchical structure for inference at higher levels of abstraction. In particular, we focus on two possible 2-layer structures - the Hierarchical-Semi Parallel Hidden Markov Model (HSPaHMM) and the Hierarchical Parallel Hidden Semi-Markov Model (HPaHSMM). The lower layer of HSPaHMM consists of multiple HMMs for each agent while the top layer consists of a single HSMM. HPaHSMM on the other hand has multiple HSMMs at the lower layer and a Markov chain at the top layer. We present efficient learning and decoding algorithms for these models and then demonstrate them first on synthetic time series data and then in an application for sign language recognition.