{"type":"ScholarlyArticle","title":"Continuous Body and Hand Gesture Recognition for Natural Human-Computer Interaction: Extended Abstract","authors":[{"name":"Yale Song","slug":"yale-song","given":"Yale","family":"Song","affiliation":"MIT CSAIL."},{"name":"Randall Davis","slug":"randall-davis","given":"Randall","family":"Davis","affiliation":"Massachusetts Instutute of Technology; MIT CSAIL."}],"abstract":"We present a new approach to gesture recognition that tracks body and hands simultaneously and recognizes gestures continuously from an unsegmented and unbounded input stream. Our system estimates 3D coordinates of upper body joints and classifies the appearance of hands into a set of canonical shapes. A novel multi-layered filtering technique with a temporal sliding window is developed to enable online sequence labeling and segmentation. Experimental results on the NATOPS dataset show the effectiveness of the approach. We also report on our recent work on multimodal gesture recognition and deep-hierarchical sequence representation learning that achieve the state-of-the-art performances on several real-world datasets.","doi":null,"url":"https://ijcai.org/proceedings/2015/598","pdf":"https://ijcai.org/proceedings/2015/0598.pdf","markdown":"https://ijcai.org/proceedings/2015/598.md","bibtex":"https://ijcai.org/proceedings/2015/bibtex/598","volume":{"slug":"2015","title":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2015","url":"https://ijcai.org/proceedings/2015/","year":2015,"location":"Buenos Aires, Argentina","dates":"25–31 July 2015","editors":"Qiang Yang and Michael Wooldridge","isbn":"978-1-57735-738-4","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Journal Track","track":"Journal Track","number":598,"pages":{"first":4212,"last":4215},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Bengio, 2009] Yoshua Bengio. Learning deep architectures for ai. FTML, 2(1), 2009.","title":"Learning deep architectures for ai","authors":"Yoshua Bengio","year":"2009","venue":"FTML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bobick and Davis, 2001] Aaron F. Bobick and James W. Davis. The recognition of human movement using temporal templates. PAMI, 23(3), 2001.","title":"The recognition of human movement using temporal templates","authors":"Aaron F Bobick, James W Davis","year":"2001","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bousmalis et al., 2011] Konstantinos Bousmalis, L Morency, and Maja Pantic. Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In FG, 2011.","title":"Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition","authors":"Konstantinos Bousmalis, Maja Morency, Pantic","year":"2011","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIB- SVM: a library for support vector machines. TIST, 2(3):27, 2011.","title":"LIB-SVM: a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"TIST","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dalal and Triggs, 2005] Navneet Dalal and Bill Triggs. His- tograms of oriented gradients for human detection. In CVPR, 2005.","title":"Histograms of oriented gradients for human detection","authors":"Triggs Dalal, Navneet Dalal, Bill Triggs","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Denavit and Hartenbert, 1955] J. Denavit and R. S Hartenbert. A kinematic notation for lower-pair mechanisms based on matrices. ASME J. Appl. Mechan.,, 22, 1955.","title":"A kinematic notation for lower-pair mechanisms based on matrices","authors":"J Denavit, R S Hartenbert","year":"1955","venue":"ASME J. Appl. Mechan","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hardoon et al., 2004] David Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. NECO, 16(12), 2004.","title":"Canonical correlation analysis: An overview with application to learning methods","authors":"David Hardoon, Sandor Szedmak, John Shawe-Taylor","year":"2004","venue":"NECO","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2011] Lixing Huang, Louis-Philippe Morency, and Jonathan Gratch. Virtual rapport 2.0. In IVA, 2011.","title":"Virtual rapport 2.0","authors":"Lixing Huang, Louis-Philippe Morency, Jonathan Gratch","year":"2011","venue":"IVA","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Isard and Blake, 1998] Michael Isard and Andrew Blake. Conden- sationconditional density propagation for visual tracking. IJCV, 29(1), 1998.","title":"Condensationconditional density propagation for visual tracking","authors":"Michael Isard, Andrew Blake","year":"1998","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2005] Kyungnam Kim, Thanarat H Chalidabhongse, David Harwood, and Larry Davis. Real-time foreground- background segmentation using codebook model. Real-time imaging, 11(3), 2005.","title":"Real-time foregroundbackground segmentation using codebook model","authors":"Kyungnam Kim, H Thanarat, David Chalidabhongse, Larry Harwood, Davis","year":"2005","venue":"Real-time imaging","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Morency et al., 2007] L Morency, Ariadna Quattoni, and Trevor Darrell. Latent-dynamic discriminative models for continuous gesture recognition. In CVPR, 2007.","title":"Latent-dynamic discriminative models for continuous gesture recognition","authors":"Ariadna Morency, Trevor Quattoni, Darrell","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Murphy, 2002] Kevin Murphy. Dynamic bayesian networks: Rep- resentation, inference and learning. PhD thesis, Ph. D. disserta- tion, UC Berkeley, 2002.","title":"Dynamic bayesian networks: Representation, inference and learning","authors":"Kevin Murphy, Murphy","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Quattoni et al., 2007] Ariadna Quattoni, Sybor Wang, Louis- Philippe Morency, Michael Collins, and Trevor Darrell. Hidden conditional random fields. PAMI, (10), 2007.","title":"Hidden conditional random fields","authors":"Ariadna Quattoni, Sybor Wang, Louis-Philippe Morency, Michael Collins, Trevor Darrell","year":"2007","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shotton et al., 2011] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kip- man, and Andrew Blake. Real-time human pose recognition in parts from single depth images. CVPR, 2011.","title":"Real-time human pose recognition in parts from single depth images","authors":"Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2011a] Yale Song, David Demirdjian, and Randall Davis. Multi-signal gesture recognition using temporal smooth- ing hidden conditional random fields. In FG, 2011.","title":"Multi-signal gesture recognition using temporal smoothing hidden conditional random fields","authors":"Song","year":"2011","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Song et al., 2011b] Yale Song, David Demirdjian, and Randall Davis. Tracking body and hands for gesture recognition: NATOPS aircraft handling signals database. In FG, 2011.","title":"Tracking body and hands for gesture recognition: NATOPS aircraft handling signals database","authors":"Song","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Song et al., 2012a] Yale Song, David Demirdjian, and Randall Davis. Continuous body and hand gesture recognition for nat- ural human-computer interaction. TiiS, 2(1):5, 2012.","title":"Continuous body and hand gesture recognition for natural human-computer interaction","authors":"Song","year":"2012","venue":"TiiS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2012b] Yale Song, Louis-Philippe Morency, and Ran- dall Davis. Multi-view latent variable discriminative models for action recognition. In CVPR, 2012.","title":"Multi-view latent variable discriminative models for action recognition","authors":"Song","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2012c] Yale Song, Louis-Philippe Morency, and Ran- dall Davis. Multimodal human behavior analysis: learning cor- relation and interaction across modalities. In ICMI, 2012.","title":"Multimodal human behavior analysis: learning correlation and interaction across modalities","authors":"Song","year":"2012","venue":"ICMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2013] Yale Song, Louis-Philippe Morency, and Ran- dall Davis. Action recognition by hierarchical sequence summa- rization. In CVPR, 2013.","title":"Action recognition by hierarchical sequence summarization","authors":"Song","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}