{"type":"ScholarlyArticle","title":"Structured Inference for Recurrent Hidden Semi-markov Model","authors":[{"name":"Hao Liu","slug":"hao-liu","given":"Hao","family":"Liu","affiliation":"SMILE Lab , University of Electronic Science and Technology of China , Chengdu , China; Yingcai Honors College , University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Lirong He","slug":"lirong-he","given":"Lirong","family":"He","affiliation":"SMILE Lab , University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Haoli Bai","slug":"haoli-bai","given":"Haoli","family":"Bai","affiliation":"Dept. of Computer Science and Engineering , The Chinese University of Hong Kong , Hong Kong"},{"name":"Bo Dai","slug":"bo-dai","given":"Bo","family":"Dai","affiliation":"College of Computing , Georgia Institute of Technology , Atlanta , GA USA"},{"name":"Kun Bai","slug":"kun-bai","given":"Kun","family":"Bai","affiliation":"Mobile Internet Group , Tencent Inc. , Shenzhen , China"},{"name":"Zenglin Xu","slug":"zenglin-xu","given":"Zenglin","family":"Xu","affiliation":"SMILE Lab , University of Electronic Science and Technology of China , Chengdu , China"}],"abstract":"Segmentation and labeling for high dimensional time series is an important yet challenging task in a number of applications, such as behavior understanding and medical diagnosis. Recent advances to model the nonlinear dynamics in such time series data, has suggested to involve recurrent neural networks into  Hidden Markov Models. However, this involvement has caused the inference procedure much more complicated, often leading to intractable inference, especially for the discrete variables of segmentation and labeling. To achieve both flexibility and tractability in modeling nonlinear dynamics of discrete variables, we present a structured and stochastic sequential neural network (SSNN), which composes with a generative network and an inference network. In detail, the generative network aims to not only capture the long-term dependencies but also model the uncertainty of the segmentation labels via semi-Markov models. More importantly, for efficient and accurate inference, the proposed bi-directional inference network reparameterizes the categorical segmentation with the Gumbel-Softmax approximation and resorts to the Stochastic Gradient Variational Bayes. We evaluate the proposed model in a number of tasks, including speech modeling, automatic segmentation and labeling in behavior understanding, and sequential multi-objects recognition. Experimental results have demonstrated that our proposed model can achieve significant improvement over the state-of-the-art methods.","doi":"10.24963/ijcai.2018/339","url":"https://ijcai.org/proceedings/2018/339","pdf":"https://ijcai.org/proceedings/2018/0339.pdf","markdown":"https://ijcai.org/proceedings/2018/339.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/339","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":339,"pages":{"first":2447,"last":2453},"keywords":["Machine Learning: Neural Networks","Machine Learning: Learning Graphical Models","Machine Learning: Learning Generative Models","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Ammar et al., 2014] Waleed Ammar, Chris Dyer, and Noah A Smith. Conditional random field autoencoders for unsupervised structured prediction. In Advances in Neural Information Pro- cessing Systems, pages 3311-3319, 2014.","title":"Conditional random field autoencoders for unsupervised structured prediction","authors":"Waleed Ammar, Chris Dyer, Noah A Smith","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Archer et al., 2015] Evan Archer, Il Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. Black box variational in- ference for state space models. arXiv preprint arXiv:1511.07367, 2015.","title":"Black box variational inference for state space models","authors":"Evan Archer, Memming Il, Lars Park, John Buesing, Liam Cunningham, Paninski","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bayer and Osendorfer, 2014] Justin Bayer and Christian Osendor- fer. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610, 2014.","title":"Learning stochastic recurrent networks","authors":"Osendorfer Bayer","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiappa and others, 2014] Silvia Chiappa et al. Explicit-duration markov switching models. Foundations and Trends R in Machine Learning, 7(6):803-886, 2014.","title":"Explicit-duration markov switching models","authors":"Silvia Chiappa","year":"2014","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2015] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recur- rent latent variable model for sequential data. In Advances in neural information processing systems, pages 2980-2988, 2015.","title":"A recurrent latent variable model for sequential data","authors":"Chung","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chung et al., 2016] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704, 2016.","title":"Hierarchical multiscale recurrent neural networks","authors":"Chung","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dai et al., 2017a] Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. Stochastic generative hashing. arXiv preprint arXiv:1701.02815, 2017.","title":"Stochastic generative hashing","authors":"Dai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2017b] Hanjun Dai, Bo Dai, Yan-Ming Zhang, Shuang Li, and Le Song. Recurrent hidden semi-markov model. ICLR, 2017.","title":"Recurrent hidden semi-markov model","authors":"Dai","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fraccaro et al., 2016] Marco Fraccaro, Søren Kaae Sønderby, Ul- rich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems, pages 2199-2207, 2016.","title":"Sequential neural models with stochastic layers","authors":"Marco Fraccaro, Kaae Søren, Ulrich Sønderby, Ole Paquet, Winther","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goyal et al., 2017] Kartik Goyal, Graham Neubig, Chris Dyer, and Taylor Bergkirkpatrick. A continuous relaxation of beam search for end-to-end training of neural sequence models. arXiv preprint arXiv:1708.00111, 2017.","title":"A continuous relaxation of beam search for end-to-end training of neural sequence models","authors":"Kartik Goyal, Graham Neubig, Chris Dyer, Taylor Bergkirkpatrick","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jang et al., 2017] Eric Jang, Shixiang Gu, and Ben Poole. Categor- ical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson and Willsky, 2013] Matthew J Johnson and Alan S Will- sky. Bayesian nonparametric hidden semi-markov models. Jour- nal of Machine Learning Research, 14(Feb):673-701, 2013.","title":"Bayesian nonparametric hidden semi-markov models","authors":"J Matthew, Alan S Johnson, Willsky","year":"2013","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson and Willsky, 2014] Matthew Johnson and Alan Willsky. Stochastic variational inference for bayesian time series models. In International Conference on Machine Learning, pages 1854- 1862, 2014.","title":"Stochastic variational inference for bayesian time series models","authors":"Matthew Johnson, Alan Willsky","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Johnson et al., 2016] Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured represen- tations and fast inference. In Advances in Neural Information Processing Systems, pages 2946-2954, 2016.","title":"Composing graphical models with neural networks for structured representations and fast inference","authors":"Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, Sandeep R Datta","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kain et al., 2013] Jamey Kain, Chris Stokes, Quentin Gaudry, Xiangzhi Song, James Foley, Rachel Wilson, and Benjamin De Bivort. Leg-tracking and automated behavioural classifica- tion in drosophila. Nature communications, 4:1910, 2013.","title":"Leg-tracking and automated behavioural classification in drosophila","authors":"Jamey Kain, Chris Stokes, Quentin Gaudry, Xiangzhi Song, James Foley, Rachel Wilson, Benjamin De Bivort","year":"2013","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Karl et al., 2016] Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes fil- ters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432, 2016.","title":"Deep variational bayes filters: Unsupervised learning of state space models from raw data","authors":"Karl","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Khan et al., 2015] Mohammad E Khan, Pierre Baqué, Franc ¸ois Fleuret, and Pascal Fua. Kullback-leibler proximal variational in- ference. In Advances in Neural Information Processing Systems, pages 3402-3410, 2015.","title":"Kullback-leibler proximal variational inference","authors":"Pierre Mohammad E Khan, Franc Baqué, Pascal ¸ois Fleuret, Fua","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krishnan et al., 2015] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint arXiv:1511.05121, 2015.","title":"Deep kalman filters","authors":"Uri Rahul G Krishnan, David Shalit, Sontag","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Krishnan et al., 2016] Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. arXiv preprint arXiv:1609.09869, 2016.","title":"Structured inference networks for nonlinear state space models","authors":"Uri Rahul G Krishnan, David Shalit, Sontag","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2017] Hao Liu, Haoli Bai, Lirong He, and Zenglin Xu. Stochastic sequential neural networks with structured inference. arXiv preprint arXiv:1705.08695, 2017.","title":"Stochastic sequential neural networks with structured inference","authors":"Hao Liu, Haoli Bai, Lirong He, Zenglin Xu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Maddison et al., 2016] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continu- ous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Andriy Chris J Maddison, Yee Whye Mnih, Teh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mnih and Rezende, 2016] Andriy Mnih and Danilo J Rezende. Variational inference for monte carlo objectives. arXiv preprint arXiv:1602.06725, 2016.","title":"Variational inference for monte carlo objectives","authors":"Andriy Mnih, Danilo J Rezende","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Murphy, 2002] Kevin P Murphy. Hidden semi-markov models (hsmms). unpublished notes, 2, 2002.","title":"Hidden semi-markov models (hsmms)","authors":"Kevin P Murphy","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Prahallad et al., 2013] Kishore Prahallad, Anandaswarup Vada- palli, Naresh Elluru, G Mantena, B Pulugundla, P Bhaskararao, HA Murthy, S King, V Karaiskos, and AW Black. The bliz- zard challenge 2013-indian language task. In Blizzard Challenge Workshop, volume 2013, 2013.","title":"The blizzard challenge 2013-indian language task","authors":"Kishore Prahallad, Anandaswarup Vadapalli, Naresh Elluru, G Mantena, B Pulugundla, P Bhaskararao, S Ha Murthy, V King, Karaiskos, Black","year":"2013","venue":"Blizzard Challenge Workshop","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Reyes-Ortiz et al., 2016] Jorge-L Reyes-Ortiz, Luca Oneto, Albert Sama, Xavier Parra, and Davide Anguita. Transition-aware hu- man activity recognition using smartphones. Neurocomputing, 171:754-767, 2016.","title":"Transition-aware human activity recognition using smartphones","authors":"Jorge-L Reyes-Ortiz, Luca Oneto, Albert Sama, Xavier Parra, Davide Anguita","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rolfe, 2016] Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:1609.02200, 2016.","title":"Discrete variational autoencoders","authors":"Jason Tyler, Rolfe","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rumelhart et al., 1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Cognitive modeling, 5(3):1, 1988.","title":"Learning representations by backpropagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1988","venue":"Cognitive modeling","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Springer et al., 2016] David B Springer, Lionel Tarassenko, and Gari D Clifford. Logistic regression-hsmm-based heart sound segmentation. IEEE Transactions on Biomedical Engineering, 63(4):822-832, 2016.","title":"Logistic regression-hsmm-based heart sound segmentation","authors":"Lionel David B Springer, Gari D Tarassenko, Clifford","year":"2016","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104- 3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Van Merriënboer et al., 2015] Bart Van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde- Farley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning. arXiv preprint arXiv:1506.00619, 2015.","title":"Blocks and fuel: Frameworks for deep learning","authors":"Van Merriënboer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}