{"type":"ScholarlyArticle","title":"Unsupervised Learning of Deep Feature Representation for Clustering Egocentric Actions","authors":[{"name":"Bharat Lal Bhatnagar","slug":"bharat-lal-bhatnagar","given":"Bharat Lal","family":"Bhatnagar","affiliation":"* International Institute of Information Technology , Hyderabad"},{"name":"Suriya Singh","slug":"suriya-singh","given":"Suriya","family":"Singh","affiliation":"* International Institute of Information Technology , Hyderabad"},{"name":"Chetan Arora","slug":"chetan-arora","given":"Chetan","family":"Arora","affiliation":"Indraprastha Institute of Information Technology , Delhi +"},{"name":"C.V. Jawahar","slug":"c-v-jawahar","given":"C.V.","family":"Jawahar","affiliation":"* International Institute of Information Technology , Hyderabad"}],"abstract":"Popularity of wearable cameras in life logging, law enforcement, assistive vision and other similar applications is leading to explosion in generation of egocentric video content. First person action recognition is an important aspect of automatic analysis of such videos. Annotating such videos is hard, not only because of obvious scalability constraints, but also because of privacy issues often associated with egocentric videos. This motivates the use of unsupervised methods for egocentric video analysis. In this work, we propose a robust and generic unsupervised approach for first person action clustering. Unlike the contemporary approaches, our technique is neither limited to any particular class of actions nor requires priors such as pre-training, fine-tuning, etc. We learn time sequenced visual and flow features from an array of weak feature extractors based on convolutional and LSTM autoencoder networks. We demonstrate that clustering of such features leads to the discovery of semantically meaningful actions present in the video. We validate our approach on four disparate public egocentric actions datasets amounting to approximately 50 hours of videos. We show that our approach surpasses the supervised state of the art accuracies without using the action labels.","doi":"10.24963/ijcai.2017/200","url":"https://ijcai.org/proceedings/2017/200","pdf":"https://ijcai.org/proceedings/2017/0200.pdf","markdown":"https://ijcai.org/proceedings/2017/200.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/200","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":200,"pages":{"first":1447,"last":1453},"keywords":["Machine Learning: Unsupervised Learning","Machine Learning: Ensemble Methods","Machine Learning: Feature Selection/Construction","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Erhan et al., 2010] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In JMLR, 2010.","title":"Why does unsupervised pre-training help deep learning?","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pierre-Antoine Courville, Pascal Manzagol, Vincent","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fathi et al., 2011a] Alireza Fathi, Ali Farhadi, and James M Rehg. Understanding egocentric activities. In ICCV, 2011. 1, 2","title":"Understanding egocentric activities","authors":"Fathi","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fathi et al., 2011b] Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocen- tric activities. In CVPR, 2011. 4, 5","title":"Learning to recognize objects in egocentric activities","authors":"Fathi","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fathi et al., 2012] Alireza Fathi, Yin Li, and James M Rehg. Learning to recognize daily actions using gaze. In ECCV, 2012.","title":"Learning to recognize daily actions using gaze","authors":"Alireza Fathi, Yin Li, James M Rehg","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jones and Shao, 2014] Simon Jones and Ling Shao. Unsu- pervised spectral dual assignment clustering of human ac- tions in context. In CVPR, 2014. 2","title":"Unsupervised spectral dual assignment clustering of human actions in context","authors":"Shao Jones, Simon Jones, Ling Shao","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 4","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kitani et al., 2011] Kris Makoto Kitani, Takahiro Okabe, Yoichi Sato, and Akihiro Sugimoto. Fast unsupervised ego-action learning for first-person sports videos. In CVPR, 2011. 1, 2, 3, 5","title":"Fast unsupervised ego-action learning for first-person sports videos","authors":"Kris Makoto Kitani, Takahiro Okabe, Yoichi Sato, Akihiro Sugimoto","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kuhn, 1955] Harold W. Kuhn. The hungarian method for the assignment problem. In Naval Research Logistics Quarterly, 1955.","title":"The hungarian method for the assignment problem","authors":"Harold W Kuhn","year":"1955","venue":"Naval Research Logistics Quarterly","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2012] Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In CVPR, 2012.","title":"Discovering important people and objects for egocentric video summarization","authors":"Jae Yong, Joydeep Lee, Kristen Ghosh, Grauman","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2015] Yin Li, Zhefan Ye, and James M. Rehg. Delving into egocentric actions. In CVPR, 2015. 2, 3","title":"Delving into egocentric actions","authors":"Yin Li, Zhefan Ye, James M Rehg","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu and Grauman, 2013] Zheng Lu and Kristen Grauman. Story-driven summarization for egocentric video. In CVPR, 2013. 2, 3","title":"Story-driven summarization for egocentric video","authors":"Zheng Lu, Kristen Grauman","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2016] Minghuang Ma, Haoqi Fan, and Kris M. Kitani. Going deeper into first-person activity recognition. In CVPR, 2016. 1, 2, 3","title":"Going deeper into first-person activity recognition","authors":"Ma","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Matsuo et al., 2014] K. Matsuo, K. Yamada, S. Ueno, and S. Naito. An attention-based activity recognition for ego- centric video. In CVPRW, 2014. 1","title":"An attention-based activity recognition for egocentric video","authors":"K Matsuo, K Yamada, S Ueno, S Naito","year":"2014","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Niebles et al., 2008] Juan Carlos Niebles, Hongcheng Wang, and Li Fei-Fei. Unsupervised learning of human action categories using spatial-temporal words. In IJCV, 2008. 2","title":"Unsupervised learning of human action categories using spatial-temporal words","authors":"Niebles","year":"2008","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ogaki et al., 2012] Keisuke Ogaki, Kris Makoto Kitani, Yusuke Sugano, and Yoichi Sato. Coupling eye-motion and ego-motion features for first-person activity recogni- tion. In CVPRW, 2012.","title":"Coupling eye-motion and ego-motion features for first-person activity recognition","authors":"Keisuke Ogaki, Kris Makoto Kitani, Yusuke Sugano, Yoichi Sato","year":"2012","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pirsiavash and Ramanan, 2012] Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. In CVPR, 2012. 1, 2, 3, 4","title":"Detecting activities of daily living in first-person camera views","authors":"Ramanan Pirsiavash, Hamed Pirsiavash, Deva Ramanan","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Poleg et al., 2014] Yair Poleg, Chetan Arora, and Shmuel Peleg. Temporal segmentation of egocentric videos. In CVPR, 2014. 1, 2, 4, 6 [","title":"Temporal segmentation of egocentric videos","authors":"Yair Poleg, Chetan Arora, Shmuel Peleg","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Poleg et al., 2015] Yair Poleg, Ariel Ephrat, Shmuel Peleg, and Chetan Arora. Compact cnn for indexing egocentric videos. In WACV, 2015. 1, 2, 4, 5, 6","title":"Compact cnn for indexing egocentric videos","authors":"Yair Poleg, Ariel Ephrat, Shmuel Peleg, Chetan Arora","year":"2015","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ryoo and Matthies, 2013] Michael S Ryoo and Larry Matthies. First-person activity recognition: What are they doing to me? In CVPR, 2013.","title":"First-person activity recognition: What are they doing to me?","authors":"S Michael, Larry Ryoo, Matthies","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ryoo et al., 2015] Michael S Ryoo, B Rothrock, and Larry Matthies. Pooled motion features for first-person videos. In CVPR, 2015.","title":"Pooled motion features for first-person videos","authors":"B Michael S Ryoo, Larry Rothrock, Matthies","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Singh et al., 2016a] Suriya Singh, Chetan Arora, and C. V. Jawahar. Trajectory aligned features for first person action recognition. In Pattern Recognition, 2016. 1, 2, 3, 4","title":"Trajectory aligned features for first person action recognition","authors":"Singh","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Singh et al., 2016b] Suriya Singh, Chetan Arora, and C V, Jawahar. First person action recognition using deep learned descriptors. In CVPR, 2016. 1, 2, 3, 4, 6","title":"First person action recognition using deep learned descriptors","authors":"Singh","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Spriggs et al., 2009] Ekaterina H Spriggs, Fernando De La Torre, and Martial Hebert. Temporal segmentation and activity classification from first-person sensing. In CVPRW, 2009.","title":"Temporal segmentation and activity classification from first-person sensing","authors":"Ekaterina H Spriggs, Fernando De, La Torre, Martial Hebert","year":"2009","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srivastava et al., 2015] Nitish Srivastava, Elman Mansi- mov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using lstms. In JMLR, 2015. 2, 4","title":"Unsupervised learning of video representations using lstms","authors":"Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stein and McKenna, 2013] Sebastian Stein and Stephen J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activi- ties. In ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2013.","title":"Combining embedded accelerometers with computer vision for recognizing food preparation activities","authors":"Sebastian Stein, Stephen J Mckenna","year":"2013","venue":"ACM International Joint Conference on Pervasive and Ubiquitous Computing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Stein and McKenna, 2016] Sebastian Stein and Stephen J. McKenna. Recognising complex activities with his- tograms of relative tracklets. Computer Vision and Image Understanding, Elsevier, 154:82-93, 2016.","title":"Recognising complex activities with histograms of relative tracklets","authors":"Mckenna Stein, Sebastian Stein, Stephen J Mckenna","year":"2016","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Rmsprop: Divide the gradient by a running aver- age of its recent magnitude. In COURSERA: Neural Net- works for Machine Learning.Technical report, 2012.","title":"Rmsprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. In JMLR, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Schmid, 2013] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, 2013. 2, 3","title":"Action recognition with improved trajectories","authors":"Wang, Schmid ; ] Heng, Wang, Cordelia Schmid","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2011] Heng Wang, Alexander Klaser, Cordelia Schmid, and Cheng-Lin Liu. Action recognition by dense trajectories. In CVPR, 2011. 2, 3","title":"Action recognition by dense trajectories","authors":"Heng Wang, Alexander Klaser, Cordelia Schmid, Cheng-Lin Liu","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2015] Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, and Baining Guo. Unsu- pervised extraction of video highlights via robust recurrent auto-encoders. In ICCV, 2015.","title":"Unsupervised extraction of video highlights via robust recurrent auto-encoders","authors":"Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, Baining Guo","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2016] Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016. 5","title":"Joint unsupervised learning of deep representations and image clusters","authors":"Jianwei Yang, Devi Parikh, Dhruv Batra","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}