{"type":"ScholarlyArticle","title":"Convolutional Memory Blocks for Depth Data Representation Learning","authors":[{"name":"Keze Wang","slug":"keze-wang","given":"Keze","family":"Wang","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China; The Hong Kong Polytechnic University"},{"name":"Liang Lin","slug":"liang-lin","given":"Liang","family":"Lin","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China; Sensetime Group Limited"},{"name":"Chuangjie Ren","slug":"chuangjie-ren","given":"Chuangjie","family":"Ren","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China"},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"Sensetime Group Limited"},{"name":"Wenxiu Sun","slug":"wenxiu-sun","given":"Wenxiu","family":"Sun","affiliation":"Sensetime Group Limited"}],"abstract":"Compared to natural RGB images, data captured by 3D / depth sensors (e.g., Microsoft Kinect) have different properties, e.g., less discriminable in appearance due to lacking color / texture information. Applying convolutional neural networks (CNNs) on these depth data would lead to unsatisfying learning efficiency, i.e., requiring large amounts of annotated training data for convergence. To address this issue, this paper proposes a novel memory network module, called Convolutional Memory Block (CMB), which empowers CNNs with the memory mechanism on handling depth data. Different from the existing memory networks that store long / short term dependency from sequential data, our proposed CMB focuses on modeling the representative dependency (correlation) among non-sequential samples. Specifically, our CMB consists of one internal memory (i.e., a set of feature maps) and three specific controllers, which enable a powerful yet efficient memory manipulation mechanism. In this way, the internal memory, being implicitly aggregated from all previous inputted samples, can learn to store and utilize representative features among the samples. Furthermore, we employ our CMB to develop a concise framework for predicting articulated pose from still depth images. Comprehensive evaluations on three public benchmarks demonstrate significant superiority (about 6%) of our framework over all the compared methods. More importantly, thanks to the enhanced learning efficiency, our framework can still achieve satisfying results using 50% less training data.","doi":"10.24963/ijcai.2018/387","url":"https://ijcai.org/proceedings/2018/387","pdf":"https://ijcai.org/proceedings/2018/0387.pdf","markdown":"https://ijcai.org/proceedings/2018/387.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/387","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":387,"pages":{"first":2790,"last":2797},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Machine Learning Applications: Humanities"],"videos":[],"references":[{"position":1,"raw":"[Ballas et al., 2016] N. Ballas, L. Yao, C. Pal, and A. Courville. Delving deeper into convolutional networks for learning video representations. In ICLR, 2016.","title":"Delving deeper into convolutional networks for learning video representations","authors":"N Ballas, L Yao, C Pal, A Courville","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bulat and Tzimiropoulos, 2016] Adrian Bulat and Georgios Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV, pages 717-732, 2016.","title":"Human pose estimation via convolutional part heatmap regression","authors":"Adrian Bulat, Georgios Tzimiropoulos","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira et al., 2016] J. Carreira, P. Agrawal, K. Fragki- adaki, and J. Malik. Human pose estimation with iterative error feed-back. In CVPR, page 4733-4742, 2016.","title":"Human pose estimation with iterative error feed-back","authors":"J Carreira, P Agrawal, K Fragkiadaki, J Malik","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graves et al., 2014] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. In arXiv:1410.5401, 2014.","title":"Neural turing machines","authors":"A Graves, G Wayne, I Danihelka","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graves et al., 2016] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. G. Col- menarejo, E. Grefenstette, T. Ramalho, J. Agapiou, and et al. Hybrid computing using a neural network with dy- namic external memory. Nature, 538:471-476, 2016.","title":"Hybrid computing using a neural network with dynamic external memory","authors":"A Graves, G Wayne, M Reynolds, T Harley, I Danihelka, A Grabska-Barwinska, S G Colmenarejo, E Grefenstette, T Ramalho, J Agapiou","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Grinciunaite et al., 2016] Agne Grinciunaite, Amogh Gudi, Emrah Tasli, and Marten den Uyl. Human pose estimation in space and time using 3d cnn. In Gang Hua and Hervé Jégou, editors, ECCV Workshops, 2016.","title":"Human pose estimation in space and time using 3d cnn","authors":"Agne Grinciunaite, Amogh Gudi, Emrah Tasli, Marten Den, Uyl","year":"2016","venue":"ECCV Workshops","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haque et al., 2016] Albert Haque, Boya Peng, Zelun Luo, Alexandre Alahi, Serena Yeung, and Li Fei-Fei. Towards viewpoint invariant 3d human pose estimation. In ECCV, pages 160-177, 2016.","title":"Towards viewpoint invariant 3d human pose estimation","authors":"Albert Haque, Boya Peng, Zelun Luo, Alexandre Alahi, Serena Yeung, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"S Hochreiter, J Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe, 2017] Sergey Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized mod- els. In Advances in Neural Information Processing Sys- tems (NIPS), pages 1945-1953, 2017.","title":"Batch renormalization: Towards reducing minibatch dependence in batch-normalized models","authors":"Sergey Ioffe","year":"2017","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jung et al., 2015] Ho Yub Jung, Soochahn Lee, Yong Seok Heo, and Il Dong Yun. Random tree walk toward instanta- neous 3d human pose estimation. In CVPR, pages 2467- 2474, 2015.","title":"Random tree walk toward instantaneous 3d human pose estimation","authors":"Jung Ho Yub, Soochahn Lee, Yong Seok Heo, Il Dong, Yun","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kendall and Cipolla, 2016] A. Kendall and R. Cipolla. Modelling uncertainty in deep learning for camera relo- calization. In ICRA, pages 4762-4769, 2016.","title":"Modelling uncertainty in deep learning for camera relocalization","authors":"A Kendall, R Cipolla","year":"2016","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1990] Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, and D. Henderson. Handwritten digit recognition with a back- propagation network. In NIPS, pages 396-404, 1990.","title":"Handwritten digit recognition with a backpropagation network","authors":"Y Lecun, B Boser, J S Denker, D Henderson, R E Howard, W Hubbard, L D Jackel, D Henderson","year":"1990","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2016] Xiaodan Liang, Xiaohui Shen, Donglai Xiang, Jiashi Feng, Liang Lin, and Shuicheng Yan. Se- mantic object parsing with local-global long short-term memory. In CVPR, pages 3185-3193, 2016.","title":"Semantic object parsing with local-global long short-term memory","authors":"Xiaodan Liang, Xiaohui Shen, Donglai Xiang, Jiashi Feng, Liang Lin, Shuicheng Yan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McColl et al., 2011] Derek McColl, Zhe Zhang, and Goldie Nejat. Human body pose interpretation and classification for social human-robot interaction. IJSR, 3(3):313, Jun 2011.","title":"Human body pose interpretation and classification for social human-robot interaction","authors":"Derek Mccoll, Zhe Zhang, Goldie Nejat","year":"2011","venue":"IJSR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Na et al., 2017] Seil Na, Sangho Lee, Jisung Kim, and Gun- hee Kim. A read-write memory network for movie story understanding. In ICCV, pages 677-685, 2017.","title":"A read-write memory network for movie story understanding","authors":"Seil Na, Sangho Lee, Jisung Kim, Gunhee Kim","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Newell et al., 2016] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose esti- mation. In ECCV, pages 483-499, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Alejandro Newell, Kaiyu Yang, Jia Deng","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Park et al., 2017] Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. Attend to you: Personalized im- age captioning with context sequence memory networks. In CVPR, pages 6432-6440, 2017.","title":"Attend to you: Personalized image captioning with context sequence memory networks","authors":"Park","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Santoro et al., 2016] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lill- icrap. Meta-learning with memory-augmented neural networks. In ICML, pages 1842-1850, 2016.","title":"Meta-learning with memory-augmented neural networks","authors":"Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2015] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai kin Wong, and Wang chun Woo. Con- volutional lstm network: A machine learning approach for precipitation nowcasting. In NIPS, pages 802-810, 2015.","title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","authors":"Shi","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shotton et al., 2011] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. In CVPR, pages 1297-1304, 2011.","title":"Real-time human pose recognition in parts from single depth images","authors":"J Shotton, A Fitzgibbon, M Cook, T Sharp, M Finocchio, R Moore, A Kipman, A Blake","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shotton et al., 2013] Jamie Shotton, Toby Sharp, Alex Kip- man, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. Real-time human pose recognition in parts from single depth images. Commun. ACM, 56(1):116-124, 2013.","title":"Real-time human pose recognition in parts from single depth images","authors":"Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, Richard Moore","year":"2013","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tompson et al., 2014] Jonathan Tompson, Arjun Jain, Yann Lecun, and Christoph Bregler. Joint training of a convo- lutional network and a graphical model for human pose estimation. In NIPS, pages 1799-1807, 2014.","title":"Joint training of a convolutional network and a graphical model for human pose estimation","authors":"Jonathan Tompson, Arjun Jain, Yann Lecun, Christoph Bregler","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Toshev and Szegedy, 2014] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural net- works. In CVPR, pages 1653-1660, 2014.","title":"Deeppose: Human pose estimation via deep neural networks","authors":"A Toshev, C Szegedy","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Keze Wang, Shengfu Zhai, Hui Cheng, Xiaodan Liang, and Liang Lin. Human pose estima- tion from depth images via inference embedded multi-task learning. In ACM MM, pages 1227-1236, 2016.","title":"Human pose estimation from depth images via inference embedded multi-task learning","authors":"Wang","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wei et al., 2016] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, pages 4724-4732, 2016.","title":"Convolutional pose machines","authors":"Shih-En Wei, Varun Ramakrishna, Takeo Kanade, Yaser Sheikh","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Weston et al., 2015] Jason Weston, Sumit Chopra, and An- toine Bordes. Memory networks. In ICLR, 2015.","title":"Memory networks","authors":"Jason Weston, Sumit Chopra","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu and Cheng, 2013] C. Xu and L. Cheng. Efficient hand pose estimation from a single depth image. In ICCV, pages 3456-3462, 2013.","title":"Efficient hand pose estimation from a single depth image","authors":"C Xu, L Cheng","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2015] Chi Xu, Ashwin Nanjappa, Xiaowei Zhang, and Li Cheng. Estimate hand poses efficiently from single depth images. IJCV, 116(1):21-45, 2015.","title":"Estimate hand poses efficiently from single depth images","authors":"Xu","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. Shufflenet: An extremely effi- cient convolutional neural network for mobile devices. In CVPR, 2018.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}