{"type":"ScholarlyArticle","title":"Combining Models from Multiple Sources for RGB-D Scene Recognition","authors":[{"name":"Xinhang Song","slug":"xinhang-song","given":"Xinhang","family":"Song","affiliation":"Key Lab of Intel. Inf. Proc. , Inst. of Comput. Tech. , Chinese Academy of Sciences , Beijing , 100190 , China; University of Chinese Academy of Sciences , Beijing , 100049 , China"},{"name":"Shuqiang Jiang","slug":"shuqiang-jiang","given":"Shuqiang","family":"Jiang","affiliation":"Key Lab of Intel. Inf. Proc. , Inst. of Comput. Tech. , Chinese Academy of Sciences , Beijing , 100190 , China; University of Chinese Academy of Sciences , Beijing , 100049 , China"},{"name":"Luis Herranz","slug":"luis-herranz","given":"Luis","family":"Herranz","affiliation":"Computer Vision Center , 08193 Bellaterra , Barcelona , Spain"}],"abstract":"Depth can complement RGB with useful cues about object volumes and scene layout. However, RGB-D image datasets are still too small for directly training deep convolutional neural networks (CNNs), in contrast to the massive monomodal RGB datasets. Previous works in RGB-D recognition typically combine two separate networks for RGB and depth data, pretrained with a large RGB dataset and then fine tuned to the respective target RGB and depth datasets. These approaches have several limitations: 1) only use low-level filters learned from RGB data, thus not being able to exploit properly depth-specific patterns, and 2) RGB and depth features are only combined at high-levels but rarely at lower-levels. In this paper, we propose a framework that leverages both knowledge acquired from large RGB datasets together with depth-specific cues learned from the limited depth data, obtaining more effective multi-source and multi-modal representations. We propose a multi-modal combination method that selects discriminative combinations of layers from the different source models and target modalities, capturing both high-level properties of the task and intrinsic low-level properties of both modalities.","doi":"10.24963/ijcai.2017/631","url":"https://ijcai.org/proceedings/2017/631","pdf":"https://ijcai.org/proceedings/2017/0631.pdf","markdown":"https://ijcai.org/proceedings/2017/631.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/631","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Robotics and Vision","track":"Main track","number":631,"pages":{"first":4523,"last":4529},"keywords":["Robotics and Vision: Robotics and Vision","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Banica and Sminchisescu, 2015] Dan Banica and Cristian Sminchisescu. Second-order constrained parametric pro- posals and sequential search-based structured prediction for semantic segmentation in rgb-d images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.","title":"Second-order constrained parametric proposals and sequential search-based structured prediction for semantic segmentation in rgb-d images","authors":"Sminchisescu Banica, Dan Banica, Cristian Sminchisescu","year":"2015","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dash and Liu, 1997] Manoranjan Dash and Huan Liu. Fea- ture selection for classification. Intelligent Data Analysis, 1(1):131 -156, 1997.","title":"Feature selection for classification","authors":"Manoranjan Dash, Huan Liu","year":"1997","venue":"Intelligent Data Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning, (ICML), 2014.","title":"DeCAF: A deep convolutional activation feature for generic visual recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2008] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. Journal of Machine Learning Research (JMLR), 9:1871-1874, 2008.","title":"Liblinear: A library for large linear classification","authors":"Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, Chih-Jen Lin","year":"2008","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gupta et al., 2013] Saurabh Gupta, Pablo Arbeláez, and Ji- tendra Malik. Perceptual organization and recognition of indoor scenes from rgb-d images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"Perceptual organization and recognition of indoor scenes from rgb-d images","authors":"Saurabh Gupta, Pablo Arbeláez, Jitendra Malik","year":"2013","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gupta et al., 2014a] Saurabh Gupta, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Indoor scene understanding with rgb-d images: Bottom-up segmentation, object detec- tion and semantic segmentation. International Journal of Computer Vision (IJCV), 112:133-149, 2014.","title":"Indoor scene understanding with rgb-d images: Bottom-up segmentation, object detection and semantic segmentation","authors":"Gupta","year":"2014","venue":"International Journal of Computer Vision (IJCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2014b] Saurabh Gupta, Ross Girshick, Ross Girshickand Pablo Arbeláez, and Jitendra Malik. Learn- ing rich features from rgb-d images for object detection and segmentation. In European Conference on Computer Vision (ECCV), 2014.","title":"Learning rich features from rgb-d images for object detection and segmentation","authors":"Gupta","year":"2014","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] Saurabh Gupta, Judy Hoffman, and Ji- tendra Malik. Cross modal distillation for supervision transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Cross modal distillation for supervision transfer","authors":"Saurabh Gupta, Judy Hoffman, Jitendra Malik","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Neural Information Pro- cessing Systems (NIPS), pages 1106-1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Modrzejewski, 1993] Maciej Modrzejewski. Feature selec- tion using rough sets theory. In Proceedings of the Euro- pean Conference on Machine Learning, 1993.","title":"Feature selection using rough sets theory","authors":"Maciej Modrzejewski","year":"1993","venue":"Proceedings of the European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mucciardi and Gose, 1971] Anthony Mucciardi and Earl Gose. A comparison of seven techniques for choosing sub- sets of pattern recognition properties. IEEE Transactions on Computers, 20(9):1023-1031, September 1971.","title":"A comparison of seven techniques for choosing subsets of pattern recognition properties","authors":"Anthony Mucciardi, Earl Gose","year":"1971","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Silberman and Fergus, 2011] Nathan Silberman and Rob Fergus. Indoor scene segmentation using a structured light sensor. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 601-608, November 2011.","title":"Indoor scene segmentation using a structured light sensor","authors":"Nathan Silberman, Rob Fergus","year":"2011","venue":"Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Silberman et al., 2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the 12th European Conference on Computer Vision - Volume Part V, ECCV'12, pages 746-760, Berlin, Heidel- berg, 2012. Springer-Verlag.","title":"Indoor segmentation and support inference from rgbd images","authors":"Silberman","year":"2012","venue":"Proceedings of the 12th European Conference on Computer Vision -Volume Part V, ECCV'12","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Song et al., 2015] Shuran Song, Samuel Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Computer Vision and Pattern Recog- nition (CVPR), 2015 IEEE Conference on, pages 567-576, June 2015.","title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","authors":"Song","year":"2015","venue":"Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2017] Xinhang Song, Luis Herranz, and Shuqiang Jiang. Depth cnns for RGB-D scene recog- nition: Learning from scratch better than transferring from rgb-cnns. In the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 4271-4277, 2017.","title":"Depth cnns for RGB-D scene recognition: Learning from scratch better than transferring from rgb-cnns","authors":"Song","year":"2017","venue":"the Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2016] Anran Wang, Jianfei Cai, Jiwen Lu, and Tat-Jen Cham. Modality and component aware fea- ture fusion for rgb-d scene classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Modality and component aware feature fusion for rgb-d scene classification","authors":"Wang","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xiao et al., 2013] Jianxiong Xiao, A. Owens, and A. Tor- ralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 1625-1632, De- cember 2013.","title":"Sun3d: A database of big spaces reconstructed using sfm and object labels","authors":"A Jianxiong Xiao, A Owens, Torralba","year":"2013","venue":"Computer Vision (ICCV), 2013 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang and Ramanan, 2015] Songfan Yang and Deva Ra- manan. Multi-scale recognition with dag-cnns. In The IEEE International Conference on Computer Vision (ICCV), December 2015.","title":"Songfan Yang and Deva Ramanan. Multi-scale recognition with dag-cnns","authors":"Ramanan Yang","year":"2015","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhou et al., 2014] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Neural Information Processing Systems (NIPS), pages 487-495, 2014.","title":"Learning deep features for scene recognition using places database","authors":"Zhou","year":"2014","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhu et al., 2016] Hongyuan Zhu, Jean-Baptiste Weibel, and Shijian Lu. Discriminative multi-modal feature fusion for rgbd indoor scene recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Discriminative multi-modal feature fusion for rgbd indoor scene recognition","authors":"Hongyuan Zhu, Jean-Baptiste Weibel, Shijian Lu","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}