{"type":"ScholarlyArticle","title":"Distortion-aware CNNs for Spherical Images","authors":[{"name":"Qiang Zhao","slug":"qiang-zhao","given":"Qiang","family":"Zhao","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Chen Zhu","slug":"chen-zhu","given":"Chen","family":"Zhu","affiliation":"University of Chinese Academy of Sciences , Beijing , China; Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Feng Dai","slug":"feng-dai","given":"Feng","family":"Dai","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Yike Ma","slug":"yike-ma","given":"Yike","family":"Ma","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Guoqing Jin","slug":"guoqing-jin","given":"Guoqing","family":"Jin","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"}],"abstract":"Convolutional neural networks are widely used in computer vision applications. Although they have achieved great success, these networks can not be applied to 360 spherical images directly due to varying distortion effect. In this paper, we present distortion-aware convolutional network for spherical images. For each pixel, our network samples a non-regular grid based on its distortion level, and convolves the sampled grid using square kernels shared by all pixels. The network successively approximates large image patches from different tangent planes of viewing sphere with small local sampling grids, thus improves the computational efficiency. Our method also deals with the boundary problem, which is an inherent issue for spherical images. To evaluate our method, we apply our network in spherical image classification problems based on transformed MNIST and CIFAR-10 datasets. Compared with the baseline method, our method can get much better performance. We also analyze the variants of our network.","doi":"10.24963/ijcai.2018/167","url":"https://ijcai.org/proceedings/2018/167","pdf":"https://ijcai.org/proceedings/2018/0167.pdf","markdown":"https://ijcai.org/proceedings/2018/167.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/167","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":167,"pages":{"first":1198,"last":1204},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Classification","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Boomsma and Frellsen, 2017] Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular modelling. In NIPS, pages 3436-3446. 2017.","title":"Spherical convolutions and their application in molecular modelling","authors":"Frellsen Boomsma, Wouter Boomsma, Jes Frellsen","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown and Lowe, 2007] Matthew Brown and David G. Lowe. Automatic panoramic image stitching using invari- ant features. IJCV, 74(1):59-73, 2007.","title":"Automatic panoramic image stitching using invariant features","authors":"Matthew Brown, David G Lowe","year":"2007","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruna et al., 2014] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and deep lo- cally connected networks on graphs. arXiv: 1312.6203v3, 2014.","title":"Spectral networks and deep locally connected networks on graphs","authors":"Joan Bruna, Wojciech Zaremba, Arthur Szlam, Yann Lecun","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cohen et al., 2017] Taco S. Cohen, Mario Geiger, Jonas Köehler, and Max Welling. Convolutional networks for spherical signals. arXiv: 1709.04893, 2017.","title":"Convolutional networks for spherical signals","authors":"S Taco, Mario Cohen, Jonas Geiger, Max Köehler, Welling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2016] Chao Dong, Chen Change Loy, and Xi- aoou Tang. Accelerating the super-resolution convolu- tional neural network. In ECCV, pages 391-407, 2016.","title":"Accelerating the super-resolution convolutional neural network","authors":"Chao Dong, Chen Change Loy, Xiaoou Tang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duvenaud et al., 2015] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Tim- othy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, pages 2224-2232. 2015.","title":"Convolutional networks on graphs for learning molecular fingerprints","authors":"Dougal David K Duvenaud, Jorge Maclaurin, Rafael Iparraguirre, Timothy Bombarell, Alan Hirzel, Ryan P Aspuru-Guzik, Adams","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2017] Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, and Min Sun. Deep 360 pilot: Learning a deep agent for piloting through 360 • sports videos. In CVPR, pages 1396-1405, 2017.","title":"Deep 360 pilot: Learning a deep agent for piloting through 360 • sports videos","authors":"Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial trans- former networks. In NIPS, pages 2017-2025. 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Max Jaderberg","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Khasanova and Frossard, 2017] Renata Khasanova and Pas- cal Frossard. Graph-based classification of omnidirec- tional images. In ICCV workshop, pages 869-878, 2017.","title":"Renata Khasanova and Pascal Frossard. Graph-based classification of omnidirectional images","authors":"Frossard Khasanova","year":"2017","venue":"ICCV workshop","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Lei Ba. Adam: a method for stochastic optimization. arXiv: 1412.6980, 2014.","title":"Adam: a method for stochastic optimization","authors":"P Diederik, Jimmy Lei Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105. 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2017] Wei-Sheng Lai, Yujia Huang, Neel Joshi, Christopher Buehler, Ming-Hsuan Yang, and Sing Bing Kang. Semantic-driven generation of hyperlapse from 360 • video. IEEE TVCG, 2017.","title":"Semantic-driven generation of hyperlapse from 360 • video","authors":"Wei-Sheng Lai, Yujia Huang, Neel Joshi, Christopher Buehler, Ming-Hsuan Yang, Bing Sing, Kang","year":"2017","venue":"IEEE TVCG","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lecun et al., 1998] Yann Lecun, Leon Bottou, Yoshua Ben- gio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521:436-444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Micusik and Kosecka, 2009] Branislav Micusik and Jana Kosecka. Piecewise planar city 3D modeling from street view panoramic sequences. In CVPR, pages 2906-2912, 2009.","title":"Piecewise planar city 3D modeling from street view panoramic sequences","authors":"Kosecka Micusik, Branislav Micusik, Jana Kosecka","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Su and Grauman, 2017] Yu-Chuan Su and Kristen Grau- man. Learning spherical convolution for fast features from 360 • imagery. In NIPS, pages 529-539. 2017.","title":"Learning spherical convolution for fast features from 360 • imagery","authors":"Yu-Chuan Su, Kristen Grauman","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Uyttendaele et al., 2004] Matthew Uyttendaele, Antonio Criminisi, Sing Bing Kang, Simon Winder, Richard Szeliski, and Richard Hartley. Image-based interactive exploration of real-world environments. IEEE CGA, 24(3):52-63, 2004.","title":"Image-based interactive exploration of real-world environments","authors":"Matthew Uyttendaele, Antonio Criminisi, Bing Sing, Simon Kang, Richard Winder, Richard Szeliski, Hartley","year":"2004","venue":"IEEE CGA","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xiao et al., 2012] Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Recognizing scene viewpoint using panoramic place representation. In CVPR, pages 2695-2702, 2012.","title":"Recognizing scene viewpoint using panoramic place representation","authors":"Jianxiong Xiao, Krista A Ehinger, Aude Oliva, Antonio Torralba","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang and Zhang, 2016] Hao Yang and Hui Zhang. Efficient 3d room shape recovery from a single panorama. In CVPR, pages 5422-5430, 2016.","title":"Efficient 3d room shape recovery from a single panorama","authors":"Hao Yang, Hui Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2013] Qiang Zhao, Liang Wan, Wei Feng, Ji- awan Zhang, and Tien-Tsin Wong. Cube2Video: Navi- gate between cubic panoramas in real-time. IEEE TMM, 15(8):1745-1754, 2013.","title":"Cube2Video: Navigate between cubic panoramas in real-time","authors":"Qiang Zhao, Liang Wan, Wei Feng, Jiawan Zhang, Tien-Tsin Wong","year":"2013","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zorin and Barr, 1995] Denis Zorin and Alan H. Barr. Cor- rection of geometric perceptual distortions in pictures. In Annual Conference on Computer Graphics and Interactive Techniques, pages 257-264, 1995.","title":"Correction of geometric perceptual distortions in pictures","authors":"Denis Zorin, Alan H Barr","year":"1995","venue":"Annual Conference on Computer Graphics and Interactive Techniques","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}