{"type":"ScholarlyArticle","title":"BARNet: Bilinear Attention Network with Adaptive Receptive Fields for Surgical Instrument Segmentation","authors":[{"name":"Zhen-Liang Ni","slug":"zhen-liang-ni","given":"Zhen-Liang","family":"Ni","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China; The School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Gui-Bin Bian","slug":"gui-bin-bian","given":"Gui-Bin","family":"Bian","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China; The School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Guan-An Wang","slug":"guan-an-wang","given":"Guan'an","family":"Wang","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China; The School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Xiao-Hu Zhou","slug":"xiao-hu-zhou","given":"Xiao-Hu","family":"Zhou","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Zeng-Guang Hou","slug":"zeng-guang-hou","given":"Zeng-Guang","family":"Hou","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China; The School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China; CAS Center for Excellence in Brain Science and Intelligence Technology , Beijing , China"},{"name":"Xiao-Liang Xie","slug":"xiao-liang-xie","given":"Xiao-Liang","family":"Xie","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Zhen Li","slug":"zhen-li","given":"Zhen","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Yu-Han Wang","slug":"yu-han-wang","given":"Yu-Han","family":"Wang","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"}],"abstract":"Surgical instrument segmentation is crucial for computer-assisted surgery. Different from common object segmentation, it is more challenging due to the large illumination variation and scale variation in the surgical scenes. In this paper, we propose a bilinear attention network with adaptive receptive fields to address these two issues. To deal with the illumination variation, the bilinear attention module models global contexts and semantic dependencies between pixels by capturing second-order statistics. With them, semantic features in challenging areas can be inferred from their neighbors, and the distinction of various semantics can be boosted. To adapt to the scale variation, our adaptive receptive field module aggregates multi-scale features and selects receptive fields adaptively. Specifically, it models the semantic relationships between channels to choose feature maps with appropriate scales, changing the receptive field of subsequent convolutions. The proposed network achieves the best performance 97.47% mean IoU on Cata7. It also takes the first place on EndoVis 2017, exceeding the second place by 10.10% mean IoU.","doi":"10.24963/ijcai.2020/116","url":"https://ijcai.org/proceedings/2020/116","pdf":"https://ijcai.org/proceedings/2020/0116.pdf","markdown":"https://ijcai.org/proceedings/2020/116.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/116","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":116,"pages":{"first":832,"last":838},"keywords":["Machine Learning: Deep Learning","Machine Learning Applications: Bio/Medicine","Robotics: Robotics and Vision","Computer Vision: Biomedical Image Understanding"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24466/file_24466.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Allan et al., 2019] Max Allan, Alex Shvets, Thomas Kur- mann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nico- la Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bo- denstedt, et al. 2017 robotic instrument segmentation chal- lenge. arXiv preprint arXiv:1902.06426, 2019.","title":"robotic instrument segmentation challenge","authors":"Allan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Attia et al., 2017] Mohamed Attia, Mohammed Hossny, Saeid Nahavandi, and Hamed Asadi. Surgical tool seg- mentation using a hybrid deep cnn-rnn auto encoder- decoder. In 2017 IEEE International Conference on Sys- tems, Man, and Cybernetics, pages 3373-3378, 2017.","title":"Surgical tool segmentation using a hybrid deep cnn-rnn auto encoderdecoder","authors":"Mohamed Attia, Mohammed Hossny, Saeid Nahavandi, Hamed Asadi","year":"2017","venue":"IEEE International Conference on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Badrinarayanan et al., 2017] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convo- lutional encoder-decoder architecture for image segmenta- tion. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481-2495, 2017.","title":"SegNet: A deep convolutional encoder-decoder architecture for image segmentation","authors":"Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaurasia and Culurciello, 2017] Abhishek Chaurasia and Eugenio Culurciello. LinkNet: Exploiting encoder rep- resentations for efficient semantic segmentation. In IEEE Visual Communications and Image Processing, pages 1-4, 2017.","title":"LinkNet: Exploiting encoder representations for efficient semantic segmentation","authors":"Culurciello Chaurasia, Abhishek Chaurasia, Eugenio Culurciello","year":"2017","venue":"IEEE Visual Communications and Image Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intel- ligence, 40(4):834-848, 2017.","title":"DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Yunpeng Chen, Yannis Kalantidis, Jian- shu Li, Shuicheng Yan, and Jiashi Feng. Aˆ2-Nets: Dou- ble attention networks. In Advances in Neural Information Processing Systems 31, pages 352-361. 2018.","title":"Aˆ2-Nets: Double attention networks","authors":"Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[García-Peraza-Herrera et al., 2017] Luis C García-Peraza- Herrera, Wenqi Li, Lucas Fidon, Caspar Gruijthuijsen, Alain Devreker, George Attilakos, Jan Deprest, Emmanuel Vander Poorten, Danail Stoyanov, Tom Vercauteren, et al. Toolnet: Holistically-nested real-time segmentation of robotic surgical tools. In the International Conference on Intelligent Robots and Systems, pages 5717-5722, 2017.","title":"Toolnet: Holistically-nested real-time segmentation of robotic surgical tools","authors":"C Luis, Wenqi García-Peraza-Herrera, Lucas Li, Caspar Fidon, Alain Gruijthuijsen, George Devreker, Jan Attilakos, Emmanuel Deprest, Danail Vander Poorten, Tom Stoyanov, Vercauteren","year":"2017","venue":"the International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Iglovikov and Shvets, 2018] Vladimir Iglovikov and Alex- ey Shvets. TernausNet: U-Net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv:1801.05746, 2018.","title":"Vladimir Iglovikov and Alexey Shvets. TernausNet: U-Net with vgg11 encoder pre-trained on imagenet for image segmentation","authors":"Shvets Iglovikov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2019] Yueming Jin, Keyun Cheng, Qi Dou, and Pheng-Ann Heng. Incorporating temporal prior from mo- tion flow for instrument segmentation in minimally in- vasive surgery video. In Medical Image Computing and Computer Assisted Intervention, pages 440-448, 2019.","title":"Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video","authors":"Yueming Jin, Keyun Cheng, Qi Dou, Pheng-Ann Heng","year":"2019","venue":"Medical Image Computing and Computer Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2018] Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, pages 1564- 1574, 2018.","title":"Bilinear attention networks","authors":"Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 510-519, 2019.","title":"Selective kernel networks","authors":"Xiang Li, Wenhai Wang, Xiaolin Hu, Jian Yang","year":"2019","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Maji, 2017] Tsung-Yu Lin and Subhransu Maji. Improved bilinear pooling with cnns. In the British Ma- chine Vision Conference, pages 117.1-117.12, 2017.","title":"Improved bilinear pooling with cnns","authors":"Lin, Maji ; Tsung-Yu Lin, Subhransu Maji","year":"2017","venue":"the British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2017a] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. RefineNet: Multi-path refinement net- works for high-resolution semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 1925-1934, 2017.","title":"RefineNet: Multi-path refinement networks for high-resolution semantic segmentation","authors":"Lin","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2017b] Tsung-Yi Lin, Piotr Dollar, Ross Gir- shick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"Feature pyramid networks for object detection","authors":"Lin","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ni et al., 2019] Zhen-Liang Ni, Gui-Bin Bian, Xiao-Hu Zhou, Zeng-Guang Hou, Xiao-Liang Xie, Chen Wang, Yan-Jie Zhou, Rui-Qi Li, and Zhen Li. RAUNet: Residual attention U-Net for semantic segmentation of cataract sur- gical instruments. In International Conference on Neural Information Processing, pages 139-149. Springer, 2019.","title":"RAUNet: Residual attention U-Net for semantic segmentation of cataract surgical instruments","authors":"Zhen-Liang Ni, Gui-Bin Bian, Xiao-Hu Zhou, Zeng-Guang Hou, Xiao-Liang Xie, Chen Wang, Yan-Jie Zhou, Rui-Qi Li, Zhen Li","year":"2019","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qin et al., 2019] Fangbo Qin, Yangming Li, Yun-Hsuan Su, De Xu, and Blake Hannaford. Surgical instrument seg- mentation for endoscopic vision with data fusion of redic- tion and kinematic pose. In International Conference on Robotics and Automation, pages 9821-9827, 2019.","title":"Surgical instrument segmentation for endoscopic vision with data fusion of rediction and kinematic pose","authors":"Fangbo Qin, Yangming Li, Yun-Hsuan Su, De Xu, Blake Hannaford","year":"2019","venue":"International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fisch- er, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Confer- ence on Medical image computing and computer-assisted intervention, pages 234-241, 2015.","title":"U-Net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computer-assisted intervention","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sarikaya et al., 2017] Duygu Sarikaya, Jason J Corso, and Khurshid A Guru. Detection and localization of robot- ic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE trans- actions on medical imaging, 36(7):1542-1549, 2017.","title":"Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection","authors":"Duygu Sarikaya, Jason J Corso, Khurshid A Guru","year":"2017","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2018] Maoke Yang, Kun Yu, Chi Zhang, Zhi- wei Li, and Kuiyuan Yang. Denseaspp for semantic seg- mentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3684-3692, 2018.","title":"Denseaspp for semantic segmentation in street scenes","authors":"Yang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2018] Xiaoning Zhang, Tiantian Wang, Jin- qing Qi, Huchuan Lu, and Gang Wang. Progressive at- tention guided recurrent network for salient object detec- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 714-722, 2018.","title":"Progressive attention guided recurrent network for salient object detection","authors":"Xiaoning Zhang, Tiantian Wang, Jinqing Qi, Huchuan Lu, Gang Wang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881- 2890, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}