{"type":"ScholarlyArticle","title":"Confusion Graph: Detecting Confusion Communities in Large Scale Image Classification","authors":[{"name":"Ruochun Jin","slug":"ruochun-jin","given":"Ruochun","family":"Jin","affiliation":"National Laboratory for Parallel and Distributed Processing , National University of Defense Technology , Changsha , Hunan , 410073 , China"},{"name":"Yong Dou","slug":"yong-dou","given":"Yong","family":"Dou","affiliation":"National Laboratory for Parallel and Distributed Processing , National University of Defense Technology , Changsha , Hunan , 410073 , China"},{"name":"Yueqing Wang","slug":"yueqing-wang","given":"Yueqing","family":"Wang","affiliation":"National Laboratory for Parallel and Distributed Processing , National University of Defense Technology , Changsha , Hunan , 410073 , China"},{"name":"Xin Niu","slug":"xin-niu","given":"Xin","family":"Niu","affiliation":"National Laboratory for Parallel and Distributed Processing , National University of Defense Technology , Changsha , Hunan , 410073 , China"}],"abstract":"For deep CNN-based image classification models, we observe that confusions between classes with high visual similarity are much stronger than those where classes are visually dissimilar. With these unbalanced confusions, classes can be organized in communities, which is similar to cliques of people in the social network. Based on this, we propose a graph-based tool named \"confusion graph\" to quantify these confusions and further reveal the community structure inside the database. With this community structure, we can diagnose the model's weaknesses and improve the classification accuracy using specialized expert sub-nets, which is comparable to other state-of-the-art techniques. Utilizing this community information, we can also employ pre-trained models to automatically identify mislabeled images in the large scale database. With our method, researchers just need to manually check approximate 3% of the ILSVRC2012 classification database to locate almost all mislabeled samples.","doi":"10.24963/ijcai.2017/275","url":"https://ijcai.org/proceedings/2017/275","pdf":"https://ijcai.org/proceedings/2017/0275.pdf","markdown":"https://ijcai.org/proceedings/2017/275.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/275","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":275,"pages":{"first":1980,"last":1986},"keywords":["Machine Learning: Classification","Machine Learning: Deep Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Ahmed et al., 2016] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. arXiv preprint arX- iv:1604.06119, 2016.","title":"Network of experts for large-scale image categorization","authors":"Karim Ahmed, Mohammad Haris Baig, Lorenzo Torresani","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blondel et al., 2008] Vincent D Blondel, Jean-Loup Guil- laume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Jour- nal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.","title":"Fast unfolding of communities in large networks","authors":"Jean-Loup Vincent D Blondel, Renaud Guillaume, Etienne Lambiotte, Lefebvre","year":"2008","venue":"Journal of statistical mechanics: theory and experiment","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Breiman and Wald Lecture, 2002] Leo Breiman and I- I Wald Lecture. Looking inside the black box. Wald Lecture II, Department of Statistics, California University, 2002.","title":"Leo Breiman and I-I Wald Lecture","authors":"Wald Breiman, Lecture","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brodley and Friedl, 1999] Carla E. Brodley and Mark A. Friedl. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11:131-167, 1999.","title":"Identifying mislabeled training data","authors":"Carla E Brodley, Mark A Friedl","year":"1999","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frénay and Verleysen, 2014] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learn- ing systems, 25(5):845-869, 2014.","title":"Classification in the presence of label noise: a survey","authors":"Verleysen Frénay","year":"2014","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Freund, 2009] Yoav Freund. A more robust boosting algo- rithm. arXiv preprint arXiv:0905.2138, 2009.","title":"A more robust boosting algorithm","authors":"Yoav Freund","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kabra et al., 2015] Mayank Kabra, Alice Robie, and Kristin Branson. Understanding classifier errors by examining in- fluential neighbors. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 3917-3925, 2015.","title":"Understanding classifier errors by examining influential neighbors","authors":"Mayank Kabra, Alice Robie, Kristin Branson","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kontschieder et al., 2015] Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 1467- 1475, 2015.","title":"Deep neural decision forests","authors":"Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, Samuel Rota Bulo","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":"/proceedings/2016/628"},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1995] Yann LeCun, LD Jackel, Leon Bottou, A Brunot, Corinna Cortes, JS Denker, Harris Drucker, I Guyon, UA Muller, Eduard Sackinger, et al. Comparison of learning algorithms for handwritten digit recognition. In International conference on artificial neural networks, volume 60, pages 53-60. Perth, Australia, 1995.","title":"Comparison of learning algorithms for handwritten digit recognition","authors":"Yann Lecun, Leon Jackel, Bottou, Corinna Brunot, Cortes, Harris Denker, I Drucker, Guyon, Eduard Muller, Sackinger","year":"1995","venue":"International conference on artificial neural networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nilsback and Zisserman, 2008] M-E. Nilsback and A. Zis- serman. Automated flower classification over a large num- ber of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.","title":"Automated flower classification over a large number of classes","authors":"M-E Nilsback, A Zisserman","year":"2008","venue":"Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sánchez et al., 2003] José Salvador Sánchez, Ricardo Barandela, Ana I Marqués, Roberto Alejo, and Jorge Badenas. Analysis of new techniques to obtain quality training sets. Pattern Recognition Letters, 24(7):1015- 1022, 2003.","title":"José Salvador Sánchez, Ricardo Barandela, Ana I Marqués, Roberto Alejo, and Jorge Badenas. Analysis of new techniques to obtain quality training sets","authors":"Sánchez","year":"2003","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arX- iv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stokes et al., 2016] Jack W Stokes, Ashish Kapoor, and De- bajyoti Ray. Asking for a second opinion: Re-querying of noisy multi-class labels. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2329-2333. IEEE, 2016.","title":"Asking for a second opinion: Re-querying of noisy multi-class labels","authors":"Jack W Stokes, Ashish Kapoor, Debajyoti Ray","year":"2016","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Van Horn et al., 2015] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recog- nition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-604, 2015.","title":"Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection","authors":"Van Horn","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vedaldi and Lenc, 2015] Andrea Vedaldi and Karel Lenc. Matconvnet: Convolutional neural networks for matlab. In Proceedings of the 23rd ACM international conference on Multimedia, pages 689-692. ACM, 2015.","title":"Matconvnet: Convolutional neural networks for matlab","authors":"Andrea Vedaldi, Karel Lenc","year":"2015","venue":"Proceedings of the 23rd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vondrick et al., 2013] Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision, pages 1-8, 2013.","title":"Hoggles: Visualizing object detection features","authors":"Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, Antonio Torralba","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yan et al., 2015] Zhicheng Yan, Hao Zhang, Robinson Pira- muthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neu- ral networks for large scale visual recognition. In Proceed- ings of the IEEE International Conference on Computer Vision, pages 2740-2748, 2015.","title":"Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition","authors":"Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis Decoste, Wei Di, Yizhou Yu","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818- 833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zelnik-Manor and Perona, 2004] Lihi Zelnik-Manor and Pietro Perona. Self-tuning spectral clustering. In NIPS, volume 17, page 16, 2004.","title":"Self-tuning spectral clustering","authors":"Zelnik-Manor, Perona ; Lihi Zelnik-Manor, Pietro Perona","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}