{"type":"ScholarlyArticle","title":"Counterexample-Guided Data Augmentation","authors":[{"name":"Tommaso Dreossi","slug":"tommaso-dreossi","given":"Tommaso","family":"Dreossi","affiliation":"University of California , Berkeley"},{"name":"Shromona Ghosh","slug":"shromona-ghosh","given":"Shromona","family":"Ghosh","affiliation":"University of California , Berkeley"},{"name":"Xiangyu Yue","slug":"xiangyu-yue","given":"Xiangyu","family":"Yue","affiliation":"University of California , Berkeley"},{"name":"Kurt Keutzer","slug":"kurt-keutzer","given":"Kurt","family":"Keutzer","affiliation":"University of California , Berkeley"},{"name":"Alberto Sangiovanni-Vincentelli","slug":"alberto-sangiovanni-vincentelli","given":"Alberto","family":"Sangiovanni-Vincentelli","affiliation":"University of California , Berkeley"},{"name":"Sanjit A. Seshia","slug":"sanjit-a-seshia","given":"Sanjit A.","family":"Seshia","affiliation":"University of California , Berkeley"}],"abstract":"We present a novel framework for augmenting data sets for\r\nmachine learning based on counterexamples. Counterexamples\r\nare misclassified examples that have \r\nimportant properties for retraining and improving the model.\r\nKey components of our framework include a \\textit{counterexample generator},\r\nwhich produces data items that are misclassified by the model and\r\nerror tables, a novel data\r\nstructure that stores information pertaining to misclassifications.\r\nError tables can be used to explain the model's\r\nvulnerabilities and are used to efficiently generate counterexamples for augmentation.\r\nWe show the efficacy of the proposed framework by comparing it\r\nto classical augmentation techniques on a case study of object detection in autonomous\r\ndriving based on deep neural networks.","doi":"10.24963/ijcai.2018/286","url":"https://ijcai.org/proceedings/2018/286","pdf":"https://ijcai.org/proceedings/2018/0286.pdf","markdown":"https://ijcai.org/proceedings/2018/286.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/286","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":286,"pages":{"first":2071,"last":2078},"keywords":["Computer Vision: Big Data and Large Scale Methods","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Statistical Methods and Machine Learning","Multidisciplinary Topics and Applications: Validation and Verification"],"videos":[],"references":[{"position":1,"raw":"[Alur et al., 2013] Rajeev Alur, Rastislav Bodik, Garvit Ju- niwal, Milo M. K. Martin, Mukund Raghothaman, San- jit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided syn- thesis. In Proceedings of the IEEE International Confer- ence on Formal Methods in Computer-Aided Design (FM- CAD), pages 1-17, October 2013.","title":"Syntax-guided synthesis","authors":"Rajeev Alur, Rastislav Bodik, Garvit Juniwal, M K Milo, Mukund Martin, Raghothaman, A Sanjit, Rishabh Seshia, Armando Singh, Emina Solar-Lezama, Abhishek Torlak, Udupa","year":"2013","venue":"Proceedings of the IEEE International Conference on Formal Methods in Computer-Aided Design (FM-CAD)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ciregan et al., 2012] Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for im- age classification. In Computer vision and pattern recogni- tion (CVPR), 2012 IEEE conference on, pages 3642-3649. IEEE, 2012.","title":"Multi-column deep neural networks for image classification","authors":"Dan Ciregan, Ueli Meier, Jürgen Schmidhuber","year":"2012","venue":"Computer vision and pattern recognition (CVPR), 2012 IEEE conference on","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cires ¸an et al., 2011] Dan C Cires ¸an, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High-performance neural networks for visual object clas- sification. arXiv preprint arXiv:1102.0183, 2011.","title":"High-performance neural networks for visual object classification","authors":"Ueli Dan C Cires ¸an, Jonathan Meier, Masci, M Luca, Jürgen Gambardella, Schmidhuber","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dreossi et al., 2017a] Tommaso Dreossi, Alexandre Donze, and Sanjit A. Seshia. Compositional falsification of cyber- physical systems with machine learning components. In Proceedings of the NASA Formal Methods Conference (NFM), pages 357-372, May 2017.","title":"Compositional falsification of cyberphysical systems with machine learning components","authors":"Dreossi","year":"2017","venue":"Proceedings of the NASA Formal Methods Conference (NFM)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dreossi et al., 2017b] Tommaso Dreossi, Shromona Ghosh, Alberto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. Systematic testing of convolutional neural networks for autonomous driving. In ICML Workshop on Reliable Ma- chine Learning in the Wild (RMLW), 2017. Published on Arxiv: abs/1708.03309.","title":"Systematic testing of convolutional neural networks for autonomous driving","authors":"Dreossi","year":"2017","venue":"ICML Workshop on Reliable Machine Learning in the Wild (RMLW)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dreossi et al., 2018] Tommaso Dreossi, Somesh Jha, and Sanjit A. Seshia. Semantic adversarial deep learning. In 30th International Conference on Computer Aided Verifi- cation (CAV), 2018.","title":"Semantic adversarial deep learning","authors":"Tommaso Dreossi, Somesh Jha, Sanjit A Seshia","year":"2018","venue":"30th International Conference on Computer Aided Verification (CAV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Halton, 1960] John H Halton. On the efficiency of cer- tain quasi-random sequences of points in evaluating multi- dimensional integrals. Numerische Mathematik, 2(1):84- 90, 1960.","title":"On the efficiency of certain quasi-random sequences of points in evaluating multidimensional integrals","authors":"John H Halton","year":"1960","venue":"Numerische Mathematik","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jha and Seshia, 2017] Susmit Jha and Sanjit A. Seshia. A Theory of Formal Synthesis via Inductive Learning. Acta Informatica, 54(7):693-726, 2017.","title":"A Theory of Formal Synthesis via Inductive Learning","authors":"Susmit Jha, Sanjit A Seshia","year":"2017","venue":"Acta Informatica","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2017] Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. Recurrent topic-transition gan for visual paragraph generation. arXiv preprint arXiv:1703.07022, 2017.","title":"Recurrent topic-transition gan for visual paragraph generation","authors":"Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, Eric P Xing","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Marchesi, 2017] Marco Marchesi. Megapixel size image creation using generative adversarial networks. arXiv preprint arXiv:1706.00082, 2017.","title":"Megapixel size image creation using generative adversarial networks","authors":"Marco Marchesi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Niederreiter, 1988] Harald Niederreiter. Low-discrepancy and low-sequences. Journal of number theory, 30(1):51- 70, 1988.","title":"Low-discrepancy and low-sequences","authors":"Harald Niederreiter, Niederreiter","year":"1988","venue":"Journal of number theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Russell et al., 2015] Stuart Russell, Tom Dietterich, Eric Horvitz, Bart Selman, Francesca Rossi, Demis Hassabis, Shane Legg, Mustafa Suleyman, Dileep George, and Scott Phoenix. Letter to the editor: Research priorities for ro- bust and beneficial artificial intelligence: An open letter. AI Magazine, 36(4), 2015.","title":"Letter to the editor: Research priorities for robust and beneficial artificial intelligence: An open letter","authors":"Stuart Russell, Tom Dietterich, Eric Horvitz, Bart Selman, Francesca Rossi, Demis Hassabis, Shane Legg, Mustafa Suleyman, Dileep George, Scott Phoenix","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Seshia et al., 2016] Sanjit A. Seshia, Dorsa Sadigh, and S. Shankar Sastry. Towards Verified Artificial Intelligence. ArXiv e-prints, July 2016.","title":"Towards Verified Artificial Intelligence","authors":"A Sanjit, Dorsa Seshia, S Shankar Sadigh, Sastry","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shrivastava et al., 2016] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Conference on Computer Vision and Pattern Recognition, CVPR, pages 761-769, 2016.","title":"Training region-based object detectors with online hard example mining","authors":"Abhinav Shrivastava, Abhinav Gupta, Ross Girshick","year":"2016","venue":"Conference on Computer Vision and Pattern Recognition, CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simard et al., 2003] Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, volume 3, pages 958-962, 2003.","title":"Best practices for convolutional neural networks applied to visual document analysis","authors":"Patrice Y Simard, David Steinkraus, John C Platt","year":"2003","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sobol, 1976] Ilya M Sobol. Uniformly distributed se- quences with an additional uniform property. USSR Computational Mathematics and Mathematical Physics, 16(5):236-242, 1976.","title":"Uniformly distributed sequences with an additional uniform property","authors":"M Ilya, Sobol","year":"1976","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Solar-Lezama et al., 2006] Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. Combinatorial sketching for finite programs. In Proceedings of the 12th International Conference on Ar- chitectural Support for Programming Languages and Op- erating Systems (ASPLOS), pages 404-415. ACM Press, October 2006.","title":"Combinatorial sketching for finite programs","authors":"Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, A Sanjit, Vijay A Seshia, Saraswat","year":"2006","venue":"Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In Conference on Intelligent Robots and Systems, IROS, pages 23-30. IEEE, 2017.","title":"Domain randomization for transferring deep neural networks from simulation to the real world","authors":"Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":"Conference on Intelligent Robots and Systems, IROS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[van Dyk and Meng, 2001] David A van Dyk and Xiao-Li Meng. The art of data augmentation. Journal of Com- putational and Graphical Statistics, 10(1):1-50, 2001.","title":"The art of data augmentation","authors":"David A Van Dyk, Xiao-Li Meng","year":"2001","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wold et al., 1987] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.","title":"Principal component analysis","authors":"Svante Wold, Kim Esbensen, Paul Geladi","year":"1987","venue":"Chemometrics and intelligent laboratory systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wong et al., 2016] Sebastien C Wong, Adam Gatt, Victor Stamatescu, and Mark D McDonnell. Understanding data augmentation for classification: when to warp? In Digital Image Computing: Techniques and Applications (DICTA), 2016 International Conference on, pages 1-6. IEEE, 2016.","title":"Understanding data augmentation for classification: when to warp?","authors":"Adam Sebastien C Wong, Victor Gatt, Mark D Stamatescu, Mcdonnell","year":"2016","venue":"Digital Image Computing: Techniques and Applications (DICTA), 2016 International Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2016] Bichen Wu, Forrest Iandola, Peter H. Jin, and Kurt Keutzer. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. 2016.","title":"Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving","authors":"Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2016] Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. Improved relation classification by deep recurrent neural networks with data augmentation. arXiv preprint arXiv:1601.03651, 2016.","title":"Improved relation classification by deep recurrent neural networks with data augmentation","authors":"Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, Zhi Jin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}