{"type":"ScholarlyArticle","title":"Counterexample-Driven Genetic Programming: Stochastic Synthesis of Provably Correct Programs","authors":[{"name":"Krzysztof Krawiec","slug":"krzysztof-krawiec","given":"Krzysztof","family":"Krawiec","affiliation":"Poznan University of Technology , Poland"},{"name":"Iwo Błądek","slug":"iwo-b-adek","given":"Iwo","family":"Błądek","affiliation":"Poznan University of Technology , Poland"},{"name":"Jerry Swan","slug":"jerry-swan","given":"Jerry","family":"Swan","affiliation":"University of York , UK"},{"name":"John H. Drake","slug":"john-h-drake","given":"John H.","family":"Drake","affiliation":"Queen Mary University of London , UK"}],"abstract":"Genetic programming is an effective technique for inductive synthesis of programs from tests, i.e. training examples of desired input-output behavior. Programs synthesized in this way are not guaranteed to generalize beyond the training set, which is unacceptable in many applications. We present Counterexample-Driven Genetic Programming (CDGP) that employs evolutionary search to synthesize provably correct programs from formal specifications. CDGP employs a Satisfiability Modulo Theories (SMT) solver to formally verify programs in the evaluation phase. A failed verification produces counterexamples that are in turn used to calculate fitness and thereby drive the search process. When compared with a range of approaches on a suite of state-of-the-art specification-based synthesis benchmarks, CDGP systematically outperforms them, typically synthesizing correct programs faster and using fewer tests.","doi":"10.24963/ijcai.2018/742","url":"https://ijcai.org/proceedings/2018/742","pdf":"https://ijcai.org/proceedings/2018/0742.pdf","markdown":"https://ijcai.org/proceedings/2018/742.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/742","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":742,"pages":{"first":5304,"last":5308},"keywords":["Heuristic Search and Game Playing: Meta-Reasoning and Meta-heuristics","Heuristic Search and Game Playing: Heuristic Search","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis"],"videos":[],"references":[{"position":1,"raw":"[Almeida et al., 2011] José Bacelar Almeida, Maria João Frade, Jorge Sousa Pinto, and Simão Melo de Sousa. An Overview of Formal Methods Tools and Techniques, pages 15-44. Springer London, London, 2011.","title":"An Overview of Formal Methods Tools and Techniques","authors":"Almeida","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alur et al., 2015] Rajeev Alur, Dana Fisman, Rishabh Singh, and Armando Solar-Lezama. Results and analy- sis of sygus-comp'15. In Proceedings Fourth Workshop on Synthesis, SYNT 2015, San Francisco, CA, USA, 18th July 2015., pages 3-26, 2015.","title":"Results and analysis of sygus-comp'15","authors":"Rajeev Alur, Dana Fisman, Rishabh Singh, Armando Solar-Lezama","year":"2015","venue":"Proceedings Fourth Workshop on Synthesis, SYNT 2015","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arcuri and Yao, 2007] Andrea Arcuri and Xin Yao. Coe- volving programs and unit tests from their specification. In Proceedings of the Twenty-second IEEE/ACM Interna- tional Conference on Automated Software Engineering, ASE '07, pages 397-400, New York, NY, USA, 2007. ACM.","title":"Coevolving programs and unit tests from their specification","authors":"Andrea Arcuri, Xin Yao","year":"2007","venue":"Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering, ASE '07","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barrett et al., 2015] Clark Barrett, Pascal Fontaine, and Ce- sare Tinelli. The SMT-LIB Standard: Version 2.5. Techni- cal report, Department of Computer Science, The Univer- sity of Iowa, 2015. Available at www.SMT-LIB.org.","title":"The SMT-LIB Standard: Version 2.5","authors":"Clark Barrett, Pascal Fontaine, Cesare Tinelli","year":"2015","venue":null,"doi":null,"url":"www.","internal_url":null},{"position":5,"raw":"[Boca et al., 2009] Paul P. Boca, Jonathan P. Bowen, and Jawed I Siddiqi. Formal Methods: State of the Art and New Directions. Springer Publishing Company, Incorpo- rated, 1st edition, 2009.","title":"Formal Methods: State of the Art and New Directions","authors":"Paul P Boca, Jonathan P Bowen, Jawed I Siddiqi","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bł ądek and Krawiec, 2017] Iwo Bł ądek and Krzysztof Krawiec. Evolutionary Program Sketching, pages 3-18. Springer International Publishing, Cham, 2017.","title":"Iwo Bł ądek and Krzysztof Krawiec","authors":"Bł Ądek, Krawiec","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cohen, 1994] B Cohen. A Brief History of Formal Meth- ods. Formal Aspects of Computing, 1(3), 1994.","title":"A Brief History of Formal Methods","authors":"Cohen","year":"1994","venue":"Formal Aspects of Computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[de Moura and Bjørner, 2008] Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient SMT solver. In C. Ramakrishnan and Jakob Rehof, editors, Tools and Algorithms for the Construction and Analysis of Systems, volume 4963 of Lecture Notes in Computer Science, chapter 24, pages 337-340. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2008.","title":"Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient SMT solver","authors":"De Moura, Bjørner","year":"2008","venue":"Tools and Algorithms for the Construction and Analysis of Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gulwani et al., 2010] Sumit Gulwani, Susmit Jha, Ashish Tiwari, and Ramarathnam Venkatesan. Component based synthesis applied to bitvector programs. Technical Report MSR-TR-2010-12, February 2010.","title":"Component based synthesis applied to bitvector programs","authors":"Sumit Gulwani, Susmit Jha, Ashish Tiwari, Ramarathnam Venkatesan","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Helmuth et al., 2015] Thomas Helmuth, Lee Spector, and James Matheson. Solving uncompromising problems with lexicase selection. IEEE Transactions on Evolutionary Computation, 19(5):630-643, October 2015.","title":"Solving uncompromising problems with lexicase selection","authors":"Thomas Helmuth, Lee Spector, James Matheson","year":"2015","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hofmann, 2010] Martin Hofmann. Igor II -an analytical inductive functional programming system. In In Pro- ceedings of the 2010 ACM SIGPLAN Workshop on Par- tial Evaluation and Program Manipulation, pages 29-32, 2010.","title":"Igor II -an analytical inductive functional programming system","authors":"Martin Hofmann","year":"2010","venue":"Proceedings of the 2010 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jha et al., 2010] Susmit Jha, Sumit Gulwani, Sanjit A. Se- shia, and Ashish Tiwari. Oracle-guided component-based program synthesis. In 29th International Conference on Software Engineering (ICSE '10), pages 215-224, May 2010.","title":"Oracle-guided component-based program synthesis","authors":"Susmit Jha, Sumit Gulwani, A Sanjit, Ashish Seshia, Tiwari","year":"2010","venue":"29th International Conference on Software Engineering (ICSE '10)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson, 2007] Colin Johnson. Genetic programming with fitness based on model checking. In Marc Ebner, Michael O'Neill, Anikó Ekárt, Leonardo Vanneschi, and Anna Is- abel Esparcia-Alcázar, editors, Proceedings of the 10th European Conference on Genetic Programming, volume 4445 of Lecture Notes in Computer Science, pages 114- 124, Valencia, Spain, 11-13 April 2007. Springer.","title":"Genetic programming with fitness based on model checking","authors":"Colin Johnson","year":"2007","venue":"Proceedings of the 10th European Conference on Genetic Programming","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Katayama, 2012] Susumu Katayama. An analytical in- ductive functional programming system that avoids unin- tended programs. In Proceedings of the ACM SIGPLAN 2012 Workshop on Partial Evaluation and Program Ma- nipulation, PEPM '12, pages 43-52, New York, NY, USA, 2012. ACM.","title":"An analytical inductive functional programming system that avoids unintended programs","authors":"Susumu Katayama","year":"2012","venue":"Proceedings of the ACM SIGPLAN 2012 Workshop on Partial Evaluation and Program Manipulation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katz and Peled, 2008] Gal Katz and Doron Peled. Genetic Programming and Model Checking: Synthesizing New Mutual Exclusion Algorithms, pages 33-47. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.","title":"Genetic Programming and Model Checking: Synthesizing New Mutual Exclusion Algorithms","authors":"Peled Katz, Gal Katz, Doron Peled","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Katz and Peled, 2016] Gal Katz and Doron Peled. Syn- thesizing, correcting and improving code, using model checking-based genetic programming. International Jour- nal on Software Tools for Technology Transfer, pages 1- 16, 2016.","title":"Synthesizing, correcting and improving code, using model checking-based genetic programming","authors":"Peled Katz, Gal Katz, Doron Peled","year":"2016","venue":"International Journal on Software Tools for Technology Transfer","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krawiec et al., 2017] Krzysztof Krawiec, Iwo Bł ądek, and Jerry Swan. Counterexample-driven genetic program- ming. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '17, pages 953-960, New York, NY, USA, 2017. ACM.","title":"Counterexample-driven genetic programming","authors":"Krzysztof Krawiec, Iwo Bł, Jerry Swan","year":"2017","venue":"Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '17","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krawiec, 2015] Krzysztof Krawiec. Behavioral Program Synthesis with Genetic Programming, volume 618 of Stud- ies in Computational Intelligence. Springer International Publishing, 2015.","title":"Behavioral Program Synthesis with Genetic Programming","authors":"Krzysztof Krawiec","year":"2015","venue":"Studies in Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Muggleton, 1994] Stephen Muggleton. Inductive Logic Programming: Derivations, successes and shortcomings. SIGART Bull., 5(1):5-11, January 1994.","title":"Inductive Logic Programming: Derivations, successes and shortcomings","authors":"Stephen Muggleton","year":"1994","venue":"SIGART Bull","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Solar-Lezama et al., 2006] Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. Combinatorial sketching for finite programs. SIGPLAN Not., 41(11):404-415, October 2006.","title":"Combinatorial sketching for finite programs","authors":"Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, Vijay Saraswat","year":"2006","venue":"SIGPLAN Not","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2010] Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. From program verification to pro- gram synthesis. In Proceedings of the 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Program- ming Languages, POPL '10, pages 313-326, New York, NY, USA, 2010. ACM.","title":"From program verification to program synthesis","authors":"Saurabh Srivastava, Sumit Gulwani, Jeffrey S Foster","year":"2010","venue":"Proceedings of the 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '10","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}