{"type":"ScholarlyArticle","title":"Correct and Optimal: The Regular Expression Inference Challenge","authors":[{"name":"Mojtaba Valizadeh","slug":"mojtaba-valizadeh","given":"Mojtaba","family":"Valizadeh","affiliation":"University of Sussex"},{"name":"Philip John Gorinski","slug":"philip-john-gorinski","given":"Philip John","family":"Gorinski","affiliation":"Huawei Noah's Ark Lab , London"},{"name":"Ignacio Iacobacci","slug":"ignacio-iacobacci","given":"Ignacio","family":"Iacobacci","affiliation":"Huawei Noah's Ark Lab , London"},{"name":"Martin Berger","slug":"martin-berger","given":"Martin","family":"Berger","affiliation":"University of Sussex; Montanarius Ltd"}],"abstract":"We propose regular expression inference (REI) as a challenge for code/language modelling, and the wider machine learning community. REI is a supervised machine learning (ML) and program optimisation task, and poses the problem of finding minimal regular expressions from examples: Given two finite sets of strings P and N and a cost function cost(·), the task is to generate an expression r that accepts all strings in P and rejects all strings in N , while no other such expression r′ exists with cost(r′) < cost(r).\r\nREI has advantages as a challenge problem: (i) regular expressions are well-known, widely used, and a natural idealisation of code; (ii) REI’s asymptotic worst-case complexity is well understood; (iii) REI has a small number of easy to understand parameters (e.g. P or N cardinality, string lengths of examples, or the cost function); this lets us easily finetune REI-hardness; (iv) REI, with its emphasis on optimisation, is an unsolved problem for deep learning based ML.\r\nRecently, an REI solver was implemented on GPUs, using program synthesis techniques. This enabled, for the first time, fast generation of minimal regular expressions for complex REI instances. Building on this advance, we generate and publish the first large-scale datasets for REI, and devise and evaluate several initial heuristic and machine learning baselines.\r\nWe invite the community to participate and explore ML methods that learn to solve REI problems. We believe that progress in REI directly translates to progress in code/language modelling.","doi":"10.24963/ijcai.2024/717","url":"https://ijcai.org/proceedings/2024/717","pdf":"https://ijcai.org/proceedings/2024/0717.pdf","markdown":"https://ijcai.org/proceedings/2024/717.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/717","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":717,"pages":{"first":6486,"last":6494},"keywords":["Machine Learning: ML: Other","Natural Language Processing: NLP: Resources and evaluation","Machine Learning: ML: Applications","Natural Language Processing: NLP: Other"],"videos":[],"references":[{"position":1,"raw":"[Angluin, 1978] Dana Angluin. On the complexity of mini- mum inference of regular sets. Information and Control, 39(3):337-350, 1978.","title":"On the complexity of minimum inference of regular sets","authors":"Dana Angluin","year":"1978","venue":"Information and Control","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Angluin, 1987] Dana Angluin. Learning Regular Sets from Queries and Counterexamples. Inf. Comput., 75(2):87-106, November 1987.","title":"Dana Angluin. Learning Regular Sets from Queries and Counterexamples","authors":"Angluin","year":"1987","venue":"Inf. Comput","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Austin et al., 2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.","title":"Program synthesis with large language models","authors":"Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP 2015, pages 632-642, Lisbon, Portugal, September 2015.","title":"A large annotated corpus for learning natural language inference","authors":"R Samuel, Gabor Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":"Proceedings of EMNLP 2015","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Budzianowski et al., 2018] Paweł Budzianowski, Tsung- Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task- oriented dialogue modelling. In Proceedings of EMNLP 2018, pages 5016-5026, Brussels, Belgium, October- November 2018.","title":"MultiWOZa large-scale multi-domain Wizard-of-Oz dataset for taskoriented dialogue modelling","authors":"Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Milica Osman Ramadan, Gašić","year":"2018","venue":"Proceedings of EMNLP 2018","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.","title":"Evaluating large language models trained on code","authors":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2023] Qiaochu Chen, Arko Banerjee, C ¸agatay Demiralp, Greg Durrett, and Is ¸ıl Dillig. Data extraction via semantic regular expression synthesis. Proc. ACM Pro- gram. Lang., 7(OOPSLA2), oct 2023.","title":"Data extraction via semantic regular expression synthesis","authors":"Qiaochu Chen, Arko Banerjee, C ¸agatay Demiralp, Greg Durrett, Is ¸ıl, Dillig","year":"2023","venue":"Proc. ACM Program. Lang","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chomsky, 1956] Noam Chomsky. Three models for the de- scription of language. IRE Transactions on information theory, 2(3):113-124, 1956.","title":"Noam Chomsky. Three models for the description of language","authors":"Chomsky","year":"1956","venue":"IRE Transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Colin et al., 2016] Emilie Colin, Claire Gardent, Yassine M'rabet, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG Challenge: Generating Text from DBPedia Data. In Proceedings of the 9th international natural lan- guage generation conference, pages 163-167, 2016.","title":"The WebNLG Challenge: Generating Text from DBPedia Data","authors":"Emilie Colin, Claire Gardent, M' Yassine, Shashi Rabet, Laura Narayan, Perez-Beltrachini","year":"2016","venue":"Proceedings of the 9th international natural language generation conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Courcelle and Engelfriet, 2012] Professor Bruno Courcelle and Dr Joost Engelfriet. Graph Structure and Monadic Second-Order Logic: A Language-Theoretic Approach. Cambridge University Press, USA, 1st edition, 2012.","title":"Graph Structure and Monadic Second-Order Logic: A Language-Theoretic Approach","authors":"Engelfriet ; Courcelle, Bruno Professor, Dr Courcelle, Engelfriet Joost","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Delétang et al., 2023] Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, El- liot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In 11th International Conference on Learning Representations, 2023.","title":"Neural networks and the chomsky hierarchy","authors":"Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Kevin Li, Elliot Wenliang, Chris Catt, Marcus Cundy, Shane Hutter, Joel Legg, Pedro A Veness, Ortega","year":"2023","venue":"11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[D'Ulizia et al., 2011] Arianna D'Ulizia, Fernando Ferri, and Patrizia Grifoni. A survey of grammatical inference methods for natural language learning. Artificial Intelli- gence Review, 36:1-27, 2011.","title":"A survey of grammatical inference methods for natural language learning","authors":"Arianna D' Ulizia, Fernando Ferri, Patrizia Grifoni","year":"2011","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Firoiu et al., 1998] Laura Firoiu, Tim Oates, and Paul R. Cohen. Learning Deterministic Finite Automaton with a Recurrent Neural Network. In Proceedings of the ICGI 1998, ICGI '98, page 90-101, Berlin, Heidelberg, 1998. Springer-Verlag. [Github, 2022] Github. Your AI pair programmer. https:// github.com/features/copilot, 2022. Blog post accessed 8 June 2023.","title":"Learning Deterministic Finite Automaton with a Recurrent Neural Network","authors":"Laura Firoiu, Tim Oates, Paul R Cohen","year":"1998","venue":"Proceedings of the ICGI 1998, ICGI '98","doi":null,"url":"https://github.com/features/copilot,2022","internal_url":null},{"position":15,"raw":"[Gold, 1967] E. Mark Gold. Language identification in the limit. Information and Control, 10(5):447-474, 1967.","title":"Language identification in the limit","authors":"E Mark Gold","year":"1967","venue":"Information and Control","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gold, 1978] E. Mark Gold. Complexity of automaton iden- tification from given data. Information and Control, 37(3):302-320, 1978.","title":"Complexity of automaton identification from given data","authors":"E Mark Gold","year":"1978","venue":"Information and Control","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Guan et al., 2021] Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. OpenMEVA: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 6394-6407, Online, August 2021.","title":"OpenMEVA: A benchmark for evaluating open-ended story generation metrics","authors":"Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, Minlie Huang","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hendrycks et al., 2021] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.","title":"Measuring coding challenge competence with apps","authors":"Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob Steinhardt","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Holtzman et al., 2019] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.","title":"The curious case of neural text degeneration","authors":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hopcroft et al., 2006] John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to Automata Theory, Languages, and Computation. Addison-Wesley, 2006.","title":"Introduction to Automata Theory, Languages, and Computation","authors":"John E Hopcroft, Rajeev Motwani, Jeffrey D Ullman","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kearns and Valiant, 1994] Michael Kearns and Leslie Valiant. Cryptographic Limitations on Learning Boolean Formulae and Finite Automata. J. ACM, 41(1):67-95, jan 1994.","title":"Cryptographic Limitations on Learning Boolean Formulae and Finite Automata","authors":"Michael Kearns, Leslie Valiant","year":"1994","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kleene, 1956] S. C. Kleene. Representation of events in nerve nets and finite automata. In Automata Studies, pages 3-41. Princeton University Press, Princeton, NJ, 1956.","title":"Representation of events in nerve nets and finite automata","authors":"S C Kleene","year":"1956","venue":"Automata Studies","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2021] Yeting Li, Shuaimin Li, Zhiwu Xu, Jialun Cao, Zixuan Chen, Yun Hu, Haiming Chen, and Shing-Chi Cheung. TransRegex: Multi-Modal Regular Expression Synthesis by Generate-and-Repair. In Proceedings of the 43rd International Conference on Software Engineering, ICSE '21, page 1210-1222. IEEE Press, 2021.","title":"TransRegex: Multi-Modal Regular Expression Synthesis by Generate-and-Repair","authors":"Yeting Li, Shuaimin Li, Zhiwu Xu, Jialun Cao, Zixuan Chen, Yun Hu, Haiming Chen, Shing-Chi Cheung","year":"2021","venue":"Proceedings of the 43rd International Conference on Software Engineering, ICSE '21","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2023] Raymond Li, Loubna Ben Allal, Yang- tian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Ja- son Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jen- nifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robin- son, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bah- danau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023.","title":"Starcoder: may the source be with you! Transactions on Machine Learning Research","authors":"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Sankalp Siva, Dmitry Patel, Marco Abulkhanov, Manan Zocca, Zhihan Dey, Nour Zhang, Urvashi Fahmy, Wenhao Bhattacharyya, Swayam Yu, Sasha Singh, Paulo Luccioni, Maxim Villegas, Fedor Kunakov, Manuel Zhdanov, Tony Romero, Nadav Lee, Jennifer Timor, Claire Ding, Hailey Schlesinger, Jan Schoelkopf, Tri Ebert, Mayank Dao, Alex Mishra, Jennifer Gu, Carolyn Jane Robinson, Brendan Anderson, Danish Dolan-Gavitt, Siva Contractor, Daniel Reddy, Dzmitry Fried, Yacine Bahdanau, Carlos Jernite, Sean Muñoz Ferrandis, Thomas Hughes, Arjun Wolf, Leandro Guha, Harm Von Werra, De Vries","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Locascio et al., 2016] Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay. Neural generation of regular expressions from natural language with minimal domain knowledge. In Proceedings of EMNLP 2016, pages 1918-1923, Austin, Texas, November 2016.","title":"Neural generation of regular expressions from natural language with minimal domain knowledge","authors":"Nicholas Locascio, Karthik Narasimhan, Eduardo Deleon, Nate Kushman, Regina Barzilay","year":"2016","venue":"Proceedings of EMNLP 2016","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Muralidaran et al., 2021] Vigneshwaran Muralidaran, Irena Spasić, and Dawn Knight. A systematic review of unsu- pervised approaches to grammar induction. Natural Lan- guage Engineering, 27(6):647-689, 2021.","title":"Vigneshwaran Muralidaran, Irena Spasić, and Dawn Knight. A systematic review of unsupervised approaches to grammar induction","authors":"Muralidaran","year":"2021","venue":"Natural Language Engineering","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nie et al., 2020] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885- 4901, Online, July 2020.","title":"Adversarial NLI: A new benchmark for natural language understanding","authors":"Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[OpenAI, 2022] OpenAI. Introducing ChatGPT. https: //openai.com/blog/chatgpt, 2022. Blog post published November 30, 2022.","title":"OpenAI. Introducing ChatGPT","authors":"Openai","year":"2022","venue":null,"doi":null,"url":"https://openai.com/blog/chatgpt,2022","internal_url":null},{"position":29,"raw":"[Park et al., 2019] Jun-U Park, Sang-Ki Ko, Marco Cognetta, and Yo-Sub Han. SoftRegex: Generating Regex from Natural Language Descriptions using Softened Regex Equivalence. In Proceedings of EMNLP- IJCNLP 2019, pages 6425-6431, Hong Kong, China, November 2019.","title":"SoftRegex: Generating Regex from Natural Language Descriptions using Softened Regex Equivalence","authors":"Park","year":"2019","venue":"Proceedings of EMNLP-IJCNLP 2019","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Pavao et al., 2022] Adrien Pavao, Isabelle Guyon, Anne- Catherine Letournel, Xavier Baró, Hugo Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. CodaLab Com- petitions: An open source platform to organize scientific challenges. Technical report, 2022.","title":"CodaLab Competitions: An open source platform to organize scientific challenges","authors":"Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Xavier Baró, Hugo Escalante, Sergio Escalera, Tyler Thomas, Zhen Xu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pitt and Warmuth, 1993] Leonard Pitt and Manfred K. War- muth. The Minimum Consistent DFA Problem Can- not Be Approximated within Any Polynomial. J. ACM, 40(1):95-142, jan 1993.","title":"Warmuth. The Minimum Consistent DFA Problem Cannot Be Approximated within Any Polynomial","authors":"Warmuth Pitt, Leonard Pitt, Manfred K","year":"1993","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tenney et al., 2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593-4601, 2019.","title":"BERT Rediscovers the Classical NLP Pipeline","authors":"Tenney","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tunstall et al., 2023] Lewis Tunstall, Nathan Lambert, Nazneen Rajani, Edward Beeching, Teven Le Scao, Leandro von Werra, Sheon Han, Philipp Schmid, and Alexander Rush. Creating a coding assis- tant with starcoder. Hugging Face Blog, 2023. https://huggingface.co/blog/starchat. [Turing and others, 1936]","title":"Creating a coding assistant with starcoder","authors":"Lewis Tunstall, Nathan Lambert, Nazneen Rajani, Edward Beeching, Le Teven, Leandro Scao, Sheon Von Werra, Philipp Han, Alexander Schmid, Rush","year":"1936","venue":"Hugging Face Blog","doi":null,"url":"https://huggingface.co/blog/starchat.[Turingandothers","internal_url":null},{"position":35,"raw":"Alan Mathison Turing et al. On computable numbers, with an application to the entschei- dungsproblem. J. of Math, 58(345-363):5, 1936.","title":"On computable numbers, with an application to the entscheidungsproblem","authors":"Alan Mathison, Turing","year":"1936","venue":"J. of Math","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vaandrager, 2017] Frits Vaandrager. Model Learning. Com- mun. ACM, 60(2):86-95, jan 2017.","title":"Model Learning. Commun","authors":"Frits Vaandrager","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Valizadeh and Berger, 2023] Mojtaba Valizadeh and Martin Berger. Search-Based Regular Expression Inference on a GPU. Proc. ACM Program. Lang., 7(PLDI), jun 2023. Draft available at https://arxiv.org/abs/2305.18575, imple- mentation: https://github.com/MojtabaValizadeh/paresy.","title":"Search-Based Regular Expression Inference on a GPU","authors":"Berger Valizadeh, Mojtaba Valizadeh, Martin Berger","year":"2023","venue":"Proc. ACM Program. Lang","doi":null,"url":"https://arxiv.org/abs/2305.18575,imple-mentation:https://github.com/MojtabaValizadeh/paresy","internal_url":null},{"position":38,"raw":"[van der Poel et al., 2023] Sam van der Poel, Dakotah Lam- bert, Kalina Kostyszyn, Tiantian Gao, Rahul Verma, Derek Andersen, Joanne Chau, Emily Peterson, Cody St. Clair, Paul Fodor, Chihiro Shibata, and Jeffrey Heinz. ML- RegTest: A Benchmark for the Machine Learning of Reg- ular Languages, 2023.","title":"ML-RegTest: A Benchmark for the Machine Learning of Regular Languages","authors":"Poel Van Der","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stick- ier benchmark for general-purpose language understand- ing systems. Advances in neural information processing systems, 32, 2019.","title":"Superglue: A stickier benchmark for general-purpose language understanding systems","authors":"Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Williams et al., 2018] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel Bowman","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yuie et al., 2021] Wang Yuie, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware Uni- fied Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. In EMNLP, 2021.","title":"CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation","authors":"Wang Yuie, Weishi Wang, Shafiq Joty, Steven C H Hoi","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhong et al., 2018] Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, Tao Xie, Jian-Guang Lou, Ting Liu, and Dong- mei Zhang. SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications. In Proceedings of EMNLP 2018, pages 1608-1618, Brussels, Belgium, October-November 2018.","title":"SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications","authors":"Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, Tao Xie, Jian-Guang Lou, Ting Liu, Dongmei Zhang","year":"2018","venue":"Proceedings of EMNLP 2018","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}