{"type":"ScholarlyArticle","title":"Seeking Practical CDCL Insights from Theoretical SAT Benchmarks","authors":[{"name":"Jan Elffers","slug":"jan-elffers","given":"Jan","family":"Elffers","affiliation":"KTH Royal Institute of Technology"},{"name":"Jesús Giráldez-Cru","slug":"jesus-giraldez-cru","given":"Jesús","family":"Giráldez-Cru","affiliation":"KTH Royal Institute of Technology"},{"name":"Stephan Gocht","slug":"stephan-gocht","given":"Stephan","family":"Gocht","affiliation":"KTH Royal Institute of Technology"},{"name":"Jakob Nordström","slug":"jakob-nordstrom","given":"Jakob","family":"Nordström","affiliation":"KTH Royal Institute of Technology"},{"name":"Laurent Simon","slug":"laurent-simon","given":"Laurent","family":"Simon","affiliation":"Université de Bordeaux"}],"abstract":"Over the last decades Boolean satisfiability (SAT) solvers based on\r\nconflict-driven clause learning (CDCL) have developed to the point\r\nwhere they can handle formulas with millions of variables.  Yet a\r\ndeeper understanding of how these solvers can be so successful has\r\nremained elusive.  In this work we shed light on CDCL performance by\r\nusing theoretical benchmarks, which have the attractive features of\r\nbeing a) scalable, b) extremal with respect to different proof search\r\nparameters, and c) theoretically easy in the sense of having short\r\nproofs in the resolution proof system underlying CDCL. This allows for\r\na systematic study of solver heuristics and how efficiently they\r\nsearch for proofs.  We report results from extensive experiments on a\r\nwide range of benchmarks. Our findings include several examples where\r\ntheory predicts and explains CDCL behaviour, but also raise a number\r\nof intriguing questions for further study.","doi":"10.24963/ijcai.2018/181","url":"https://ijcai.org/proceedings/2018/181","pdf":"https://ijcai.org/proceedings/2018/0181.pdf","markdown":"https://ijcai.org/proceedings/2018/181.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/181","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Constraints and SAT","track":"Main track","number":181,"pages":{"first":1300,"last":1308},"keywords":["Constraints and SAT: SAT: Solvers and Tools","Constraints and SAT: SAT: Evaluation and Analysis","Constraints and SAT: SAT"],"videos":[],"references":[{"position":1,"raw":"[Alekhnovich and Razborov, 2008] M. Alekhnovich and A. A. Razborov. Resolution is not automatizable unless W[P] is tractable. SICOMP, 38(4):1347-1363, 2008.","title":"Resolution is not automatizable unless W[P] is tractable","authors":"M Alekhnovich, A A Razborov","year":"2008","venue":"SICOMP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alekhnovich et al., 2002] M. Alekhnovich, E. Ben-Sasson, A. A. Razborov, and A. Wigderson. Space complexity in propositional calculus. SICOMP, 31(4):1184-1211, 2002.","title":"Space complexity in propositional calculus","authors":"M Alekhnovich, E Ben-Sasson, A A Razborov, A Wigderson","year":"2002","venue":"SICOMP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alekhnovich et al., 2007] M. Alekhnovich, J. Johannsen, T. Pitassi, and A. Urquhart. An exponential separation between regular and general resolution. Theory Comp., 3(5):81-102, 2007.","title":"An exponential separation between regular and general resolution","authors":"M Alekhnovich, J Johannsen, T Pitassi, A Urquhart","year":"2007","venue":"Theory Comp","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Atserias and Dalmau, 2008] A. Atserias and V. Dalmau. A combinatorial characterization of resolution width. JCSS, 74(3):323-334, 2008.","title":"A combinatorial characterization of resolution width","authors":"A Atserias, V Dalmau","year":"2008","venue":"JCSS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Atserias et al., 2016] A. Atserias, M. Lauria, and J. Nord- ström. Narrow proofs be maximally long. ACM ToCL, 17(3):19:1-19:30, 2016.","title":"Narrow proofs be maximally long","authors":"A Atserias, M Lauria, J Nordström","year":"2016","venue":"ACM ToCL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Audemard and Simon, 2009] G. Audemard and L. Simon. Predict- ing learnt clauses quality in modern SAT solvers. In Proc. 21st Joint Conf. Artif. Intell. (IJCAI'09), pp. 399-404, 2009.","title":"Predicting learnt clauses quality in modern SAT solvers","authors":"G Audemard, L Simon","year":"2009","venue":"Proc. 21st Joint Conf. Artif. Intell. (IJCAI'09)","doi":null,"url":null,"internal_url":"/proceedings/2009/74"},{"position":7,"raw":"[Audemard and Simon, 2012] G. Audemard and L. Simon. Refin- ing restarts strategies for SAT and UNSAT. In Proc. 18th Conf. Principles Practice Constr. Prog. (CP'12), pp. 118-126, 2012.","title":"Refining restarts strategies for SAT and UNSAT","authors":"G Audemard, L Simon","year":"2012","venue":"Proc. 18th Conf. Principles Practice Constr. Prog. (CP'12)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bayardo Jr. and Schrag, 1997] R. J. Bayardo Jr. and R. Schrag. Using CSP look-back techniques to solve real-world SAT in- stances. In Proc. 14th National Conf. Artif. Intell. (AAAI'97), pp. 203-208, 1997.","title":"Using CSP look-back techniques to solve real-world SAT instances","authors":"R J Bayardo, R Schrag","year":"1997","venue":"Proc. 14th National Conf. Artif. Intell. (AAAI'97)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Beame et al., 2004] P. Beame, H. Kautz, and A. Sabharwal. To- wards understanding and harnessing the potential of clause learn- ing. JAIR, 22:319-351, 2004.","title":"Towards understanding and harnessing the potential of clause learning","authors":"P Beame, H Kautz, A Sabharwal","year":"2004","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Beame et al., 2016] P. Beame, C. Beck, and R. Impagliazzo. Time- space tradeoffs in resolution: Superpolynomial lower bounds for superlinear space. SICOMP, 45(4):1612-1645, 2016.","title":"Timespace tradeoffs in resolution: Superpolynomial lower bounds for superlinear space","authors":"P Beame, C Beck, R Impagliazzo","year":"2016","venue":"SICOMP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Beck et al., 2013] C. Beck, J. Nordström, and B. Tang. Some trade-off results for polynomial calculus. In Proc. 45th Symp. Theory Comp. (STOC'13), pp. 813-822, 2013.","title":"Some trade-off results for polynomial calculus","authors":"C Beck, J Nordström, B Tang","year":"2013","venue":"Proc. 45th Symp. Theory Comp. (STOC'13)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ben-Sasson and Nordström, 2008] E. Ben-Sasson and J. Nord- ström. Short proofs may be spacious: An optimal separation of space and length in resolution. In Proc. 49th Symp. Found. Comp. Sci. (FOCS'08), pp. 709-718, 2008.","title":"Short proofs may be spacious: An optimal separation of space and length in resolution","authors":"Ben-Sasson Nordström, ] E Ben-Sasson, J Nordström","year":"2008","venue":"Proc. 49th Symp. Found. Comp. Sci. (FOCS'08)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ben-Sasson and Nordström, 2011] E. Ben-Sasson and J. Nord- ström. Understanding space in proof complexity: Separations and trade-offs via substitutions. In Proc. 2nd Symp. Innovations Comp. Sci. (ICS'11), pp. 401-416, 2011.","title":"Understanding space in proof complexity: Separations and trade-offs via substitutions","authors":"Ben-Sasson Nordström, ] E Ben-Sasson, J Nordström","year":"2011","venue":"Proc. 2nd Symp. Innovations Comp. Sci. (ICS'11)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ben-Sasson and Wigderson, 2001] E. Ben-Sasson and A. Wigder- son. Short proofs are narrow-resolution made simple. J. ACM, 48(2):149-169, 2001.","title":"Short proofs are narrow-resolution made simple","authors":"; E Ben-Sasson And Wigderson, A Ben-Sasson, Wigderson","year":"2001","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ben-Sasson et al., 2004] E. Ben-Sasson, R. Impagliazzo, and A. Wigderson. Near optimal separation of tree-like and general resolution. Combinatorica, 24(4):585-603, 2004.","title":"Near optimal separation of tree-like and general resolution","authors":"E Ben-Sasson, R Impagliazzo, A Wigderson","year":"2004","venue":"Combinatorica","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ben-Sasson, 2009] E. Ben-Sasson. Size-space tradeoffs for reso- lution. SICOMP, 38(6):2511-2525, 2009.","title":"Size-space tradeoffs for resolution","authors":"E Ben-Sasson","year":"2009","venue":"SICOMP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Beyersdorff et al., 2013] O. Beyersdorff, N. Galesi, and M. Lau- ria. Parameterized complexity of DPLL search procedures. ACM ToCL, 14(3):20:1-20:21, 2013.","title":"Parameterized complexity of DPLL search procedures","authors":"O Beyersdorff, N Galesi, M Lauria","year":"2013","venue":"ACM ToCL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Biere and Fröhlich, 2015a] A. Biere and A. Fröhlich. Evaluating CDCL restart schemes. In Proc. Workshop on Pragmatics of SAT (PoS'15), 2015.","title":"Evaluating CDCL restart schemes","authors":"Fröhlich Biere, A Biere, A Fröhlich","year":"2015","venue":"Proc. Workshop on Pragmatics of SAT (PoS'15)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Biere and Fröhlich, 2015b] A. Biere and A. Fröhlich. Evaluating CDCL variable scoring schemes. In Proc. 18th Conf. Theory Appl. Sat. Testing (SAT'15), pp. 405-422, 2015.","title":"Evaluating CDCL variable scoring schemes","authors":"Fröhlich Biere, A Biere, A Fröhlich","year":"2015","venue":"Proc. 18th Conf. Theory Appl. Sat. Testing (SAT'15)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Biere et al., 2009] A. Biere, M. J. H. Heule, H. van Maaren, and T. Walsh, editors. Handbook of Satisfiability. 2009.","title":"Handbook of Satisfiability","authors":null,"year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Bonet and Galesi, 2001] M. L. Bonet and N. Galesi. Optimal- ity of size-width tradeoffs for resolution. Comp. Complex., 10(4):261-276, 2001.","title":"Optimality of size-width tradeoffs for resolution","authors":"M L Bonet, N Galesi","year":"2001","venue":"Comp. Complex","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Bonet et al., 2014] M. L. Bonet, S. Buss, and J. Johannsen. Im- proved separations of regular resolution from clause learning proof systems. JAIR, 49:669-703, 2014.","title":"Improved separations of regular resolution from clause learning proof systems","authors":"M L Bonet, S Buss, J Johannsen","year":"2014","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Buss and Kołodziejczyk, 2014] S. R. Buss and L. Kołodziejczyk. Small stone in pool. LMCS, 10(2):16:1-16:22, 2014.","title":"Small stone in pool","authors":"S R Buss, L Kołodziejczyk","year":"2014","venue":"LMCS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Chvátal and Szemerédi, 1988] V. Chvátal and E. Szemerédi. Many hard examples for resolution. J. ACM, 35(4):759-768, 1988.","title":"Many hard examples for resolution","authors":"V Chvátal, E Szemerédi","year":"1988","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Cook and Reckhow, 1979] S. A. Cook and R. Reckhow. The rel- ative efficiency of propositional proof systems. J. Symb. Logic, 44(1):36-50, 1979.","title":"The relative efficiency of propositional proof systems","authors":"S A Cook, R Reckhow","year":"1979","venue":"J. Symb. Logic","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Crawford and Auton, 1996] J. M. Crawford and L. D. Auton. Ex- perimental results on the crossover point in random 3-SAT. Artif. Intell., 81(1-2):31-57, 1996.","title":"Experimental results on the crossover point in random 3-SAT","authors":"J M Crawford, L D Auton","year":"1996","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Davis and Putnam, 1960] M. Davis and H. Putnam. A computing procedure for quantification theory. J. ACM, 7(3):201-215, 1960.","title":"A computing procedure for quantification theory","authors":"M Davis, H Putnam","year":"1960","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Davis et al., 1962] M. Davis, G. Logemann, and D. Loveland. A machine program for theorem proving. Comm. ACM, 5(7):394-397, 1962.","title":"A machine program for theorem proving","authors":"M Davis, G Logemann, D Loveland","year":"1962","venue":"Comm. ACM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Eén and Sörensson, 2004] N. Eén and N. Sörensson. An extensible SAT-solver. In 6th Conf. Theory Appl. Sat. Testing (SAT'03), Selected Revised Papers, pp. 502-518, 2004.","title":"An extensible SAT-solver","authors":"N Eén, N Sörensson","year":"2004","venue":"6th Conf. Theory Appl. Sat. Testing (SAT'03), Selected Revised Papers","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Esteban and Torán, 2001] J. L. Esteban and J. Torán. Space bounds for resolution. Inform. Comp., 171(1):84-97, 2001.","title":"Space bounds for resolution","authors":"J L Esteban, J Torán","year":"2001","venue":"Inform. Comp","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Huang, 2007] J. Huang. The effect of restarts on the efficiency of clause learning. In Proc. 20th Joint Conf. Artif. Intell. (IJCAI'07), pp. 2318-2323, 2007.","title":"The effect of restarts on the efficiency of clause learning","authors":"J Huang","year":"2007","venue":"Proc. 20th Joint Conf. Artif. Intell. (IJCAI'07)","doi":null,"url":null,"internal_url":"/proceedings/2007/373"},{"position":32,"raw":"[Järvisalo et al., 2012] M. Järvisalo, A. Matsliah, J. Nordström, and S. Živný. Relating proof complexity measures and practical hard- ness of SAT. In Proc. 18th Conf. Principles Practice Constr. Prog. (CP'12), pp. 316-331, 2012.","title":"Relating proof complexity measures and practical hardness of SAT","authors":"M Järvisalo, A Matsliah, J Nordström, S Živný","year":"2012","venue":"Proc. 18th Conf. Principles Practice Constr. Prog. (CP'12","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Katebi et al., 2011] H. Katebi, K. A. Sakallah, and J. P. Marques- Silva. Empirical study of the anatomy of modern SAT solvers. In Proc. 14th Conf. Theory Appl. Sat. Testing (SAT'11), pp. 343-356, 2011.","title":"Empirical study of the anatomy of modern SAT solvers","authors":"H Katebi, K A Sakallah, J P Marques-Silva","year":"2011","venue":"Proc. 14th Conf. Theory Appl. Sat. Testing (SAT'11)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lauria et al., 2017] M. Lauria, J. Elffers, J. Nordström, and M. Vinyals. CNFgen: A generator of crafted benchmarks. In Proc. 20th Conf. Theory Appl. Sat. Testing (SAT'17), pp. 464-473, 2017.","title":"CNFgen: A generator of crafted benchmarks","authors":"M Lauria, J Elffers, J Nordström, M Vinyals","year":"2017","venue":"Proc. 20th Conf. Theory Appl. Sat. Testing (SAT'17)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Liang et al., 2016] J. H. Liang, V. Ganesh, P. Poupart, and K. Czar- necki. Learning rate based branching heuristic for SAT solvers. In Proc. 19th Conf. Theory Appl. Sat. Testing (SAT'16), pp. 123-140, 2016.","title":"Learning rate based branching heuristic for SAT solvers","authors":"J H Liang, V Ganesh, P Poupart, K Czarnecki","year":"2016","venue":"Proc. 19th Conf. Theory Appl. Sat. Testing (SAT'16)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Lynce and Marques-Silva, 2002] I. Lynce and J. P. Marques-Silva. Building state-of-the-art SAT solvers. In Proc. 15th European Conf. Artif. Intell. (ECAI'02), pp. 166-170. 2002.","title":"Building state-of-the-art SAT solvers","authors":"Marques-Silva ; I Lynce, J P Lynce, Marques-Silva","year":"2002","venue":"Proc. 15th European Conf. Artif. Intell. (ECAI'02)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Markström, 2006] K. Markström. Locality and hard SAT- instances. JSAT, 2(1-4):221-227, 2006.","title":"Locality and hard SATinstances","authors":"K Markström","year":"2006","venue":"JSAT","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Marques-Silva and Sakallah, 1999] J. P. Marques-Silva and K. A. Sakallah. GRASP: A search algorithm for propositional satisfia- bility. IEEE Trans. Comp., 48(5):506-521, 1999.","title":"GRASP: A search algorithm for propositional satisfiability","authors":"J P Marques-Silva, K A Sakallah","year":"1999","venue":"IEEE Trans. Comp","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Mikša and Nordström, 2014] M. Mikša and J. Nordström. Long proofs of (seemingly) simple formulas. In Proc. 17th Conf. The- ory Appl. Sat. Testing (SAT'14), pp. 121-137, 2014.","title":"Long proofs of (seemingly) simple formulas","authors":"M Mikša, J Nordström","year":"2014","venue":"Proc. 17th Conf. Theory Appl. Sat. Testing (SAT'14)","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Moskewicz et al., 2001] M. W. Moskewicz, C. F. Madigan, Y. Zhao, L. Zhang, and S. Malik. Chaff: Engineering an efficient SAT solver. In Proc. 38th Design Automation Conf. (DAC'01), pp. 530-535, 2001.","title":"Chaff: Engineering an efficient SAT solver","authors":"M W Moskewicz, C F Madigan, Y Zhao, L Zhang, S Malik","year":"2001","venue":"Proc. 38th Design Automation Conf. (DAC'01)","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Nordström, 2015] J. Nordström. On the interplay between proof complexity and SAT solving. ACM SIGLOG News, 2(3):19-44, 2015.","title":"On the interplay between proof complexity and SAT solving","authors":"J Nordström","year":"2015","venue":"ACM SIGLOG News","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Petke and Jeavons, 2009] J. Petke and P. Jeavons. Tractable bench- marks for constraint programming. Technical Report RR-09-07, Oxford University, 2009.","title":"Tractable benchmarks for constraint programming","authors":"Petke, ] J Jeavons, P Petke, Jeavons","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Pipatsrisawat and Darwiche, 2007] K. Pipatsrisawat and A. Dar- wiche. A lightweight component caching scheme for satisfi- ability solvers. In Proc. 10th Conf. Theory Appl. Sat. Testing (SAT'07), pp. 294-299, 2007.","title":"A lightweight component caching scheme for satisfiability solvers","authors":"K Pipatsrisawat, A Darwiche","year":"2007","venue":"Proc. 10th Conf. Theory Appl. Sat. Testing (SAT'07)","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Pipatsrisawat and Darwiche, 2011] K. Pipatsrisawat and A. Dar- wiche. On the power of clause-learning SAT solvers as resolution engines. Artif. Intell., 175(2):512-525, 2011.","title":"On the power of clause-learning SAT solvers as resolution engines","authors":"K Pipatsrisawat, A Darwiche","year":"2011","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Selman et al., 1992] B. Selman, H. J. Levesque, and D. G. Mitchell. A new method for solving hard satisfiability prob- lems. In Proc. 10th National Conf. Artif. Intell. (AAAI'92), pp. 440-446, 1992.","title":"A new method for solving hard satisfiability problems","authors":"B Selman, H J Levesque, D G Mitchell","year":"1992","venue":"Proc. 10th National Conf. Artif. Intell. (AAAI'92)","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Stålmarck, 1996] G. Stålmarck. Short resolution proofs for a se- quence of tricky formulas. Acta Inform., 33(3):277-280, 1996.","title":"Short resolution proofs for a sequence of tricky formulas","authors":"G Stålmarck","year":"1996","venue":"Acta Inform","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Thapen, 2016] N. Thapen. A trade-off between length and width in resolution. Theory Comp., 12(5):1-14, 2016.","title":"A trade-off between length and width in resolution","authors":"N Thapen","year":"2016","venue":"Theory Comp","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Urquhart, 1987] A. Urquhart. Hard examples for resolution. J. ACM, 34(1):209-219, 1987.","title":"Hard examples for resolution","authors":"A Urquhart","year":"1987","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Van Gelder and Spence, 2010] A. Van Gelder and I. Spence. Zero- one designs produce small hard SAT instances. In Proc. 13th Conf. Theory Appl. Sat. Testing (SAT'10), pp. 388-397, 2010.","title":"Zeroone designs produce small hard SAT instances","authors":"Van Gelder, ; A Spence, I Van Gelder, Spence","year":"2010","venue":"Proc. 13th Conf. Theory Appl. Sat. Testing (SAT'10)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}