{"type":"ScholarlyArticle","title":"Diagnosing Software Faults Using Multiverse Analysis","authors":[{"name":"Prantik Chatterjee","slug":"prantik-chatterjee","given":"Prantik","family":"Chatterjee","affiliation":"Indian Institute of Technology , Kanpur , India"},{"name":"Abhijit Chatterjee","slug":"abhijit-chatterjee","given":"Abhijit","family":"Chatterjee","affiliation":"Indian Institute of Technology , Kanpur , India"},{"name":"Jose Campos","slug":"jose-campos","given":"Jose","family":"Campos","affiliation":"LASIGE , Faculdade de Ciências , University of Lisbon , Portugal"},{"name":"Rui Abreu","slug":"rui-abreu","given":"Rui","family":"Abreu","affiliation":"INESC-ID and IST , University of Lisbon , Portugal"},{"name":"Subhajit Roy","slug":"subhajit-roy","given":"Subhajit","family":"Roy","affiliation":"Indian Institute of Technology , Kanpur , India"}],"abstract":"Spectrum-based Fault Localization (SFL) approaches aim to efficiently localize faulty components from examining program behavior. This is done by collecting the execution patterns of various\r\ncombinations of components and the corresponding outcomes into a spectrum. Efficient fault localization depends heavily on the quality of the spectra. Previous approaches, including the current\r\nstate-of-the-art Density- Diversity-Uniqueness (DDU) approach, attempt to generate “good” test-suites by improving certain structural properties of the spectra. In this work, we propose a different\r\napproach, Multiverse Analysis, that considers multiple hypothetical universes, each corresponding to a scenario where one of the components is assumed to be faulty, to generate a spectrum that\r\nattempts to reduce the expected worst-case wasted effort over all the universes. Our experiments show that the Multiverse Analysis not just improves the efficiency of fault localization but also achieves better coverage and generates smaller test-suites over DDU, the current state-of-the-art technique. On average, our approach reduces the developer effort over DDU by over 16% for more than 92% of the instances. Further, the improvements over DDU are indeed statistically significant on the paired Wilcoxon Signed-rank test.","doi":"10.24963/ijcai.2020/226","url":"https://ijcai.org/proceedings/2020/226","pdf":"https://ijcai.org/proceedings/2020/0226.pdf","markdown":"https://ijcai.org/proceedings/2020/226.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/226","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Knowledge Representation and Reasoning","track":"Main track","number":226,"pages":{"first":1629,"last":1635},"keywords":["Multidisciplinary Topics and Applications: Knowledge-based Software Engineering","Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25316/file_25316.m3u8","length_sec":272,"provider":"hls"}],"references":[{"position":1,"raw":"[Abreu et al., 2006] Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. An Evaluation of Similarity Coeffi- cients for Software Fault Localization. In Proc. of the 12th PRDC, page 39-46. IEEE Computer Society, 2006.","title":"An Evaluation of Similarity Coefficients for Software Fault Localization","authors":"Rui Abreu, Peter Zoeteweij, Arjan J C Van Gemund","year":"2006","venue":"Proc. of the 12th PRDC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abreu et al., 2009a] Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. Spectrum-Based Multiple Fault Local- ization. In Proc. of the 2009 IEEE/ACM ASE, page 88-99. IEEE Computer Society, 2009.","title":"Spectrum-Based Multiple Fault Localization","authors":"Abreu","year":"2009","venue":"Proc. of the 2009 IEEE/ACM ASE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Abreu et al., 2009b] Rui Abreu, Peter Zoeteweij, Rob Gol- steijn, and Arjan J. C. van Gemund. A Practical Evalua- tion of Spectrum-Based Fault Localization. J. Syst. Softw., 82(11):1780-1792, 2009.","title":"A Practical Evaluation of Spectrum-Based Fault Localization","authors":"Abreu","year":"2009","venue":"J. Syst. Softw","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Abreu et al., 2009c] Rui Abreu, Peter Zoeteweij, and Arjan J. C. Van Gemund. A New Bayesian Approach to Multiple Intermittent Fault Diagnosis. In Proc. of the 21st IJCAI, page 653-658.","title":"A New Bayesian Approach to Multiple Intermittent Fault Diagnosis","authors":"Abreu","year":"2009","venue":"Proc. of the 21st IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/114"},{"position":5,"raw":"[Baudry et al., 2006] Benoit Baudry, Franck Fleurey, and Yves Le Traon. Improving Test Suites for Efficient Fault Localization. In Proc. of the 28th ICSE, page 82-91. ACM, 2006.","title":"Improving Test Suites for Efficient Fault Localization","authors":"Franck Benoit Baudry, Yves Fleurey, Le Traon","year":"2006","venue":"Proc. of the 28th ICSE","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bell et al., 2018] Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. DeFlaker: Automatically Detecting Flaky Tests. In Proc. of the 40th ICSE, page 433-444. ACM, 2018.","title":"DeFlaker: Automatically Detecting Flaky Tests","authors":"Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, Darko Marinov","year":"2018","venue":"Proc. of the 40th ICSE","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Campos et al., 2012] José Campos, André Riboira, Alexan- dre Perez, and Rui Abreu. GZoltar: An Eclipse Plug-in for Testing and Debugging. In Proc. of the 27th IEEE/ACM ASE, page 378-381. ACM, 2012.","title":"GZoltar: An Eclipse Plug-in for Testing and Debugging","authors":"Campos","year":"2012","venue":"Proc. of the 27th IEEE/ACM ASE","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Campos et al., 2013] José Campos, Rui Abreu, Gordon Fraser, and Marcelo d'Amorim. Entropy-based test gener- ation for improved fault localization. In Proc. of the 28th IEEE/ACM ASE, page 257-267. IEEE Press, 2013.","title":"Entropy-based test generation for improved fault localization","authors":"Campos","year":"2013","venue":"Proc. of the 28th IEEE/ACM ASE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Campos et al., 2014] José Campos, Andrea Arcuri, Gordon Fraser, and Rui Abreu. Continuous test generation: En- hancing continuous integration with automated test gener- ation. In Proc. of the 29th ACM/IEEE ASE, page 55-66. ACM, 2014.","title":"Continuous test generation: Enhancing continuous integration with automated test generation","authors":"Campos","year":"2014","venue":"Proc. of the 29th ACM/IEEE ASE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fraser and Arcuri, 2011] Gordon Fraser and Andrea Arcuri. EvoSuite: Automatic Test Suite Generation for Object- Oriented Software. In Proc. of the 19th ACM ESEC/FSE, page 416-419. ACM, 2011.","title":"EvoSuite: Automatic Test Suite Generation for Object-Oriented Software","authors":"Arcuri Fraser, Gordon Fraser, Andrea Arcuri","year":"2011","venue":"Proc. of the 19th ACM ESEC/FSE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fraser and Arcuri, 2015] Gordon Fraser and Andrea Arcuri. 1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite. Empiri- cal Softw. Engg., 20(3):611-639, 2015.","title":"Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite","authors":"Gordon Fraser, Andrea Arcuri","year":"1600","venue":"Empirical Softw. Engg","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gonzalez-Sanchez et al., 2011] Alberto Gonzalez-Sanchez, Hans-Gerhard Gross, and Arjan JC van Gemund. Model- ing the Diagnostic Efficiency of Regression Test Suites. In Proc. of the 4th IEEE ICSTW, pages 634-643, 2011.","title":"Modeling the Diagnostic Efficiency of Regression Test Suites","authors":"Alberto Gonzalez-Sanchez, Hans-Gerhard Gross, Arjan Jc Van Gemund","year":"2011","venue":"Proc. of the 4th IEEE ICSTW","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jost, 2006] Lou Jost. Entropy and diversity. Oikos, 113(2):363-375, 2006.","title":"Entropy and diversity","authors":"Jost","year":"2006","venue":"Oikos","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Just et al., 2014] René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proc. of the 23rd ISSTA, page 437-440. ACM, 2014.","title":"Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs","authors":"Just","year":"2014","venue":"Proc. of the 23rd ISSTA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Kui Liu, Anil Koyuncu, Tegawendé F Bis- syandé, Dongsun Kim, Jacques Klein, and Yves Le Traon. You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems. In Proc. of the 12th IEEE ICST, pages 102-113, 2019.","title":"You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems","authors":"Kui Liu, Anil Koyuncu, F Tegawendé, Dongsun Bissyandé, Jacques Kim, Yves Klein, Le Traon","year":"2019","venue":"Proc. of the 12th IEEE ICST","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lucia et al., 2014] Lucia Lucia, David Lo, Lingxiao Jiang, Ferdian Thung, and Aditya Budi. Extended Comprehen- sive Study of Association Measures for Fault Localization. J. Softw. Evol. Process, 26(2):172-219, 2014.","title":"Extended Comprehensive Study of Association Measures for Fault Localization","authors":"Lucia Lucia, David Lo, Lingxiao Jiang, Ferdian Thung, Aditya Budi","year":"2014","venue":"J. Softw. Evol. Process","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McMinn, 2011] Phil McMinn. Search-Based Software Testing: Past, Present and Future. In Proc. of the 4th IEEE ICSTW, pages 153-163, 2011.","title":"Phil McMinn. Search-Based Software Testing: Past, Present and Future","authors":"Mcminn","year":"2011","venue":"Proc. of the 4th IEEE ICSTW","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mohd Razali and Yap, 2011] Nornadiah Mohd Razali and Bee Yap. Power Comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling Tests. J. Stat. Model. Analytics, 2:21-33, 2011.","title":"Nornadiah Mohd Razali and Bee Yap. Power Comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling Tests","authors":"Mohd Razali, Yap","year":"2011","venue":"J. Stat. Model. Analytics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Namin and Andrews, 2009] Akbar Siami Namin and James H. Andrews. The Influence of Size and Coverage on Test Suite Effectiveness. In Proc. of the 18th ISSTA, page 57-68. ACM, 2009.","title":"The Influence of Size and Coverage on Test Suite Effectiveness","authors":"Akbar Siami Namin, James H Andrews","year":"2009","venue":"Proc. of the 18th ISSTA","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Paterson et al., 2019] David Paterson, José Campos, Rui Abreu, Gregory M Kapfhammer, Gordon Fraser, and Phil McMinn. An Empirical Study on the Use of Defect Pre- diction for Test Case Prioritization. In Proc. of the 12th IEEE ICST, pages 346-357, 2019.","title":"An Empirical Study on the Use of Defect Prediction for Test Case Prioritization","authors":"David Paterson, José Campos, Rui Abreu, Gregory M Kapfhammer, Gordon Fraser, Phil Mcminn","year":"2019","venue":"Proc. of the 12th IEEE ICST","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pearson et al., 2017] Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, De- ric Pang, and Benjamin Keller. Evaluating and Improv- ing Fault Localization. In Proc. of the 39th ICSE, page 609-620. IEEE Press, 2017.","title":"Evaluating and Improving Fault Localization","authors":"Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D Ernst, Deric Pang, Benjamin Keller","year":"2017","venue":"Proc. of the 39th ICSE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perez et al., 2017a] Alexandre Perez, Rui Abreu, and Marcelo d'Amorim. Prevalence of Single-Fault Fixes and Its Impact on Fault Localization. In Proc. of the 10th IEEE ICST, pages 12-22, 2017.","title":"Prevalence of Single-Fault Fixes and Its Impact on Fault Localization","authors":"Perez","year":"2017","venue":"Proc. of the 10th IEEE ICST","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Perez et al., 2017b] Alexandre Perez, Rui Abreu, and Arie van Deursen. A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches. In Proc. of the 39th ICSE, page 654-664. IEEE Press, 2017.","title":"A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches","authors":"Perez","year":"2017","venue":"Proc. of the 39th ICSE","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shamshiri et al., 2015] Sina Shamshiri, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effec- tiveness and Challenges. In Proc. of the 30th IEEE/ACM ASE, pages 201-211, 2015.","title":"Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges","authors":"Shamshiri","year":"2015","venue":"Proc. of the 30th IEEE/ACM ASE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Staats et al., 2012] Matt Staats, Gregory Gay, Michael Whalen, and Mats Heimdahl. On the danger of coverage directed test case generation. In FASE, pages 409-424. Springer Berlin Heidelberg, 2012.","title":"On the danger of coverage directed test case generation","authors":"Matt Staats, Gregory Gay, Michael Whalen, Mats Heimdahl","year":"2012","venue":"FASE","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wong et al., 2016] W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. A Survey on Software Fault Localization. IEEE TSE, 42(8):707-740, 2016.","title":"A Survey on Software Fault Localization","authors":"W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, Franz Wotawa","year":"2016","venue":"IEEE TSE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}