{"type":"ScholarlyArticle","title":"Exact Bernoulli Scan Statistics using Binary Decision Diagrams","authors":[{"name":"Masakazu Ishihata","slug":"masakazu-ishihata","given":"Masakazu","family":"Ishihata","affiliation":"NTT Communication Science Laboratories"},{"name":"Takanori Maehara","slug":"takanori-maehara","given":"Takanori","family":"Maehara","affiliation":"RIKEN Center for Advanced Intelligence Project"}],"abstract":"In combinatorial statistics, we are interested in a statistical test of combinatorial correlation, i.e., existence a subset from an underlying combinatorial structure such that the observation is large on the subset.\r\nThe combinatorial scan statistics has been proposed for such a statistical test; however, it is not commonly used in practice because of its high computational cost.\r\nIn this study, we restrict our attention to the case that the number of data points is moderately small (e.g., 50), the outcome is binary, and the underlying combinatorial structure is represented by a zero-suppressed binary decision diagram (ZDD), and consider the problem of computing the p-value of the combinatorial scan statistics exactly. First, we prove that this problem is a #P-hard problem. Then, we propose a practical algorithm that solves the problem. Here, the algorithm constructs a binary decision diagram (BDD) for a set of realizations of the random variables by a dynamic programming on the ZDD, and computes the p-value by a dynamic programming on the BDD. We conducted experiments to evaluate the performance of the proposed algorithm using real-world datasets.","doi":"10.24963/ijcai.2019/795","url":"https://ijcai.org/proceedings/2019/795","pdf":"https://ijcai.org/proceedings/2019/0795.pdf","markdown":"https://ijcai.org/proceedings/2019/795.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/795","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Uncertainty in AI","track":"Main track","number":795,"pages":{"first":5737,"last":5743},"keywords":["Knowledge Representation and Reasoning: Tractable Languages and Knowledge compilation","Uncertainty in AI: Exact Probabilistic Inference"],"videos":[],"references":[{"position":1,"raw":"[Addario-Berry et al., 2010] Louigi Addario-Berry, Nicolas Broutin, Luc Devroye, Gábor Lugosi, et al. On combinato- rial testing problems. The Annals of Statistics, 38(5):3063- 3092, 2010.","title":"On combinatorial testing problems","authors":"Addario-Berry","year":"2010","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alon et al., 1995] Noga Alon, Raphael Yuster, and Uri Zwick. Color-coding. J. ACM, 42(4):844-856, 1995.","title":"Color-coding","authors":"Noga Alon, Raphael Yuster, Uri Zwick","year":"1995","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arias-Castro et al., 2008] Ery Arias-Castro, Emmanuel J Candès, Hannes Helgason, Ofer Zeitouni, et al. Searching for a trail of evidence in a maze. The Annals of Statistics, 36(4):1726-1757, 2008.","title":"Searching for a trail of evidence in a maze","authors":"Ery Arias-Castro, Emmanuel J Candès, Hannes Helgason, Ofer Zeitouni","year":"2008","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arias-Castro et al., 2011] Ery Arias-Castro, Emmanuel J Candes, Arnaud Durand, et al. Detection of an anomalous cluster in a network. The Annals of Statistics, 39(1):278- 304, 2011.","title":"Detection of an anomalous cluster in a network","authors":"Ery Arias-Castro, Emmanuel J Candes, Arnaud Durand","year":"2011","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bryant, 1992] Randal E Bryant. Symbolic boolean manipu- lation with ordered binary-decision diagrams. ACM Com- puting Surveys (CSUR), 24(3):293-318, 1992.","title":"Symbolic boolean manipulation with ordered binary-decision diagrams","authors":"Randal E Bryant","year":"1992","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cadena et al., 2017] Jose Cadena, Feng Chen, and Anil Vul- likanti. Near-optimal and practical algorithms for graph scan statistics. In Proceedings of the 2017 SIAM Interna- tional Conference on Data Mining, pages 624-632. SIAM, 2017.","title":"Near-optimal and practical algorithms for graph scan statistics","authors":"Jose Cadena, Feng Chen, Anil Vullikanti","year":"2017","venue":"Proceedings of the 2017 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Imai et al., 1999] Hiroshi Imai, Kyoko Sekine, and Keiko Imai. Computational investigations of all-terminal net- work reliability via bdds. IEICE transactions on funda- mentals of electronics, communications and computer sci- ences, 82(5):714-721, 1999.","title":"Computational investigations of all-terminal network reliability via bdds","authors":"Hiroshi Imai, Kyoko Sekine, Keiko Imai","year":"1999","venue":"IEICE transactions on fundamentals of electronics, communications and computer sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ishihata et al., 2010] Masakazu Ishihata, Yoshitaka Kameya, Taisuke Sato, and Shin-ichi Minato. An em algorithm on bdds with order encoding for logic-based probabilistic models. In Proceedings of 2nd Asian Conference on Machine Learning, pages 161-176, 2010.","title":"An em algorithm on bdds with order encoding for logic-based probabilistic models","authors":"Masakazu Ishihata, Yoshitaka Kameya, Taisuke Sato, Shin-Ichi Minato","year":"2010","venue":"Proceedings of 2nd Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jerrum, 2003] Mark Jerrum. Counting, sampling and inte- grating: algorithms and complexity. Springer Science & Business Media, 2003.","title":"Counting, sampling and integrating: algorithms and complexity","authors":"Mark Jerrum","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kawahara et al., 2017] Jun Kawahara, Takeru Inoue, Hi- roaki Iwashita, and Shin-ichi Minato. Frontier-based search for enumerating all constrained subgraphs with compressed representation. IEICE Transactions on Fun- damentals of Electronics, Communications and Computer Sciences, 100(9):1773-1784, 2017.","title":"Frontier-based search for enumerating all constrained subgraphs with compressed representation","authors":"Kawahara","year":"2017","venue":"IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Knuth, 2009] Donald E Knuth. The art of computer pro- gramming: Bitwise tricks & techniques. Binary Decision Diagrams, 4, 2009.","title":"The art of computer programming: Bitwise tricks & techniques","authors":"E Donald, Knuth","year":"2009","venue":"Binary Decision Diagrams","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lugosi, 2017] Gábor Lugosi. Lectures on combinatorial statistics, 2017.","title":"Gábor Lugosi. Lectures on combinatorial statistics","authors":"Lugosi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[MacKinnon, 2009] James G MacKinnon. Bootstrap hy- pothesis testing. Handbook of Computational Economet- rics, 183:213, 2009.","title":"Bootstrap hypothesis testing","authors":"James G Mackinnon","year":"2009","venue":"Handbook of Computational Econometrics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maehara et al., 2017] Takanori Maehara, Hirofumi Suzuki, and Masakazu Ishihata. Exact computation of influence spread by binary decision diagrams. In Proceedings of the 26th International Conference on World Wide Web, pages 947-956. International World Wide Web Confer- ences Steering Committee, 2017.","title":"Exact computation of influence spread by binary decision diagrams","authors":"Takanori Maehara, Hirofumi Suzuki, Masakazu Ishihata","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Minato, 1993] Shin-ichi Minato. Zero-suppressed bdds for set manipulation in combinatorial problems. In 30th ACM/IEEE Design Automation Conference, pages 272- 277. IEEE, 1993.","title":"Zero-suppressed bdds for set manipulation in combinatorial problems","authors":"Shin-Ichi Minato","year":"1993","venue":"30th ACM/IEEE Design Automation Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Minato, 2013] Shin-ichi Minato. Techniques of bdd/zdd: brief history and recent activity. IEICE TRANSACTIONS on Information and Systems, 96(7):1419-1429, 2013.","title":"Techniques of bdd/zdd: brief history and recent activity","authors":"Shin-Ichi Minato, Minato","year":"2013","venue":"IEICE TRANSACTIONS on Information and Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qian et al., 2014] Jing Qian, Venkatesh Saligrama, and Yut- ing Chen. Connected sub-graph detection. In Artificial Intelligence and Statistics, pages 796-804, 2014.","title":"Connected sub-graph detection","authors":"Jing Qian, Venkatesh Saligrama, Yuting Chen","year":"2014","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Taylor and Tibshirani, 2015] Jonathan Taylor and Robert J Tibshirani. Statistical learning and selective infer- ence. Proceedings of the National Academy of Sciences, 112(25):7629-7634, 2015.","title":"Statistical learning and selective inference","authors":"Tibshirani Taylor, Jonathan Taylor, Robert J Tibshirani","year":"2015","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Weerahandi, 2013] Samaradasa Weerahandi. Exact statisti- cal methods for data analysis. Springer Science & Busi- ness Media, 2013.","title":"Samaradasa Weerahandi. Exact statistical methods for data analysis","authors":"Weerahandi","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}