{"type":"ScholarlyArticle","title":"Computationally Assisted Quality Control for Public Health Data Streams","authors":[{"name":"Ananya Joshi","slug":"ananya-joshi","given":"Ananya","family":"Joshi","affiliation":"Carnegie Mellon University"},{"name":"Kathryn Mazaitis","slug":"kathryn-mazaitis","given":"Kathryn","family":"Mazaitis","affiliation":"Carnegie Mellon University"},{"name":"Roni Rosenfeld","slug":"roni-rosenfeld","given":"Roni","family":"Rosenfeld","affiliation":"Carnegie Mellon University"},{"name":"Bryan Wilder","slug":"bryan-wilder","given":"Bryan","family":"Wilder","affiliation":"Carnegie Mellon University"}],"abstract":"Irregularities in public health data streams (like COVID-19 Cases) hamper data-driven decision-making for public health stakeholders. A real-time, computer-generated list of the most important, outlying data points from thousands of public health data streams could assist an expert reviewer in identifying these irregularities. However, existing outlier detection frameworks perform poorly on this task because they do not account for the data volume or for the statistical properties of public health streams. Accordingly, we developed FlaSH (Flagging Streams in public Health), a practical outlier detection framework for public health data users that uses simple, scalable models to capture these statistical properties explicitly. In an experiment where human experts evaluate FlaSH and existing methods (including deep learning approaches), FlaSH scales to the data volume of this task, matches or exceeds these other methods in mean accuracy, and identifies the outlier points that users empirically rate as more helpful. Based on these results, FlaSH has been deployed on data streams used by public health stakeholders.","doi":"10.24963/ijcai.2023/666","url":"https://ijcai.org/proceedings/2023/666","pdf":"https://ijcai.org/proceedings/2023/0666.pdf","markdown":"https://ijcai.org/proceedings/2023/666.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/666","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":666,"pages":{"first":6004,"last":6012},"keywords":["AI for Good: Humans and AI","AI for Good: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2022] Pulak Agarwal, Pranav Aluru, and B Aditya Prakash. Real-time anomaly detection in epi- demic data streams. In Epidemiology meets Data Mining and Knowledge discovery, 2022.","title":"Real-time anomaly detection in epidemic data streams","authors":"Pulak Agarwal, Pranav Aluru, Aditya Prakash","year":"2022","venue":"Epidemiology meets Data Mining and Knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An and Cho, 2015] Jinwon An and Sungzoon Cho. Varia- tional autoencoder based anomaly detection using recon- struction probability. Special lecture on IE, 2(1):1-18, 2015.","title":"Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability","authors":"An, Cho","year":"2015","venue":"Special lecture on IE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Angiulli and Pizzuti, 2002] Fabrizio Angiulli and Clara Piz- zuti. Fast outlier detection in high dimensional spaces. In Principles of Data Mining and Knowledge Discovery: 6th European Conference, pages 15-27. Springer, 2002.","title":"Fabrizio Angiulli and Clara Pizzuti. Fast outlier detection in high dimensional spaces","authors":"Pizzuti Angiulli","year":"2002","venue":"Principles of Data Mining and Knowledge Discovery: 6th European Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blázquez-García et al., 2021] Ane Blázquez-García, Angel Conde, Usue Mori, and Jose A Lozano. A review on out- lier/anomaly detection in time series data. ACM Comput- ing Surveys (CSUR), 54(3):1-33, 2021.","title":"A review on outlier/anomaly detection in time series data","authors":"Ane Blázquez-García, Angel Conde, Usue Mori, Jose A Lozano","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Breunig et al., 2000] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: iden- tifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000. [CDC, 2022] US CDC. Covid-19 by county. https://www.cdc.gov/coronavirus/2019-ncov/your-health/ covid-by-county.html, publisher=Centers for Disease Control and Prevention, Aug 2022.","title":"Lof: identifying density-based local outliers","authors":"Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, Jörg Sander","year":"2000","venue":"Proceedings of the 2000 ACM SIGMOD international conference on Management of data","doi":null,"url":"https://www.cdc.gov/coronavirus/2019-ncov/your-health/covid-by-county.html","internal_url":null},{"position":6,"raw":"[Chimmula et al., 2020] Vinay Chimmula, Reddy Kumar, and Lei Zhang. Time series forecasting of covid-19 trans- mission in canada using lstm networks. Chaos, Solitons & Fractals, 135:109864, 2020.","title":"Time series forecasting of covid-19 transmission in canada using lstm networks","authors":"Vinay Chimmula, Reddy Kumar, Lei Zhang","year":"2020","venue":"Chaos, Solitons & Fractals","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cramer et al., 2022] Estee Y Cramer, Yuxin Huang, Yijin Wang, Evan L Ray, Matthew Cornell, Johannes Bracher, Andrea Brennen, Alvaro J Castro Rivadeneira, Aaron Gerding, Katie House, et al. The united states covid-19 forecast hub dataset. Scientific data, 9(1):462, 2022.","title":"The united states covid-19 forecast hub dataset","authors":"Cramer","year":"2022","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2022] Ensheng Dong, Jeremy Ratcliff, Tamara D Goyea, Aaron Katz, Ryan Lau, Timothy K Ng, Beatrice Garcia, Evan Bolt, Sarah Prata, David Zhang, et al. The johns hopkins university center for systems science and engineering covid-19 dashboard: data collection process, challenges faced, and lessons learned. The Lancet Infectious Diseases, 2022.","title":"The johns hopkins university center for systems science and engineering covid-19 dashboard: data collection process, challenges faced, and lessons learned","authors":"Dong","year":"2022","venue":"The Lancet Infectious Diseases","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2017] Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagno- sis from system logs through deep learning. In Proceed- ings of the 2017 ACM SIGSAC conference on computer and communications security, pages 1285-1298, 2017.","title":"Deeplog: Anomaly detection and diagnosis from system logs through deep learning","authors":"Du","year":"2017","venue":"Proceedings of the 2017 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Farrow et al., 2015] David C Farrow, Logan C Brooks, Aaron Rumack, Ryan J Tibshirani, and Roni Rosenfeld. Delphi epidata api. The Lancet Infectious Diseases. https://github. com/cmu-delphi/delphi-epidata, 2015.","title":"Delphi epidata api","authors":"Logan C David C Farrow, Aaron Brooks, Ryan J Rumack, Roni Tibshirani, Rosenfeld","year":null,"venue":null,"doi":null,"url":"https://github.com/cmu-delphi/delphi-epidata,2015","internal_url":null},{"position":11,"raw":"[Hundman et al., 2018] Kyle Hundman, Valentino Con- stantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. arXiv preprint arXiv:1802.04431, 2018.","title":"Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding","authors":"Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, Tom Soderstrom","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jombart et al., 2021] Thibaut Jombart, Stéphane Ghozzi, Dirk Schumacher, Timothy J Taylor, Quentin J Leclerc, Mark Jit, Stefan Flasche, Felix Greaves, Tom Ward, Ros- alind M Eggo, et al. Real-time monitoring of covid-19 dynamics using automated trend fitting and anomaly de- tection. Philosophical Transactions of the Royal Society B, 376(1829):20200266, 2021.","title":"Real-time monitoring of covid-19 dynamics using automated trend fitting and anomaly detection","authors":"Thibaut Jombart, Stéphane Ghozzi, Dirk Schumacher, Timothy J Taylor, Quentin J Leclerc, Mark Jit, Stefan Flasche, Felix Greaves, Tom Ward, Rosalind M Eggo","year":"1829","venue":"Philosophical Transactions of the Royal Society B","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joshi et al., 2023] Ananya Joshi, Kathryn Mazaitis, Roni Rosenfeld, and Bryan Wilder. Osf pre-data collection reg- istration: Towards detecting points of interest from public health data streams. https://osf.io/2v8f5, 2023.","title":"Osf pre-data collection registration: Towards detecting points of interest from public health data streams","authors":"Ananya Joshi, Kathryn Mazaitis, Roni Rosenfeld, Bryan Wilder","year":"2023","venue":null,"doi":null,"url":"https://osf.io/2v8f5","internal_url":null},{"position":14,"raw":"[Karadayi et al., 2020] Yildiz Karadayi, Mehmet N Aydin, and Arif Selc ¸uk Öǧrencí. Unsupervised anomaly detection in multivariate spatio-temporal data using deep learning: early detection of covid-19 outbreak in italy. Ieee Access, 8:164155-164177, 2020.","title":"Unsupervised anomaly detection in multivariate spatio-temporal data using deep learning: early detection of covid-19 outbreak in italy","authors":"Yildiz Karadayi, Arif Mehmet N Aydin, Selc ¸uk Öǧrencí","year":"2020","venue":"Ieee Access","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Killick et al., 2012] Rebecca Killick, Paul Fearnhead, and Idris A Eckley. Optimal detection of changepoints with a linear computational cost. Journal of the American Statis- tical Association, 107(500):1590-1598, 2012.","title":"Optimal detection of changepoints with a linear computational cost","authors":"Rebecca Killick, Paul Fearnhead, Idris A Eckley","year":"2012","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kraemer et al., 2021] Moritz UG Kraemer, Samuel V Scarpino, Vukosi Marivate, Bernardo Gutierrez, Bo Xu, Graham Lee, Jared B Hawkins, Caitlin Rivers, David M Pigott, Rebecca Katz, et al. Data curation during a pan- demic and lessons learned from covid-19. Nature Compu- tational Science, 1(1):9-10, 2021.","title":"Data curation during a pandemic and lessons learned from covid-19","authors":"Moritz Ug Kraemer, Vukosi Samuel V Scarpino, Bernardo Marivate, Bo Gutierrez, Graham Xu, Jared B Lee, Caitlin Hawkins, David M Rivers, Rebecca Pigott, Katz","year":"2021","venue":"Nature Computational Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kreps and Kriner, 2020] Sarah E Kreps and Douglas L Kriner. Model uncertainty, political contestation, and pub- lic trust in science: Evidence from the covid-19 pandemic. Science advances, 6(43):eabd4563, 2020.","title":"Model uncertainty, political contestation, and public trust in science: Evidence from the covid-19 pandemic","authors":"E Sarah, Douglas L Kreps, Kriner","year":"2020","venue":"Science advances","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lai et al., 2021] Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track, 2021.","title":"Revisiting time series outlier detection: Definitions and benchmarks","authors":"Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, Xia Hu","year":"2021","venue":"Thirty-fifth conference on neural information processing systems datasets and benchmarks track","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leslie et al., 2021] David Leslie, Anjali Mazumder, Aidan Peppin, Maria K Wolters, and Alexa Hagerty. Does \"ai\" stand for augmenting inequality in the era of covid-19 healthcare? bmj, 372, 2021.","title":"Does \"ai\" stand for augmenting inequality in the era of covid-19 healthcare?","authors":"David Leslie, Anjali Mazumder, Aidan Peppin, Maria K Wolters, Alexa Hagerty","year":"2021","venue":"bmj","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2008] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.","title":"Isolation forest","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi-Hua Zhou","year":"2008","venue":"2008 eighth ieee international conference on data mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McDonald et al., 2021] Daniel J McDonald, Jacob Bien, Alden Green, Addison J Hu, Nat DeFries, Sangwon Hyun, Natalia L Oliveira, James Sharpnack, Jingjing Tang, Robert Tibshirani, et al. Can auxiliary indica- tors improve covid-19 forecasting and hotspot predic- tion? Proceedings of the National Academy of Sciences, 118(51):e2111453118, 2021.","title":"Can auxiliary indicators improve covid-19 forecasting and hotspot prediction?","authors":"J Daniel, Jacob Mcdonald, Alden Bien, Green, J Addison, Nat Hu, Sangwon Defries, Natalia L Hyun, James Oliveira, Jingjing Sharpnack, Robert Tang, Tibshirani","year":"2021","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Paleyes et al., 2022] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence. Challenges in deploying machine learning: a survey of case studies. ACM Computing Sur- veys, 55(6):1-29, 2022.","title":"Challenges in deploying machine learning: a survey of case studies","authors":"Andrei Paleyes, Raoul-Gabriel Urma, Neil D Lawrence","year":"2022","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pang et al., 2021] Guansong Pang, Chunhua Shen, Long- bing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1-38, 2021.","title":"Deep learning for anomaly detection: A review","authors":"Pang","year":"2021","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pevnỳ, 2016] Tomáš Pevnỳ. Loda: Lightweight on-line de- tector of anomalies. Machine Learning, 102:275-304, 2016.","title":"Loda: Lightweight on-line detector of anomalies","authors":"Tomáš Pevnỳ, Pevnỳ","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Reinhart et al., 2021] Alex Reinhart, Logan Brooks, Maria Jahja, Aaron Rumack, Jingjing Tang, Sumit Agrawal, Wael Al Saeed, Taylor Arnold, Amartya Basu, Jacob Bien, et al. An open repository of real-time covid-19 indica- tors. Proceedings of the National Academy of Sciences, 118(51):e2111452118, 2021.","title":"An open repository of real-time covid-19 indicators","authors":"Alex Reinhart, Logan Brooks, Maria Jahja, Aaron Rumack, Jingjing Tang, Sumit Agrawal, Al Wael, Taylor Saeed, Amartya Arnold, Jacob Basu, Bien","year":"2021","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sáez et al., 2021] Carlos Sáez, Nekane Romero, J Alberto Conejero, and Juan M García-Gómez. Potential limi- tations in covid-19 machine learning due to data source variability: A case study in the ncov2019 dataset. Jour- nal of the American Medical Informatics Association, 28(2):360-364, 2021.","title":"Potential limitations in covid-19 machine learning due to data source variability: A case study in the ncov2019 dataset","authors":"Carlos Sáez, Nekane Romero, Alberto Conejero, Juan M García-Gómez","year":"2021","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sejr and Schneider-Kamp, 2021] Jonas Herskind Sejr and Anna Schneider-Kamp. Explainable outlier detection: What, for whom and why? Machine Learning with Ap- plications, 6:100172, 2021.","title":"Jonas Herskind Sejr and Anna Schneider-Kamp. Explainable outlier detection: What, for whom and why?","authors":"Schneider-Kamp Sejr","year":"2021","venue":"Machine Learning with Applications","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simon, 2021] Sara Simon. Inconsistent reporting practices hampered our ability to analyze covid-19 data. here are three common problems we identified. The COVID Track- ing Project, Apr 2021.","title":"Inconsistent reporting practices hampered our ability to analyze covid-19 data. here are three common problems we identified","authors":"Sara Simon","year":"2021","venue":"The COVID Tracking Project","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Truong et al., 2020] Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing, 167:107299, 2020.","title":"Selective review of offline change point detection methods","authors":"Charles Truong, Laurent Oudre, Nicolas Vayatis","year":"2020","venue":"Signal Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021] Guannan Wang, Zhiling Gu, Xinyi Li, Shan Yu, Myungjin Kim, Yueying Wang, Lei Gao, and Li Wang. Comparing and integrating us covid-19 data from multiple sources with anomaly detection and repair- ing. Journal of Applied Statistics, pages 1-27, 2021.","title":"Comparing and integrating us covid-19 data from multiple sources with anomaly detection and repairing","authors":"Guannan Wang, Zhiling Gu, Xinyi Li, Shan Yu, Myungjin Kim, Yueying Wang, Lei Gao, Li Wang","year":"2021","venue":"Journal of Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Webber et al., 2010] William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rank- ings. ACM Transactions on Information Systems (TOIS), 28(4):1-38, 2010.","title":"A similarity measure for indefinite rankings","authors":"William Webber, Alistair Moffat, Justin Zobel","year":"2010","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wong, 2004] Weng-Keen Wong. Data mining for early disease outbreak detection. Carnegie Mellon University, 2004.","title":"Data mining for early disease outbreak detection","authors":"Weng-Keen Wong","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu and Keogh, 2021] Renjie Wu and Eamonn Keogh. Cur- rent time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE Transac- tions on Knowledge and Data Engineering, 2021.","title":"Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress","authors":"Keogh Wu, Renjie Wu, Eamonn Keogh","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2021] Shuo Yu, Qing Qing, Chen Zhang, Ah- san Shehzad, Giles Oatley, and Feng Xia. Data- driven decision-making in covid-19 response: A sur- vey. IEEE Transactions on Computational Social Systems, 8(4):1016-1029, 2021.","title":"Datadriven decision-making in covid-19 response: A survey","authors":"Shuo Yu, Qing Qing, Chen Zhang, Ahsan Shehzad, Giles Oatley, Feng Xia","year":"2021","venue":"IEEE Transactions on Computational Social Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2021] Di Zhu, Xinyue Ye, and Steven Manson. Revealing the spatial shifting pattern of covid-19 pan- demic in the united states. Scientific reports, 11(1):1-9, 2021.","title":"Revealing the spatial shifting pattern of covid-19 pandemic in the united states","authors":"Di Zhu, Xinyue Ye, Steven Manson","year":"2021","venue":"Scientific reports","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}