{"type":"ScholarlyArticle","title":"What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature","authors":[{"name":"Byron C. Wallace","slug":"byron-c-wallace","given":"Byron C.","family":"Wallace","affiliation":"College of Computer Sciences , Northeastern University , Boston , MA , USA"}],"abstract":"Ideally decisions regarding medical treatments would be informed by the totality of the available evidence. The best evidence we currently have is in published natural language articles describing the conduct and results of clinical trials. Because these are unstructured, it is difficult for domain experts (e.g., physicians) to sort through and appraise the evidence pertaining to a given clinical question. Natural language technologies have the potential to improve access to the evidence via semi-automated processing of the biomedical literature. In this brief paper I highlight work on developing tasks, corpora, and models to support semi-automated evidence retrieval and extraction. The aim is to design models that can consume articles describing clinical trials and automatically extract from these key clinical variables and findings, and estimate their reliability. Completely automating `machine reading' of evidence remains a distant aim given current technologies; the more immediate hope is to use such technologies to help domain experts access and make sense of unstructured biomedical evidence more efficiently, with the ultimate aim of improving patient care. Aside from their practical importance, these tasks pose core NLP challenges that directly motivate methodological innovation.","doi":"10.24963/ijcai.2019/899","url":"https://ijcai.org/proceedings/2019/899","pdf":"https://ijcai.org/proceedings/2019/0899.pdf","markdown":"https://ijcai.org/proceedings/2019/899.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/899","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Early Career","track":"Early Career","number":899,"pages":{"first":6416,"last":6420},"keywords":["Natural Language Processing: Natural Language Processing","Multidisciplinary Topics and Applications: Biology and Medicine","Natural Language Processing: Information Extraction"],"videos":[],"references":[{"position":1,"raw":"[Bastian et al., 2010] Hilda Bastian, Paul Glasziou, and Iain Chalmers. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med, 7(9):e1000326, 2010.","title":"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?","authors":"Bastian","year":"2010","venue":"PLoS Med","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beltagy et al., 2019] Iz Beltagy, Arman Cohan, and Kyle Lo. SciBERT: Pretrained contextualized embeddings for scientific text. arXiv preprint arXiv:1903.10676, 2019.","title":"SciBERT: Pretrained contextualized embeddings for scientific text","authors":"Iz Beltagy, Arman Cohan, Kyle Lo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blake and Lucic, 2015] Catherine Blake and Ana Lucic. Auto- matic endpoint detection to support the systematic review pro- cess. Journal of Biomedical Informatics, 56:42-56, 2015.","title":"Catherine Blake and Ana Lucic. Automatic endpoint detection to support the systematic review process","authors":"Lucic Blake","year":"2015","venue":"Journal of Biomedical Informatics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boudin et al., 2010] Florian Boudin, Jian-Yun Nie, and Martin Dawes. Positional language models for clinical information re- trieval. In EMNLP, pages 108-115, 2010.","title":"Positional language models for clinical information retrieval","authors":"Florian Boudin, Jian-Yun Nie, Martin Dawes","year":"2010","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohen et al., 2006] Aaron M Cohen, William R Hersh, K Peter- son, and Po-Yin Yen. Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13(2):206-219, 2006.","title":"Reducing workload in systematic review preparation using automated citation classification","authors":"William R Aaron M Cohen, K Hersh, Po-Yin Peterson, Yen","year":"2006","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cohen et al., 2015] Aaron M Cohen, Neil R Smalheiser, Marian S McDonagh, Clement Yu, Clive E Adams, John M Davis, and Philip S Yu. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. Journal of the American Medical Informatics Asso- ciation, 22(3):707-717, 2015.","title":"Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine","authors":"Neil R Aaron M Cohen, Marian S Smalheiser, Clement Mcdonagh, Clive E Yu, John M Adams, Philip S Davis, Yu","year":"2015","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Craven et al., 1999] Mark Craven, Johan Kumlien, et al. Con- structing biological knowledge bases by extracting information from text sources. In ISMB, volume 1999, pages 77-86, 1999.","title":"Constructing biological knowledge bases by extracting information from text sources","authors":"Mark Craven, Johan Kumlien","year":"1999","venue":"ISMB","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dickersin et al., 1994] Kay Dickersin, Roberta Scherer, and Carol Lefebvre. Systematic reviews: identifying relevant studies for systematic reviews. BMJ, 309(6964):1286-1291, 1994.","title":"Systematic reviews: identifying relevant studies for systematic reviews","authors":"Kay Dickersin, Roberta Scherer, Carol Lefebvre","year":"1994","venue":"BMJ","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Higgins et al., 2011] Julian PT Higgins, Douglas G Altman, Pe- ter C Gøtzsche, Peter Jüni, David Moher, Andrew D Oxman, Je- lena Savović, Kenneth F Schulz, Laura Weeks, and Jonathan AC Sterne. The Cochrane collaboration's tool for assessing risk of bias in randomised trials. British Medical Journal, 343:d5928, 2011.","title":"The Cochrane collaboration's tool for assessing risk of bias in randomised trials","authors":"Julian Pt Higgins, Douglas G Altman, Peter C Gøtzsche, Peter Jüni, David Moher, Jelena Andrew D Oxman, Kenneth F Savović, Laura Schulz, Jonathan Ac Weeks, Sterne","year":"2011","venue":"British Medical Journal","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2006] Xiaoli Huang, Jimmy Lin, and Dina Demner- Fushman. Evaluation of PICO as a knowledge representation for clinical questions. In AMIA, volume 2006, page 359, 2006.","title":"Evaluation of PICO as a knowledge representation for clinical questions","authors":"Xiaoli Huang, Jimmy Lin, Dina Demner-Fushman","year":"2006","venue":"AMIA","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin and Szolovits, 2018] Di Jin and Peter Szolovits. PICO element detection in medical text via long short-term memory neural net- works. In BioNLP, pages 67-75, 2018.","title":"PICO element detection in medical text via long short-term memory neural networks","authors":"Di Jin, Peter Szolovits","year":"2018","venue":"BioNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiritchenko et al., 2010] Svetlana Kiritchenko, Berry de Bruijn, Simona Carini, Joel Martin, and Ida Sim. Exact: automatic ex- traction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10(1):56, 2010.","title":"Exact: automatic extraction of clinical trial characteristics from journal publications","authors":"Svetlana Kiritchenko, Berry De Bruijn, Simona Carini, Joel Martin, Ida Sim","year":"2010","venue":"BMC medical informatics and decision making","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lehman et al., 2019] Eric Lehman, Jay DeYoung, Regina Barzi- lay, and Byron C. Wallace. Inferring Which Medical Treatments Work from Reports of Clinical Trials. In NAACL, 2019.","title":"Inferring Which Medical Treatments Work from Reports of Clinical Trials","authors":"Eric Lehman, Jay Deyoung, Regina Barzilay, Byron C Wallace","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marshall et al., 2015] Iain J Marshall, Joël Kuiper, and Byron C Wallace. Automating risk of bias assessment for clinical tri- als. IEEE Journal of Biomedical and Health Informatics, 19(4):1406-1412, 2015.","title":"Automating risk of bias assessment for clinical trials","authors":"Iain J Marshall, Joël Kuiper, Byron C Wallace","year":"2015","venue":"IEEE Journal of Biomedical and Health Informatics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Marshall et al., 2017] Iain Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. Automating Biomedical Evidence Syn- thesis: RobotReviewer. In ACL, pages 7-12, 2017.","title":"Automating Biomedical Evidence Synthesis: RobotReviewer","authors":"Iain Marshall, Joël Kuiper, Edward Banner, Byron C Wallace","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Marshall et al., 2018] Iain J Marshall, Anna Noel-Storr, Joël Kuiper, James Thomas, and Byron C Wallace. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Research synthesis methods, 9(4):602-614, 2018.","title":"Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide","authors":"Iain J Marshall, Anna Noel-Storr, Joël Kuiper, James Thomas, Byron C Wallace","year":"2018","venue":"Research synthesis methods","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Millard et al., 2015] Louise AC Millard, Peter A Flach, and Ju- lian PT Higgins. Machine learning to assist risk-of-bias assess- ments in systematic reviews. International journal of epidemiol- ogy, 45(1):266-277, 2015.","title":"Machine learning to assist risk-of-bias assessments in systematic reviews","authors":"Louise Ac Millard, Peter A Flach, Julian Pt Higgins","year":"2015","venue":"International journal of epidemiology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mintz et al., 2009] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without la- beled data. In ACL/AFNLP, pages 1003-1011, 2009.","title":"Distant supervision for relation extraction without labeled data","authors":"Mike Mintz, Steven Bills, Rion Snow, Dan Jurafsky","year":"2009","venue":"ACL/AFNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mortensen et al., 2017] Michael L Mortensen, Gaelen P Adam, Thomas A Trikalinos, Tim Kraska, and Byron C Wallace. An exploration of crowdsourcing citation screening for systematic reviews. Research synthesis methods, 8(3):366-386, 2017.","title":"An exploration of crowdsourcing citation screening for systematic reviews","authors":"Michael L Mortensen, P Gaelen, Thomas A Adam, Tim Trikalinos, Byron C Kraska, Wallace","year":"2017","venue":"Research synthesis methods","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen et al., 2015] An Thanh Nguyen, Byron C Wallace, and Matthew Lease. Combining crowd and expert labels using de- cision theoretic active learning. In HCOMP, 2015.","title":"Combining crowd and expert labels using decision theoretic active learning","authors":"Nguyen","year":"2015","venue":"An Thanh Nguyen","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen et al., 2017] An T Nguyen, Byron C Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggregating and predict- ing sequence labels from crowd annotations. In ACL, volume 2017, pages 299-310, 2017.","title":"Aggregating and predicting sequence labels from crowd annotations","authors":"Nguyen","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nye et al., 2018] Benjamin Nye, Junyi Jessy Li, Roma Patel, Yin- fei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL, volume 2018, pages 197-207, 2018.","title":"A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature","authors":"Benjamin Nye, Jessy Junyi, Roma Li, Yinfei Patel, Iain J Yang, Ani Marshall, Byron C Nenkova, Wallace","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[O'Mara-Eves et al., 2015] Alison O'Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. Using text mining for study identification in systematic reviews: a sys- tematic review of current approaches. Systematic reviews, 4(1):5, 2015.","title":"Using text mining for study identification in systematic reviews: a systematic review of current approaches","authors":"Alison O' Mara-Eves, James Thomas, John Mcnaught, Makoto Miwa, Sophia Ananiadou","year":"2015","venue":"Systematic reviews","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Patel et al., 2018] Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron C Wallace. Syntactic patterns improve information extraction for medical search. In NAACL, volume 2018, pages 371-375, 2018.","title":"Syntactic patterns improve information extraction for medical search","authors":"Patel","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peng et al., 2017] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. Cross-sentence n-ary rela- tion extraction with graph lstms. TACL, 5:101-115, 2017.","title":"Cross-sentence n-ary relation extraction with graph lstms","authors":"Peng","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Przybyła et al., 2018] Piotr Przybyła, Austin J Brockmeier, Geor- gios Kontonatsios, Marie-Annick Le Pogam, John McNaught, Erik von Elm, Kay Nolan, and Sophia Ananiadou. Prioritis- ing references for systematic reviews with robotanalyst: A user study. Research synthesis methods, 9(3):470-488, 2018.","title":"Prioritising references for systematic reviews with robotanalyst: A user study","authors":"Piotr Przybyła, Austin J Brockmeier, Georgios Kontonatsios, Marie-Annick Le Pogam, John Mcnaught, Kay Erik Von Elm, Sophia Nolan, Ananiadou","year":"2018","venue":"Research synthesis methods","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Scells et al., 2017] Harrisen Scells, Guido Zuccon, Bevan Koop- man, Anthony Deacon, Leif Azzopardi, and Shlomo Geva. In- tegrating the framing of clinical questions via pico into the re- trieval of medical literature for systematic reviews. In CIKM, pages 2291-2294. ACM, 2017.","title":"Integrating the framing of clinical questions via pico into the retrieval of medical literature for systematic reviews","authors":"Scells","year":"2017","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Settles, 2009] Burr Settles. Active learning literature survey. Tech- nical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.","title":"Burr Settles. Active learning literature survey","authors":"Settles","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Small et al., 2011] Kevin Small, Byron C Wallace, Carla E Brod- ley, and Thomas A Trikalinos. The constrained weight space svm: learning with ranked features. In ICML, pages 865-872.","title":"The constrained weight space svm: learning with ranked features","authors":"Kevin Small, Byron C Wallace, Carla E Brodley, Thomas A Trikalinos","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Soboczenski et al., 2019] Frank Soboczenski, Thomas A. Trikali- nos, Joël Kuiper, Randolph G. Bias, Byron C. Wallace, and Iain J. Marshall. Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Medi- cal Informatics and Decision Making, 2019.","title":"Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study","authors":"Frank Soboczenski, Thomas A Trikalinos, Joël Kuiper, Randolph G Bias, Byron C Wallace, Iain J Marshall","year":"2019","venue":"BMC Medical Informatics and Decision Making","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Strout et al., 2019] Julia Strout, Ye Zhang, and Raymond J Mooney. Do human rationales improve machine explanations? arXiv preprint arXiv:1905.13714, 2019.","title":"Do human rationales improve machine explanations","authors":"Julia Strout, Ye Zhang, Raymond J Mooney","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wallace et al., 2010a] Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. Active learning for biomedi- cal citation screening. In KDD, pages 173-182, 2010.","title":"Active learning for biomedical citation screening","authors":"Wallace","year":"2010","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wallace et al., 2010b] Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. Semi- automated screening of biomedical citations for systematic re- views. BMC bioinformatics, 11(1):55, 2010.","title":"Semiautomated screening of biomedical citations for systematic reviews","authors":"Wallace","year":"2010","venue":"BMC bioinformatics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wallace et al., 2016] Byron C. Wallace, Joël Kuiper, Aakash Sharma, Mingxi (Brian) Zhu, and Iain Marshall. Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision. Journal of Machine Learning Research, 17(132):1- 25, 2016.","title":"Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision","authors":"Byron C Wallace, Joël Kuiper, Aakash Sharma, Iain Mingxi (brian) Zhu, Marshall","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wallace et al., 2017] Byron C Wallace, Anna Noel-Storr, Iain J Marshall, Aaron M Cohen, Neil R Smalheiser, and James Thomas. Identifying reports of randomized controlled trials (rcts) via a hybrid machine learning and crowdsourcing ap- proach. Journal of the American Medical Informatics Associa- tion, 24(6):1165-1168, 2017.","title":"Identifying reports of randomized controlled trials (rcts) via a hybrid machine learning and crowdsourcing approach","authors":"Anna Byron C Wallace, Iain J Noel-Storr, Aaron M Marshall, Neil R Cohen, James Smalheiser, Thomas","year":"2017","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2019] Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, Dan Roth, and David McAllester. Evidence sentence extraction for machine reading comprehension. arXiv preprint arXiv:1902.08852, 2019.","title":"Evidence sentence extraction for machine reading comprehension","authors":"Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, Dan Roth, David Mcallester","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang et al., 2019] Yinfei Yang, Oshin Agarwal, Chris Tar, By- ron C. Wallace, and Ani Nenkova. Predicting annotation diffi- culty to improve task routing and model performance for biomed- ical information extraction. In NAACL, pages 1471-1480, June 2019.","title":"Predicting annotation difficulty to improve task routing and model performance for biomedical information extraction","authors":"Yinfei Yang, Oshin Agarwal, Chris Tar, Byron C Wallace, Ani Nenkova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zaidan et al., 2007] Omar Zaidan, Jason Eisner, and Christine Pi- atko. Using \"annotator rationales\" to improve machine learning for text categorization. In NAACL, pages 260-267, 2007.","title":"Using \"annotator rationales\" to improve machine learning for text categorization","authors":"Omar Zaidan, Jason Eisner, Christine Piatko","year":"2007","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2016] Ye Zhang, Iain Marshall, and Byron C Wal- lace. Rationale-augmented convolutional neural networks for text classification. In EMNLP, page 795, 2016.","title":"Rationale-augmented convolutional neural networks for text classification","authors":"Zhang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}