{"type":"ScholarlyArticle","title":"Evolving AI from Research to Real Life – Some Challenges and Suggestions","authors":[{"name":"Sandya Mannarswamy","slug":"sandya-mannarswamy","given":"Sandya","family":"Mannarswamy","affiliation":"Conduent Labs India"},{"name":"Shourya Roy","slug":"shourya-roy","given":"Shourya","family":"Roy","affiliation":"American Express Big Data Labs"}],"abstract":"Artificial Intelligence (AI) has come a long way from the stages of being just scientific fiction or academic research curiosity to a point, where it is poised to impact human life significantly. AI driven applications such as autonomous vehicles, medical diagnostics, conversational agents etc. are becoming a reality. In this position paper, we argue that there are certain challenges AI still needs to overcome in its evolution from Research to Real Life. We outline some of these challenges and our suggestions to address them. We provide pointers to similar issues and their resolutions in disciplines such as psychology and medicine from which AI community can leverage the learning. More importantly, this paper is intended to focus the attention of AI research community on translating AI research efforts into real world deployments.","doi":"10.24963/ijcai.2018/717","url":"https://ijcai.org/proceedings/2018/717","pdf":"https://ijcai.org/proceedings/2018/0717.pdf","markdown":"https://ijcai.org/proceedings/2018/717.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/717","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Evolution of the contours of AI","track":"Evolution of the contours of AI","number":717,"pages":{"first":5172,"last":5179},"keywords":["Machine Learning: Deep Learning","Machine Learning: Experimental Methodology;Replicability"],"videos":[],"references":[{"position":1,"raw":"[A.Campolo, 2017] A.Campolo. AI Now 2017 Report. pages 1-4, 2017. https://ainowinstitute.org/AI Now 2017 Report.pdf.","title":"AI Now","authors":"A Campolo","year":"2017","venue":null,"doi":null,"url":"https://ainowinstitute.org/AINow2017Report.pdf","internal_url":null},{"position":2,"raw":"[Brooks, 2017] Rodney Brooks. The seven deadly sins of predicting the future of AI. MIT Technology Review, pages 1-4, 2017. https://tinyurl.com/y7z5wrl9.","title":"The seven deadly sins of predicting the future of AI","authors":"Rodney Brooks, Brooks","year":"2017","venue":"MIT Technology Review","doi":null,"url":"https://tinyurl.com/y7z5wrl9","internal_url":null},{"position":3,"raw":"[Chalmers et al., 1992] Iain Chalmers, Kay Dickersin, and Thierry Coquand Chalmers. Getting to grips with Archie Cochrane's agenda. BMJ, 305 6857:786-8, 1992.","title":"Getting to grips with Archie Cochrane's agenda","authors":"Iain Chalmers, Kay Dickersin, Thierry Coquand, Chalmers","year":"1992","venue":"BMJ","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2016] Ying Chen, JD Elenee Argentinis, and Griff Weber. IBM Watson: How cognitive computing can be applied to life sciences research. Clinical Therapeutics, 38:688-701, 2016.","title":"IBM Watson: How cognitive computing can be applied to life sciences research","authors":"Ying Chen, , Jd Elenee Argentinis, Griff Weber","year":"2016","venue":"Clinical Therapeutics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chollet, 2017] Francois Chollet. Deep Learning with Python. Manning Publications Co., Greenwich, CT, USA, 1st edition, 2017.","title":"Deep Learning with Python","authors":"Francois Chollet","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Kenneth Ward Church. Emerging trends: I did it, I did it, I did it, but. . . Natural Language Engineer- ing, 23(3):473-480, 2017.","title":"Emerging trends: I did it, I did it, I did it, but","authors":"Kenneth Ward Church","year":"2017","venue":"Natural Language Engineering","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diakopoulos, 2016] Nicholas Diakopoulos. Accountability in algorithmic decision making. Communications of the ACM, 59(2):56-62, January 2016.","title":"Accountability in algorithmic decision making","authors":"Nicholas Diakopoulos","year":"2016","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2017] Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1150-1159, 2017. [Editorial, 2017] Editorial. Promoting reproducibility with registered reports. Nature Human Behavior, pages 1-2, 2017. https://www.nature.com/articles/s41562-016-0034.","title":"Visualizing and understanding neural machine translation","authors":"Ding","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","doi":null,"url":"https://www.nature.com/articles/s41562-016-0034","internal_url":null},{"position":9,"raw":"[Greenfield and Rich, 2012] Sheldon Greenfield and Eu- gene C. Rich. Welcome to the journal of comparative ef- fectiveness research. Journal of comparative effectiveness research, 1:1-3, 2012.","title":"Sheldon Greenfield and Eugene C. Rich. Welcome to the journal of comparative effectiveness research","authors":"Rich Greenfield","year":"2012","venue":"Journal of comparative effectiveness research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulshan et al., 2016] Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Wid- ner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316 22:2402-2410, 2016.","title":"Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs","authors":"Lily Varun Gulshan, Marc Peng, Martin C Coram, Derek Stumpe, Arunachalam Wu, Subhashini Narayanaswamy, Kasumi Venugopalan, Tom Widner, Jorge Madams, Ramasamy Cuadros, Rajiv Kim, Philip C Raman, Jessica L Nelson, Dale R Mega, Webster","year":"2016","venue":"JAMA","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gundersen and Kjensmo, 2018] Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18), pages 1- 8, 2018.","title":"Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence","authors":"Kjensmo Gundersen","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guyatt et al., 1992] Gordon Guyatt, John Cairns, and David Churchill. Evidence-based medicine: A new approach to teaching the practice of medicine. JAMA, 268(17):2420- 2425, 1992.","title":"Evidence-based medicine: A new approach to teaching the practice of medicine","authors":"Guyatt","year":"1992","venue":"JAMA","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hendler, 2008] James Hendler. Avoiding another AI winter. Intelligent Systems, IEEE, 23:2-4, 04 2008.","title":"Avoiding another AI winter","authors":"James Hendler","year":"2008","venue":"Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hickok, 1997] Durlin E. Hickok. Outcomes-based research: What is it and how do we do it? Seminars in Perinatology, 21(6):467 -471, 1997.","title":"Outcomes-based research: What is it and how do we do it?","authors":"E Durlin, Hickok","year":"1997","venue":"Seminars in Perinatology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hutson, 2018] Matthew Hutson. Missing data hinder repli- cation of artificial intelligence studies. Science, pages 1-4, 2018.","title":"Missing data hinder replication of artificial intelligence studies","authors":"Matthew Hutson","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ilyas et al., 2017] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Query-efficient black-box adver- sarial examples. CoRR, abs/1712.07113:1-8, 2017.","title":"Query-efficient black-box adversarial examples","authors":"Andrew Ilyas, Logan Engstrom, Anish Athalye, Jessy Lin","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ioannidis, 2005] John P. A. Ioannidis. Why most published research findings are false. PLoS Med, 2(8):e124, 2005.","title":"Why most published research findings are false","authors":"P A John, Ioannidis","year":"2005","venue":"PLoS Med","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- ial examples for evaluating reading comprehension sys- tems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021- 2031. Association for Computational Linguistics, 2017.","title":"Adversarial examples for evaluating reading comprehension systems","authors":"Liang Jia, Robin Jia, Percy Liang","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jo and Bengio, 2017] Jason Jo and Yoshua Bengio. Mea- suring the tendency of CNNs to learn surface statistical regularities. CoRR, abs/1711.11561:1-8, 2017.","title":"Measuring the tendency of CNNs to learn surface statistical regularities","authors":"Jo, Bengio ; ] Jason, Jo, Yoshua Bengio","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kawaguchi et al., 2017] Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. CoRR, abs/1710.05468:1-8, 2017.","title":"Generalization in deep learning","authors":"Kenji Kawaguchi, Leslie Pack Kaelbling, Yoshua Bengio","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Koh and Liang, 2017] Pang Wei Koh and Percy Liang. Un- derstanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1885-1894, 2017.","title":"Understanding black-box predictions via influence functions","authors":"Pang Wei, Koh, Percy Liang","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lighthill, 1973] James Lighthill. Artificial intelligence: A general survey. In Artificial Intelligence: a paper sympo- sium, pages 1-21, 1973.","title":"Artificial intelligence: A general survey","authors":"James Lighthill","year":"1973","venue":"Artificial Intelligence: a paper symposium","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lipton, 2016] Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.03490:1-8, 2016.","title":"The mythos of model interpretability","authors":"Zachary Chase, Lipton","year":"2016","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Luce et al., 2010] Bryan Luce, Michael Drummond, Bengt J Ã ¶nsson, Peter Neumann, J Sanford Schwartz, Uwe Siebert, and Sean D Sullivan. EBM, HTA, and CER: clear- ing the confusion. The Milbank quarterly, 88:256-76, 06 2010.","title":"and CER: clearing the confusion","authors":"Bryan Luce, Michael Drummond, Bengt J Ã ¶nsson, Peter Neumann, Sanford Schwartz, Uwe Siebert, Sean D Sullivan, Ebm, Hta","year":"2010","venue":"The Milbank quarterly","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lucic et al., 2017] M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are GANs Created Equal? A Large-Scale Study. ArXiv e-prints, pages 1-8, 2017.","title":"Are GANs Created Equal? A Large-Scale Study","authors":"M Lucic, K Kurach, M Michalski, S Gelly, O Bousquet","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Manning, 2015] Christopher D. Manning. Computational linguistics and deep learning. Computational Linguistics, 41:701-707, 2015.","title":"Computational linguistics and deep learning","authors":"D Christopher, Manning","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Marcus, 2018] Gary Marcus. Deep learning: A critical ap- praisal. CoRR, abs/1801.00631:1-27, 2018.","title":"Deep learning: A critical appraisal","authors":"Gary Marcus","year":"2018","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mauro, 2017] Gordon Mauro. Six graphs to understand the state of AI academic research. pages 1-4, 2017. https: //blog.ai-academy.com/.","title":"Six graphs to understand the state of AI academic research","authors":"Gordon Mauro","year":"2017","venue":null,"doi":null,"url":"https://blog.ai-academy.com/","internal_url":null},{"position":29,"raw":"[Miller et al., 2017] Alexander Miller, Will Feng, Dhruv Ba- tra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 79-84. Association for Computa- tional Linguistics, 2017.","title":"ParlAI: A dialog research software platform","authors":"Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, Jason Weston","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mulcahy, 2017] Nick Mulcahy. Big Data bust: MD Anderson-Watson project dies. Medscape, pages 1-3, 2017.","title":"Big Data bust: MD Anderson-Watson project dies","authors":"Nick Mulcahy","year":"2017","venue":"Medscape","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Munafò et al., 2017] Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christo- pher D. Chambers, Nathalie Percie du Sert, Uri Simon- sohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. A manifesto for reproducible science. Na- ture Human Behaviour, 1:0021+, 2017.","title":"A manifesto for reproducible science","authors":"Marcus R Munafò, Brian A Nosek, Dorothy V M Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie Du, Uri Sert, Eric-Jan Simonsohn, Jennifer J Wagenmakers, John P A Ware, Ioannidis","year":"2017","venue":"Nature Human Behaviour","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Neyshabur et al., 2017] Behnam Neyshabur, Srinadh Bho- janapalli, David Mcallester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural In- formation Processing Systems 30, pages 5947-5956, 2017.","title":"Exploring generalization in deep learning","authors":"Srinadh Behnam Neyshabur, David Bhojanapalli, Nati Mcallester, Srebro","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Nichol et al., 2018] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR, abs/1804.03720, 2018.","title":"Gotta learn fast: A new benchmark for generalization in RL","authors":"Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Olah et al., 2018] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, pages 1-8, 2018. https://distill.pub/2018/building-blocks.","title":"The building blocks of interpretability","authors":"Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, Alexander Mordvintsev","year":"2018","venue":"Distill","doi":null,"url":"https://distill.pub/2018/building-blocks","internal_url":null},{"position":35,"raw":"[O'Neil, 2016] Cathy O'Neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democ- racy. Crown Publishing Group, New York, NY, USA, 2016.","title":"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy","authors":"O' Cathy, Neil","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Parker, 2006] Ruth M. Parker. What an informed patient means for the future of healthcare. PharmacoEconomics, pages 29-33, 2006.","title":"What an informed patient means for the future of healthcare","authors":"Ruth M Parker","year":"2006","venue":"PharmacoEconomics","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Polanyi, 1966] Michael Polanyi. The Tacit Dimension. Routledge and Kegan Paul, 1966.","title":"The Tacit Dimension","authors":"Michael Polanyi","year":"1966","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[R Deshpande et al., 2011] Prasanna R Deshpande, Su- rulivel Rajan, B Lakshmi Sudeepthi, and C P Abdul Nazir. Patient-Reported Outcomes: A new era in clinical re- search. 2:137-44, 10 2011.","title":"Patient-Reported Outcomes: A new era in clinical research","authors":"Surulivel Prasanna R Deshpande, Lakshmi Rajan, C P Abdul Sudeepthi, Nazir","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Rahimi and Recht, 2017] Ali Rahimi and Ben Recht. Re- flections on random kitchen sinks. pages 1-3, 2017. http: //www.argmin.net/2017/12/05/kitchen-sinks/.","title":"Reflections on random kitchen sinks","authors":"Recht Rahimi, Ali Rahimi, Ben Recht","year":"2017","venue":null,"doi":null,"url":"http://www.argmin.net/2017/12/05/kitchen-sinks/","internal_url":null},{"position":40,"raw":"[Rizwanuddin, 2003] Ahmad Syed Rizwanuddin. Adverse drug event monitoring at the Food and Drug Administra- tion. Journal of General Internal Medicine, 18(1):57-60, 2003.","title":"Adverse drug event monitoring at the Food and Drug Administration","authors":"Ahmad Syed, Rizwanuddin","year":"2003","venue":"Journal of General Internal Medicine","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sculley et al., 2018] D Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. Winner's curse? on pace, progress, and empirical rigor. pages 1-4, 2018. https://openreview.net/forum?id=rJWF0Fywf. [Sedgwick, 2014] Philip Sedgwick. What are the four phases of clinical research trials? 348:1-4, 2014.","title":"Winner's curse? on pace, progress, and empirical rigor","authors":"Jasper Sculley, Alex Snoek, Ali Wiltschko, ; Rahimi, Philip Sedgwick, Sedgwick","year":"2014","venue":null,"doi":null,"url":"https://openreview.net/forum?id=rJWF0Fywf","internal_url":null},{"position":42,"raw":"[Sethi et al., 2018] Akshay Sethi, Anush Sankaran, Naveen Panwar, Shreya Khare, and Senthil Mani. DLPaper2code: Auto-generation of code from deep learning research pa- pers. CoRR, abs/1711.03543:1-8, 2018.","title":"DLPaper2code: Auto-generation of code from deep learning research papers","authors":"Akshay Sethi, Anush Sankaran, Naveen Panwar, Shreya Khare, Senthil Mani","year":"2018","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, and et al. Mastering the game of Go without human knowl- edge. Nature, 2017.","title":"Mastering the game of Go without human knowledge","authors":"David Silver, Julian Schrittwieser","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Stone et al., 2016] Peter Stone, Rodney Brooks, Erik Bryn- jolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, William Press, AnnaLee Saxenian, Julie Shah, Milind Tambe, and As- tro Teller. Ai and life in 2030. Technical report, 2016. http://ai100.stanford.edu/2016-report.","title":"Ai and life in 2030","authors":"Peter Stone, Rodney Brooks, Erik Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, William Press, Annalee Saxenian, Julie Shah, Milind Tambe, Astro Teller","year":"2016","venue":null,"doi":null,"url":"http://ai100.stanford.edu/2016-report","internal_url":null},{"position":45,"raw":"[Su et al., 2017] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. CoRR, abs/1710.08864:1-8, 2017.","title":"One pixel attack for fooling deep neural networks","authors":"Jiawei Su, Danilo Vasconcellos Vargas, Kouichi Sakurai","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Vanschoren et al., 2014] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Net- worked science in machine learning. SIGKDD Exploration Newsletter, 15(2):49-60, June 2014.","title":"OpenML: Networked science in machine learning","authors":"Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, Luis Torgo","year":"2014","venue":"SIGKDD Exploration Newsletter","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Venkatesh et al., 2018] Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Ange- liki Metallinou, Rahul Goel, Shaohua Yang, and Anirudh Raju. On evaluating and comparing conversational agents. CoRR, abs/1801.03625:1-10, 2018.","title":"On evaluating and comparing conversational agents","authors":"Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, Rahul Goel, Shaohua Yang, Anirudh Raju","year":"2018","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Woolf, 2008] Stephen Woolf. The meaning of translational research and why it matters. JAMA, 299(2):211-213, 2008.","title":"The meaning of translational research and why it matters","authors":"Stephen Woolf, Woolf","year":"2008","venue":"JAMA","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yuan et al., 2017] Xiaoyong Yuan, Pan He, Qile Zhu, Ra- jendra Rana Bhat, and Xiaolin Li. Adversarial exam- ples: Attacks and defenses for deep learning. CoRR, abs/1712.07107:1-8, 2017.","title":"Adversarial examples: Attacks and defenses for deep learning","authors":"Xiaoyong Yuan, Pan He, Qile Zhu, Rajendra, Rana Bhat, Xiaolin Li","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2016] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2016.","title":"Understanding deep learning requires rethinking generalization","authors":"Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}