{"type":"ScholarlyArticle","title":"Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach (Extended Abstract)","authors":[{"name":"Ashish Sharma","slug":"ashish-sharma","given":"Ashish","family":"Sharma","affiliation":"School of Computer Science & Engineering , University of Washington"},{"name":"Inna W. Lin","slug":"inna-w-lin","given":"Inna W.","family":"Lin","affiliation":"School of Computer Science & Engineering , University of Washington"},{"name":"Adam S. Miner","slug":"adam-s-miner","given":"Adam S.","family":"Miner","affiliation":"Department of Psychiatry and Behavioral Sciences , Stanford University; Center for Biomedical Informatics Research , Stanford University"},{"name":"Dave C. Atkins","slug":"dave-c-atkins","given":"Dave C.","family":"Atkins","affiliation":"Department of Psychiatry and Behavioral Sciences , University of Washington"},{"name":"Tim Althoff","slug":"tim-althoff","given":"Tim","family":"Althoff","affiliation":"School of Computer Science & Engineering , University of Washington"}],"abstract":"Online peer-to-peer support platforms enable conversations between millions of people who seek and provide mental health support. If successful, web-based mental health conversations could improve access to treatment and reduce the global disease burden. Psychologists have repeatedly demonstrated that empathy, the ability to understand and feel the emotions and experiences of others, is a key component leading to positive outcomes in supportive conversations. However, recent studies have shown that highly empathic conversations are rare in online mental health platforms. \r\n\r\nIn this paper, we work towards improving empathy in online mental health support conversations. We introduce a new task of empathic rewriting which aims to transform low-empathy conversational posts to higher empathy. Learning such transformations is challenging and requires a deep understanding of empathy while maintaining conversation quality through text fluency and specificity to the conversational context. Here we propose Partner, a deep reinforcement learning (RL) agent that learns to make sentence-level edits to posts in order to increase the expressed level of empathy while maintaining conversation quality. Our RL agent leverages a policy network, based on a transformer language model adapted from GPT-2, which performs the dual task of generating candidate empathic sentences and adding those sentences at appropriate positions. Through a combination of automatic and human evaluation, we demonstrate that Partner successfully generates more empathic, specific, and diverse responses and outperforms NLP methods from related tasks such as style transfer and empathic dialogue generation.","doi":"10.24963/ijcai.2022/747","url":"https://ijcai.org/proceedings/2022/747","pdf":"https://ijcai.org/proceedings/2022/0747.pdf","markdown":"https://ijcai.org/proceedings/2022/747.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/747","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":747,"pages":{"first":5339,"last":5343},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987257","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985400","length_sec":717,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bohart et al., 2002] Arthur Bohart, Robert Elliott, Leslie Greenberg, and Jeanne Watson. Empathy. J. C. Norcross (Ed.), 2002.","title":"Empathy","authors":"Arthur Bohart, Robert Elliott, Leslie Greenberg, Jeanne Watson, ; J C Norcross","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown et al., 2020] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.","title":"Language models are few-shot learners","authors":"Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buechel et al., 2018] Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. Modeling empathy and distress in reaction to news stories. In EMNLP, 2018.","title":"Modeling empathy and distress in reaction to news stories","authors":"Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, João Sedoc","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clark et al., 2018] Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. Creative writing with a machine in the loop: Case studies on slo- gans and stories. In IUI, 2018.","title":"Creative writing with a machine in the loop: Case studies on slogans and stories","authors":"Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, Noah A Smith","year":"2018","venue":"IUI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Collins et al., 2011] Pamela Y Collins, Vikram Patel, Sarah S Joestl, Dana March, Thomas R Insel, Abdallah S Daar, Isabel A Bordin, E Jane Costello, Maureen Durkin, Christopher Fairburn, et al. Grand challenges in global mental health. Nature, 2011.","title":"Grand challenges in global mental health","authors":"Pamela Y Collins, Vikram Patel, Sarah S Joestl, Dana March, Thomas R Insel, S Abdallah, Isabel A Daar, Jane Bordin, Maureen Costello, Christopher Durkin, Fairburn","year":"2011","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elliott et al., 2011] Robert Elliott, Arthur Bohart, Jeanne Watson, and Leslie S Greenberg. Empathy. Psychother- apy, 2011.","title":"Empathy. Psychotherapy","authors":"Robert Elliott, Arthur Bohart, Jeanne Watson, Leslie S Greenberg","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elliott et al., 2018] Robert Elliott, Arthur Bohart, Jeanne Watson, and David Murphy. Therapist empathy and client outcome: An updated meta-analysis. Psychotherapy, 2018.","title":"Therapist empathy and client outcome: An updated meta-analysis","authors":"Robert Elliott, Arthur Bohart, Jeanne Watson, David Murphy","year":"2018","venue":"Psychotherapy","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2019] Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. A probabilistic formulation of unsupervised text style transfer. In ICLR, 2019.","title":"A probabilistic formulation of unsupervised text style transfer","authors":"He","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Holtzman et al., 2020] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neu- ral text degeneration. In ICLR, 2020.","title":"The curious case of neural text degeneration","authors":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, and et al. Goyal, Naman. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.","title":"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In NAACL-HLT, 2016.","title":"A diversity-promoting objective function for neural conversation models","authors":"Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020] Ron C Li, Steven M Asch, and Nigam H Shah. Developing a delivery science for artificial intelli- gence in healthcare. NPJ Digital Medicine, 2020.","title":"Developing a delivery science for artificial intelligence in healthcare","authors":"Ron C Li, Steven M Asch, H Nigam, Shah","year":"2020","venue":"NPJ Digital Medicine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luxton et al., 2012] David Luxton, Jennifer June, and Jonathan Fairall. Social media and suicide: a public health perspective. American journal of public health, 2012.","title":"Social media and suicide: a public health perspective","authors":"David Luxton, Jennifer June, Jonathan Fairall","year":"2012","venue":"American journal of public health","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Majumder et al., 2020] Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexan- der Gelbukh, Rada Mihalcea, and Soujanya Poria. Mime: Mimicking emotions for empathetic response generation. In EMNLP, 2020.","title":"Mime: Mimicking emotions for empathetic response generation","authors":"Majumder","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miner et al., 2019] Adam Miner, Nigam Shah, Kim Bul- lock, Bruce Arnow, Jeremy Bailenson, and Jeff Hancock. Key considerations for incorporating conversational ai in psychotherapy. Frontiers in psychiatry, 10, 2019.","title":"Key considerations for incorporating conversational ai in psychotherapy","authors":"Adam Miner, Nigam Shah, Kim Bullock, Bruce Arnow, Jeremy Bailenson, Jeff Hancock","year":"2019","venue":"Frontiers in psychiatry","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Olfson, 2016] Mark Olfson. Building the mental health workforce capacity needed to treat adults with serious mental illnesses. Health Affairs, 2016.","title":"Building the mental health workforce capacity needed to treat adults with serious mental illnesses","authors":"Mark Olfson","year":"2016","venue":"Health Affairs","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rashkin et al., 2019] Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: A new benchmark and dataset. In ACL, 2019.","title":"Towards empathetic open-domain conversation models: A new benchmark and dataset","authors":"Eric Michael Hannah Rashkin, Margaret Smith, Y-Lan Li, Boureau","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Robert et al., 2011] Elliot Robert, Arthur Bohart, JC Wat- son, and LS Greenberg. Empathy. Psychotherapy, 2011.","title":"Empathy. Psychotherapy","authors":"Elliot Robert, Arthur Bohart, J C Watson, Greenberg","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Selman, 1980] Robert L Selman. Growth of interpersonal understanding. Academic Press, 1980.","title":"Growth of interpersonal understanding","authors":"Robert L Selman","year":"1980","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sharma et al., 2020] Ashish Sharma, Adam Miner, Dave Atkins, and Tim Althoff. A computational approach to un- derstanding empathy expressed in text-based mental health support. In EMNLP, 2020.","title":"A computational approach to understanding empathy expressed in text-based mental health support","authors":"Ashish Sharma, Adam Miner, Dave Atkins, Tim Althoff","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2021] Ashish Sharma, Inna Lin, Adam Miner, Dave Atkins, and Tim Althoff. Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach. In TheWebConf, 2021.","title":"Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach","authors":"Ashish Sharma, Inna Lin, Adam Miner, Dave Atkins, Tim Althoff","year":"2021","venue":"TheWebConf","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tanana et al., 2019] Michael Tanana, Christina Soma, Vivek Srikumar, David Atkins, and Zac Imel. De- velopment and evaluation of clientbot: Patient-like conversational agent to train basic counseling skills. JMIR, 2019.","title":"Development and evaluation of clientbot: Patient-like conversational agent to train basic counseling skills","authors":"Michael Tanana, Christina Soma, Vivek Srikumar, David Atkins, Zac Imel","year":"2019","venue":"JMIR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[White and Dorman, 2001] Marsha White and Steve M Dor- man. Receiving social support online: implications for health education. Health education research, 2001.","title":"Receiving social support online: implications for health education","authors":"Marsha White, Steve M Dorman","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response gener- ation. In ACL, system demonstration, 2020.","title":"Dialogpt: Large-scale generative pre-training for conversational response generation","authors":"Zhang","year":"2020","venue":"ACL, system demonstration","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}