{"type":"ScholarlyArticle","title":"SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning","authors":[{"name":"Baihan Lin","slug":"baihan-lin","given":"Baihan","family":"Lin","affiliation":"Columbia University , New York , NY 10027 , USA"},{"name":"Guillermo Cecchi","slug":"guillermo-cecchi","given":"Guillermo","family":"Cecchi","affiliation":"IBM TJ Watson Research Center , Yorktown Heights , NY 10598 , USA"},{"name":"Djallel Bouneffouf","slug":"djallel-bouneffouf","given":"Djallel","family":"Bouneffouf","affiliation":"IBM TJ Watson Research Center , Yorktown Heights , NY 10598 , USA"}],"abstract":"We present a novel recommendation system designed to provide real-time treatment strategies to therapists during psychotherapy sessions. Our system utilizes a turn-level rating mechanism that forecasts the therapeutic outcome by calculating a similarity score between the profound representation of a scoring inventory and the patient's current spoken sentence. By transcribing and segmenting the continuous audio stream into patient and therapist turns, our system conducts immediate evaluation of their therapeutic working alliance. The resulting dialogue pairs, along with their computed working alliance ratings, are then utilized in a deep reinforcement learning recommendation system. In this system, the sessions are treated as users, while the topics are treated as items. To showcase the system's effectiveness, we not only evaluate its performance using an existing dataset of psychotherapy sessions but also demonstrate its practicality through a web app. Through this demo, we aim to provide a tangible and engaging experience of our recommendation system in action.","doi":"10.24963/ijcai.2023/837","url":"https://ijcai.org/proceedings/2023/837","pdf":"https://ijcai.org/proceedings/2023/0837.pdf","markdown":"https://ijcai.org/proceedings/2023/837.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/837","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Demo Track","track":"Demo Track","number":837,"pages":{"first":7149,"last":7153},"keywords":["Natural Language Processing: NLP: Dialogue and interactive systems","Humans and AI: HAI: Applications","Data Mining: DM: Recommender systems","Humans and AI: HAI: Brain sciences","Humans and AI: HAI: Human-computer interaction","Multidisciplinary Topics and Applications: MDA: Health and medicine","Humans and AI: HAI: Computational sustainability and human wellbeing","Humans and AI: HAI: Intelligent user interfaces","Natural Language Processing: NLP: Applications"],"videos":[],"references":[{"position":1,"raw":"[Adorf, 2013] Julius Adorf. Web speech api. KTH Royal Institute of Technology, pages 1-11, 2013.","title":"Web speech api","authors":"Adorf","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aggarwal and others, 2016] Charu C Aggarwal et al. Rec- ommender systems, volume 1. Springer, 2016.","title":"Recommender systems","authors":"C Charu, Aggarwal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Basu et al., 1998] Chumki Basu, Haym Hirsh, William Co- hen, et al. Recommendation as classification: Using so- cial and content-based information in recommendation. In Aaai/iaai, pages 714-720, 1998.","title":"Recommendation as classification: Using social and content-based information in recommendation","authors":"Basu","year":"1998","venue":"Aaai/iaai","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on ma- chine learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on ma- chine learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collab- orative filtering. In Proceedings of the 26th international conference on world wide web, pages 173-182, 2017.","title":"Neural collaborative filtering","authors":"He","year":"2017","venue":"Proceedings of the 26th international conference on world wide web","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Horvath, 1981] Adam O Horvath. An exploratory study of the working alliance: Its measurement and relationship to therapy outcome. PhD thesis, University of British Columbia, 1981.","title":"An exploratory study of the working alliance: Its measurement and relationship to therapy outcome","authors":"O Adam, Horvath","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koren et al., 2022] Yehuda Koren, Steffen Rendle, and Robert Bell. Advances in collaborative filtering. Recom- mender systems handbook, pages 91-142, 2022.","title":"Advances in collaborative filtering","authors":"Koren","year":"2022","venue":"Recommender systems handbook","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In In- ternational conference on machine learning, pages 1188- 1196. PMLR, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural atten- tive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1419-1428, 2017.","title":"Neural attentive session-based recommendation","authors":"Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, Jun Ma","year":"2017","venue":"Proceedings of the 2017 ACM on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin and Zhang, 2020a] Baihan Lin and Xinxin Zhang. Speaker diarization as a fully online learning problem in minivox. arXiv preprint arXiv:2006.04376, 2020.","title":"Speaker diarization as a fully online learning problem in minivox","authors":"Lin, Zhang, Baihan Lin, Xinxin Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Zhang, 2020b] Baihan Lin and Xinxin Zhang. Voiceid on the fly: A speaker recognition system that learns from scratch. In INTERSPEECH, 2020.","title":"Voiceid on the fly: A speaker recognition system that learns from scratch","authors":"Lin, Zhang, Baihan Lin, Xinxin Zhang","year":"2020","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin and Zhang, 2021] Baihan Lin and Xinxin Zhang. Speaker diarization as a fully online bandit learning prob- lem in minivox. In ACML, pages 1660-1674, 2021.","title":"Speaker diarization as a fully online bandit learning problem in minivox","authors":"Lin, Zhang, Baihan Lin, Xinxin Zhang","year":"2021","venue":"ACML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2020a] Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. A story of two streams: Reinforcement learning models from human be- havior and neuropsychiatry. In Proceedings of the 19th International Conference on Autonomous Agents and Mul- tiAgent Systems, pages 744-752, 2020.","title":"A story of two streams: Reinforcement learning models from human behavior and neuropsychiatry","authors":"Lin","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and Mul-tiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2020b] Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. Unified models of human behavioral agents in bandits, contextual bandits and rl. arXiv preprint arXiv:2005.04544, 2020.","title":"Unified models of human behavioral agents in bandits, contextual bandits and rl","authors":"Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2021] Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. Models of hu- man behavioral agents in bandits, contextual bandits and rl. In International Workshop on Human Brain and Artifi- cial Intelligence, pages 14-33. Springer, 2021.","title":"Models of human behavioral agents in bandits, contextual bandits and rl","authors":"Lin","year":"2021","venue":"International Workshop on Human Brain and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2022a] Baihan Lin, Djallel Bouneffouf, and Guillermo Cecchi. Predicting human decision making in psychological tasks with recurrent neural networks. PloS one, 2022.","title":"Predicting human decision making in psychological tasks with recurrent neural networks","authors":"Lin","year":"2022","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2022b] Baihan Lin, Guillermo Cecchi, and Djal- lel Bouneffouf. Working alliance transformer for psychotherapy dialogue classification. arXiv preprint arXiv:2210.15603, 2022.","title":"Working alliance transformer for psychotherapy dialogue classification","authors":"Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2023a] Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, and Ravi Tejwani. Neural topic modeling of psychotherapy sessions. In International Workshop on Health Intelligence. Springer, 2023.","title":"Neural topic modeling of psychotherapy sessions","authors":"Lin","year":"2023","venue":"International Workshop on Health Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2023b] Baihan Lin, Guillermo Cecchi, and Djal- lel Bouneffouf. Deep annotation of therapeutic working alliance in psychotherapy. In International Workshop on Health Intelligence. Springer, 2023.","title":"Deep annotation of therapeutic working alliance in psychotherapy","authors":"Lin","year":"2023","venue":"International Workshop on Health Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2023c] Baihan Lin, Guillermo Cecchi, and Djal- lel Bouneffouf. Helping therapists with nlp-annotated rec- ommendation. In Joint Proceedings of the ACM IUI Work- shops, 2023.","title":"Helping therapists with nlp-annotated recommendation","authors":"Lin","year":"2023","venue":"Joint Proceedings of the ACM IUI Workshops","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lin et al., 2023d] Baihan Lin, Guillermo Cecchi, and Djal- lel Bouneffouf. Psychotherapy AI companion with re- inforcement learning recommendations and interpretable policy dynamics. In Proceedings of the Web Conference 2023, 2023.","title":"Psychotherapy AI companion with reinforcement learning recommendations and interpretable policy dynamics","authors":"Lin","year":"2023","venue":"Proceedings of the Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lin et al., 2023e] Baihan Lin, Stefan Zecevic, Djallel Boun- effouf, and Guillermo Cecchi. Therapyview: Visualiz- ing therapy sessions with temporal topic modeling and ai- generated arts. arXiv preprint arXiv:2302.10845, 2023.","title":"Therapyview: Visualizing therapy sessions with temporal topic modeling and aigenerated arts","authors":"Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lin, 2022a] Baihan Lin. Computational inference in cogni- tive science: Operational, societal and ethical considera- tions. arXiv preprint arXiv:2210.13526, 2022.","title":"Computational inference in cognitive science: Operational, societal and ethical considerations","authors":"Lin ;, Baihan Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin, 2022b] Baihan Lin. Knowledge management system with nlp-assisted annotations: A brief survey and outlook. In CIKM Workshops, 2022.","title":"Knowledge management system with nlp-assisted annotations: A brief survey and outlook","authors":"Lin ;, Baihan Lin","year":"2022","venue":"CIKM Workshops","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lin, 2022c] Baihan Lin. Reinforcement learning and ban- dits for speech and language processing: Tutorial, review and outlook. arXiv preprint arXiv:2210.13623, 2022.","title":"Reinforcement learning and bandits for speech and language processing: Tutorial, review and outlook","authors":"Lin ;, Baihan Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lin, 2022d] Baihan Lin. Voice2Alliance: automatic speaker diarization and quality assurance of conversational align- ment. In INTERSPEECH, 2022.","title":"Voice2Alliance: automatic speaker diarization and quality assurance of conversational alignment","authors":"Lin ;, Baihan Lin","year":"2022","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ludewig and Jannach, 2018] Malte Ludewig and Dietmar Jannach. Evaluation of session-based recommendation al- gorithms. User Modeling and User-Adapted Interaction, 28(4):331-390, 2018.","title":"Evaluation of session-based recommendation algorithms","authors":"Malte Ludewig, Dietmar Jannach","year":"2018","venue":"User Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Martin et al., 2000] Daniel J Martin, John P Garske, and M Katherine Davis. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. Journal of consulting and clinical psychology, 68(3):438, 2000.","title":"Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review","authors":"John P Daniel J Martin, M Katherine Garske, Davis","year":"2000","venue":"Journal of consulting and clinical psychology","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Patel et al., 2018] Vikram Patel, Shekhar Saxena, Crick Lund, Graham Thornicroft, Florence Baingana, Paul Bolton, Dan Chisholm, Pamela Y Collins, Janice L Cooper, Julian Eaton, et al. The lancet commission on global mental health and sustainable development. The lancet, 392(10157):1553-1598, 2018.","title":"The lancet commission on global mental health and sustainable development","authors":"Vikram Patel, Shekhar Saxena, Crick Lund, Graham Thornicroft, Florence Baingana, Paul Bolton, Dan Chisholm, Pamela Y Collins, Janice L Cooper, Julian Eaton","year":"2018","venue":"The lancet","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Pazzani and Billsus, 2007] Michael J Pazzani and Daniel Billsus. Content-based recommendation systems. In The adaptive web, pages 325-341. Springer, 2007.","title":"Content-based recommendation systems","authors":"J Michael, Daniel Pazzani, Billsus","year":"2007","venue":"The adaptive web","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sarwar et al., 2001] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filter- ing recommendation algorithms. In Proceedings of inter- national conference on World Wide Web, pages 285-295, 2001.","title":"Item-based collaborative filtering recommendation algorithms","authors":"Badrul Sarwar, George Karypis, Joseph Konstan, John Riedl","year":"2001","venue":"Proceedings of international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Street, 2023] Alexander Street. counseling and psychother- apy transcripts series. Alexander Street Publishing, 2023.","title":"counseling and psychotherapy transcripts series","authors":"Alexander Street","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Su and Khoshgoftaar, 2009] Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering tech- niques. Advances in artificial intelligence, 2009, 2009.","title":"A survey of collaborative filtering techniques","authors":"Xiaoyuan Su, M Taghi, Khoshgoftaar","year":"2009","venue":"Advances in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tracey and Kokotovic, 1989] Terence J Tracey and Anna M Kokotovic. Factor structure of the working alliance inven- tory. Psychological Assessment: A journal of consulting and clinical psychology, 1(3):207, 1989.","title":"Factor structure of the working alliance inventory","authors":"J Terence, Anna M Tracey, Kokotovic","year":"1989","venue":"Psychological Assessment: A journal of consulting and clinical psychology","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wampold, 2015] Bruce E Wampold. How important are the common factors in psychotherapy? an update. World Psych, 14:270-277, 2015.","title":"How important are the common factors in psychotherapy? an update","authors":"Bruce E Wampold","year":"2015","venue":"World Psych","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2014] Xinxi Wang, Yi Wang, David Hsu, and Ye Wang. Exploration in interactive personalized mu- sic recommendation: a reinforcement learning approach. ACM TOMM, 11(1):1-22, 2014.","title":"Exploration in interactive personalized music recommendation: a reinforcement learning approach","authors":"Wang","year":"2014","venue":"ACM TOMM","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2020] Rui Wang, Xuemeng Hu, Deyu Zhou, Yulan He, Yuxuan Xiong, Chenchen Ye, and Haiyang Xu. Neural topic modeling with bidirectional adversarial train- ing. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 340-350, 2020.","title":"Neural topic modeling with bidirectional adversarial training","authors":"Rui Wang, Xuemeng Hu, Deyu Zhou, Yulan He, Yuxuan Xiong, Chenchen Ye, Haiyang Xu","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Watkins Jr, 2013] C Edward Watkins Jr. Being and be- coming a psychotherapy supervisor: The crucial triad of learning difficulties. American Journal of Psychotherapy, 67(2):134-150, 2013.","title":"Being and becoming a psychotherapy supervisor: The crucial triad of learning difficulties","authors":"Edward Watkins","year":"2013","venue":"American Journal of Psychotherapy","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wu et al., 2019] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In Proceed- ings of AAAI, volume 33, pages 346-353, 2019.","title":"Session-based recommendation with graph neural networks","authors":"Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, Tieniu Tan","year":"2019","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zheng et al., 2018] Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Drn: A deep reinforcement learning frame- work for news recommendation. In Proceedings of the 2018 world wide web conference, pages 167-176, 2018.","title":"Drn: A deep reinforcement learning framework for news recommendation","authors":"Zheng","year":"2018","venue":"Proceedings of the 2018 world wide web conference","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zou et al., 2020] Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. Pseudo dyna-q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 816-824, 2020.","title":"Pseudo dyna-q: A reinforcement learning framework for interactive recommendation","authors":"Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, Dawei Yin","year":"2020","venue":"Proceedings of the 13th International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}