{"type":"ScholarlyArticle","title":"ReBandit: Random Effects Based Online RL Algorithm for Reducing Cannabis Use","authors":[{"name":"Susobhan Ghosh","slug":"susobhan-ghosh","given":"Susobhan","family":"Ghosh","affiliation":"Department of Computer Science , Harvard University"},{"name":"Yongyi Guo","slug":"yongyi-guo","given":"Yongyi","family":"Guo","affiliation":"Department of Statistics , University of Wisconsin-Madison"},{"name":"Pei-Yao Hung","slug":"pei-yao-hung","given":"Pei-Yao","family":"Hung","affiliation":"Institute for Social Research , University of Michigan"},{"name":"Lara Coughlin","slug":"lara-coughlin","given":"Lara","family":"Coughlin","affiliation":"Department of Psychiatry , University of Michigan susobhan"},{"name":"Erin Bonar","slug":"erin-bonar","given":"Erin","family":"Bonar","affiliation":"Department of Psychiatry , University of Michigan susobhan"},{"name":"Inbal Nahum-Shani","slug":"inbal-nahum-shani","given":"Inbal","family":"Nahum-Shani","affiliation":"Institute for Social Research , University of Michigan"},{"name":"Maureen Walton","slug":"maureen-walton","given":"Maureen","family":"Walton","affiliation":"Department of Psychiatry , University of Michigan susobhan"},{"name":"Susan Murphy","slug":"susan-murphy","given":"Susan","family":"Murphy","affiliation":"Department of Computer Science , Harvard University"}],"abstract":"The escalating prevalence of cannabis use, and associated cannabis-use disorder (CUD), poses a significant public health challenge globally. With a notably wide treatment gap, especially among emerging adults (EAs; ages 18-25), addressing cannabis use and CUD remains a pivotal objective within the 2030 United Nations Agenda for Sustainable Development Goals (SDG). In this work, we develop an online reinforcement learning (RL) algorithm called reBandit which will be utilized in a mobile health study to deliver personalized mobile health interventions aimed at reducing cannabis use among EAs. reBandit utilizes random effects and informative Bayesian priors to learn quickly and efficiently in noisy mobile health environments. Moreover, reBandit employs Empirical Bayes and optimization techniques to autonomously update its hyper-parameters online. To evaluate the performance of our algorithm, we construct a simulation testbed using data from a prior study, and compare against commonly used algorithms in mobile health studies. We show that reBandit performs equally well or better than all the baseline algorithms, and the performance gap widens as population heterogeneity increases in the simulation environment, proving its adeptness to adapt to diverse population of study participants.","doi":"10.24963/ijcai.2024/805","url":"https://ijcai.org/proceedings/2024/805","pdf":"https://ijcai.org/proceedings/2024/0805.pdf","markdown":"https://ijcai.org/proceedings/2024/805.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/805","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI for Good","track":"AI for Good","number":805,"pages":{"first":7278,"last":7286},"keywords":["Multidisciplinary Topics and Applications: General","Uncertainty in AI: General","Machine Learning: General"],"videos":[],"references":[{"position":1,"raw":"[Aouali et al., 2023] Imad Aouali, Branislav Kveton, and Sumeet Katariya. Mixed-effect thompson sampling. In International Conference on Artificial Intelligence and Statistics, pages 2087- 2115. PMLR, 2023.","title":"Mixed-effect thompson sampling","authors":"Imad Aouali, Branislav Kveton, Sumeet Katariya","year":"2023","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Basu et al., 2021] Soumya Basu, Branislav Kveton, Manzil Za- heer, and Csaba Szepesvári. No regrets for learning the prior in bandits. Advances in neural information processing systems, 34:28029-28041, 2021.","title":"No regrets for learning the prior in bandits","authors":"Soumya Basu, Branislav Kveton, Manzil Zaheer, Csaba Szepesvári","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carliner et al., 2017] Hannah Carliner, Qiana L Brown, Aaron L Sarvet, and Deborah S Hasin. Cannabis use, attitudes, and legal status in the us: A review. Preventive medicine, 104:13-23, 2017.","title":"Cannabis use, attitudes, and legal status in the us: A review","authors":"Hannah Carliner, Qiana L Brown, Aaron L Sarvet, Deborah S Hasin","year":"2017","venue":"Preventive medicine","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2021] Gary CK Chan, Denise Becker, Peter Butter- worth, Lindsey Hines, Carolyn Coffey, Wayne Hall, and George Patton. Young-adult compared to adolescent onset of regular cannabis use: A 20-year prospective cohort study of later con- sequences. Drug and Alcohol Review, 40(4):627-636, 2021.","title":"Young-adult compared to adolescent onset of regular cannabis use: A 20-year prospective cohort study of later consequences","authors":"Denise Gary Ck Chan, Peter Becker, Lindsey Butterworth, Carolyn Hines, Wayne Coffey, George Hall, Patton","year":"2021","venue":"Drug and Alcohol Review","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Clarke et al., 2017] Shanice Clarke, Luis G Jaimes, and Miguel A Labrador. mstress: A mobile recommender system for just-in- time interventions for stress. In 2017 14th IEEE annual consumer communications & networking conference (CCNC), pages 1-5. IEEE, 2017.","title":"mstress: A mobile recommender system for just-intime interventions for stress","authors":"Shanice Clarke, Luis G Jaimes, Miguel A Labrador","year":"2017","venue":"14th IEEE annual consumer communications & networking conference (CCNC)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deshpande et al., 2018] Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive lin- ear models. In International Conference on Machine Learning, pages 1194-1203. PMLR, 2018.","title":"Accurate inference for adaptive linear models","authors":"Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, Matt Taddy","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Forman et al., 2019] Evan M Forman, Stephanie G Kerrigan, Meghan L Butryn, Adrienne S Juarascio, Stephanie M Manasse, Santiago Ontañón, Diane H Dallal, Rebecca J Crochiere, and Danielle Moskow. Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss? Journal of behavioral medicine, 42:276-290, 2019.","title":"Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?","authors":"Stephanie G Evan M Forman, Meghan L Kerrigan, Adrienne S Butryn, Stephanie M Juarascio, Santiago Manasse, Diane H Ontañón, Rebecca J Dallal, Danielle Crochiere, Moskow","year":"2019","venue":"Journal of behavioral medicine","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Golbus et al., 2021] Jessica R Golbus, Walter Dempsey, Eliza- beth A Jackson, Brahmajee K Nallamothu, and Predrag Klasnja. Microrandomized trial design for evaluating just-in-time adap- tive interventions through mobile health technologies for cardio- vascular disease. Circulation: Cardiovascular Quality and Out- comes, 14(2):e006760, 2021.","title":"Microrandomized trial design for evaluating just-in-time adaptive interventions through mobile health technologies for cardiovascular disease","authors":"Walter Jessica R Golbus, Elizabeth A Dempsey, Brahmajee K Jackson, Predrag Nallamothu, Klasnja","year":"2021","venue":"Circulation: Cardiovascular Quality and Outcomes","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Greenewald et al., 2017] Kristjan Greenewald, Ambuj Tewari, Su- san Murphy, and Predag Klasnja. Action centered contextual bandits. Advances in neural information processing systems, 30, 2017.","title":"Action centered contextual bandits","authors":"Kristjan Greenewald, Ambuj Tewari, Susan Murphy, Predag Klasnja","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hall, 2009] Wayne Hall. The adverse health effects of cannabis use: what are they, and what are their implications for policy? International Journal of drug policy, 20(6):458-466, 2009.","title":"The adverse health effects of cannabis use: what are they, and what are their implications for policy?","authors":"Wayne Hall, Hall","year":"2009","venue":"International Journal of drug policy","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hasin et al., 2016] Deborah Hasin, Bradley Kerridge, Tulshi Saha, Boji Huang, Roger Pickering, Sharon Smith, Jeesun Jung, Haitao Zhang, and Bridget Grant. Prevalence and correlates of dsm- 5 cannabis use disorder, 2012-2013: Findings from the na- tional epidemiologic survey on alcohol and related conditions-iii. American Journal of Psychiatry, 173(6):588-599, 2016.","title":"Prevalence and correlates of dsm-5 cannabis use disorder, 2012-2013: Findings from the national epidemiologic survey on alcohol and related conditions-iii","authors":"Deborah Hasin, Bradley Kerridge, Tulshi Saha, Boji Huang, Roger Pickering, Sharon Smith, Jeesun Jung, Haitao Zhang, Bridget Grant","year":"2016","venue":"American Journal of Psychiatry","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hong et al., 2022] Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical bayesian bandits. In International Conference on Artificial Intelligence and Statistics, pages 7724-7741. PMLR, 2022.","title":"Hierarchical bayesian bandits","authors":"Hong","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jaimes et al., 2015] Luis G Jaimes, Martin Llofriu, and Andrew Raij. Preventer, a selection mechanism for just-in-time preven- tive interventions. IEEE Transactions on Affective Computing, 7(3):243-257, 2015.","title":"Preventer, a selection mechanism for just-in-time preventive interventions","authors":"Martin Luis G Jaimes, Andrew Llofriu, Raij","year":"2015","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2015] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Con- ference on Autonomous Agents and Multiagent Systems, pages 1181-1189, 2015.","title":"The dependence of effective planning horizon on model accuracy","authors":"Nan Jiang, Alex Kulesza, Satinder Singh, Richard Lewis","year":"2015","venue":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":"/proceedings/2016/626"},{"position":15,"raw":"[Kalvit and Zeevi, 2021] Anand Kalvit and Assaf Zeevi. A closer look at the worst-case behavior of multi-armed bandit algorithms. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8807-8819.","title":"A closer look at the worst-case behavior of multi-armed bandit algorithms","authors":"Anand Kalvit, Assaf Zeevi","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kramer et al., 2019] Jan-Niklas Kramer, Florian Künzler, Varun Mishra, Bastien Presset, David Kotz, Shawna Smith, Urte Scholz, Tobias Kowatsch, et al. Investigating intervention components and exploring states of receptivity for a smartphone app to pro- mote physical activity: protocol of a microrandomized trial. JMIR research protocols, 8(1):e11540, 2019.","title":"Investigating intervention components and exploring states of receptivity for a smartphone app to promote physical activity: protocol of a microrandomized trial","authors":"Kramer","year":"2019","venue":"JMIR research protocols","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laird and Ware, 1982] Nan Laird and James Ware. Random- effects models for longitudinal data. Biometrics, pages 963-974, 1982.","title":"Randomeffects models for longitudinal data","authors":"Laird, Ware ;, Nan Laird, James Ware","year":"1982","venue":"Biometrics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Laird, 2004] Nan Laird. Random effects and the linear mixed model. In Analysis of Longitudinal and Cluster-Correlated Data, volume 8, pages 79-96. Institute of Mathematical Statistics, 2004.","title":"Random effects and the linear mixed model","authors":"Nan Laird","year":"2004","venue":"Analysis of Longitudinal and Cluster-Correlated Data","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Langford and Zhang, 2007] John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Ad- vances in neural information processing systems, 20(1):96-1, 2007.","title":"The epoch-greedy algorithm for contextual multi-armed bandits","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lapham et al., 2019] Gwen T Lapham, Cynthia I Campbell, Bobbi Jo H Yarborough, Rulin C Hechter, Brian K Ahmedani, Irina V Haller, Andrea H Kline-Simon, Derek D Satre, Amy M Loree, Constance Weisner, et al. The prevalence of healthcare effective- ness data and information set (hedis) initiation and engagement in treatment among patients with cannabis use disorders in 7 us health systems. Substance Abuse, 40(3):268-277, 2019.","title":"The prevalence of healthcare effectiveness data and information set (hedis) initiation and engagement in treatment among patients with cannabis use disorders in 7 us health systems","authors":"Gwen T Lapham, Cynthia I Campbell, Bobbi Jo, H Yarborough, Rulin C Hechter, Brian K Ahmedani, Irina V Haller, Andrea H Kline-Simon, Derek D Satre, Amy M Loree, Constance Weisner","year":"2019","venue":"Substance Abuse","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news ar- ticle recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liao et al., 2019] Peng Liao, Kristjan H. Greenewald, Predrag V. Klasnja, and Susan A. Murphy. Personalized heartsteps: A re- inforcement learning algorithm for optimizing physical activity. CoRR, abs/1909.03539, 2019.","title":"Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity","authors":"Peng Liao, Kristjan H Greenewald, V Predrag, Susan A Klasnja, Murphy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Morris, 1983] Carl N Morris. Parametric empirical bayes infer- ence: theory and applications. Journal of the American statistical Association, 78(381):47-55, 1983.","title":"Parametric empirical bayes inference: theory and applications","authors":"Morris Carl","year":"1983","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nahum-Shani et al., 2018] Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. Just-in-time adaptive interven- tions (jitais) in mobile health: key components and design prin- ciples for ongoing health behavior support. Annals of Behavioral Medicine, pages 1-17, 2018.","title":"Just-in-time adaptive interventions (jitais) in mobile health: key components and design principles for ongoing health behavior support","authors":"Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, Susan A Murphy","year":"2018","venue":"Annals of Behavioral Medicine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nahum-Shani et al., 2021] Inbal Nahum-Shani, Mashfiqui Rabbi, Jamie Yap, Meredith L Philyaw-Kotov, Predrag Klasnja, Erin E Bonar, Rebecca M Cunningham, Susan A Murphy, and Mau- reen A Walton. Translating strategies for promoting engage- ment in mobile health: A proof-of-concept microrandomized trial. Health Psychology, 40(12):974, 2021.","title":"Translating strategies for promoting engagement in mobile health: A proof-of-concept microrandomized trial","authors":"Inbal Nahum-Shani, Mashfiqui Rabbi, Jamie Yap, L Meredith, Predrag Philyaw-Kotov, Erin E Klasnja, Rebecca M Bonar, Susan A Cunningham, Maureen A Murphy, Walton","year":"2021","venue":"Health Psychology","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nahum-Shani et al., 2022] Inbal Nahum-Shani, Steven D Shaw, Stephanie M Carpenter, Susan A Murphy, and Carolyn Yoon. En- gagement in digital interventions. American Psychologist, 2022.","title":"Engagement in digital interventions","authors":"Inbal Nahum-Shani, Steven D Shaw, Stephanie M Carpenter, Susan A Murphy, Carolyn Yoon","year":"2022","venue":"American Psychologist","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Paredes et al., 2014] Pablo Paredes, Ran Gilad-Bachrach, Mary Czerwinski, Asta Roseway, Kael Rowan, and Javier Hernandez. Poptherapy: Coping with stress through pop-culture. In Proceed- ings of the 8th international conference on pervasive computing technologies for healthcare, pages 109-117, 2014.","title":"Poptherapy: Coping with stress through pop-culture","authors":"Paredes","year":"2014","venue":"Proceedings of the 8th international conference on pervasive computing technologies for healthcare","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Peacock et al., 2018] Amy Peacock, Janni Leung, Sarah Larney, Samantha Colledge, Matthew Hickman, Jürgen Rehm, Gary A Giovino, Robert West, Wayne Hall, Paul Griffiths, et al. Global statistics on alcohol, tobacco and illicit drug use: 2017 status re- port. Addiction, 113(10):1905-1926, 2018.","title":"Global statistics on alcohol, tobacco and illicit drug use: 2017 status report","authors":"Amy Peacock, Janni Leung, Sarah Larney, Samantha Colledge, Matthew Hickman, Jürgen Rehm, Gary A Giovino, Robert West, Wayne Hall, Paul Griffiths","year":"2018","venue":"Addiction","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Peleg et al., 2022] Amit Peleg, Naama Pearl, and Ron Meir. Met- alearning linear bandits by prior update. In International Confer- ence on Artificial Intelligence and Statistics, pages 2885-2926. PMLR, 2022.","title":"Metalearning linear bandits by prior update","authors":"Amit Peleg, Naama Pearl, Ron Meir","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rabbi et al., 2015] Mashfiqui Rabbi, Min Hane Aung, Mi Zhang, and Tanzeem Choudhury. Mybehavior: automatic personalized health feedback from user behaviors and preferences using smart- phones. In Proceedings of the 2015 ACM international joint con- ference on pervasive and ubiquitous computing, pages 707-718, 2015.","title":"Mybehavior: automatic personalized health feedback from user behaviors and preferences using smartphones","authors":"Rabbi","year":"2015","venue":"Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rabbi et al., 2018] Mashfiqui Rabbi, Meredith Philyaw Kotov, Re- becca Cunningham, Erin E Bonar, Inbal Nahum-Shani, Predrag Klasnja, Maureen Walton, Susan Murphy, et al. Toward increas- ing engagement in substance use data collection: development of the substance abuse research assistant app and protocol for a mi- crorandomized trial using adolescents and emerging adults. JMIR research protocols, 7(7):e9850, 2018.","title":"Toward increasing engagement in substance use data collection: development of the substance abuse research assistant app and protocol for a microrandomized trial using adolescents and emerging adults","authors":"Rabbi","year":"2018","venue":"JMIR research protocols","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rabbi et al., 2019] Mashfiqui Rabbi, Predrag Klasnja, Tanzeem Choudhury, Ambuj Tewari, and Susan Murphy. Optimizing mhealth interventions with a bandit. Digital Phenotyping and Mobile Sensing: New Developments in Psychoinformatics, pages 277-291, 2019.","title":"Optimizing mhealth interventions with a bandit","authors":"Rabbi","year":"2019","venue":"Digital Phenotyping and Mobile Sensing: New Developments in Psychoinformatics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Robinson, 1991] George K Robinson. That blup is a good thing: the estimation of random effects. Statistical science, pages 15- 32, 1991.","title":"That blup is a good thing: the estimation of random effects","authors":"George K Robinson","year":"1991","venue":"Statistical science","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Russo and Van Roy, 2014] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Op- erations Research, 39(4):1221-1243, 2014. [SAMHSA, 2023] SAMHSA. 2021 NSDUH De- tailed Tables. https://www.samhsa.gov/data/report/ 2021-nsduh-detailed-tables, 2023. Accessed: 2024-02-14.","title":"Learning to optimize via posterior sampling","authors":"Van Russo, Daniel Roy, Benjamin Russo, Roy Van","year":"2014","venue":"NSDUH Detailed Tables","doi":null,"url":"https://www.samhsa.gov/data/report/2021-nsduh-detailed-tables,2023","internal_url":null},{"position":35,"raw":"[Shrier et al., 2018] Lydia A Shrier, Pamela J Burke, Meredith Kells, Emily A Scherer, Vishnudas Sarda, Cassandra Jonestrask, Ziming Xuan, and Sion Kim Harris. Pilot randomized trial of moment, a motivational counseling-plus-ecological momentary intervention to reduce marijuana use in youth. Mhealth, 4, 2018.","title":"Pilot randomized trial of moment, a motivational counseling-plus-ecological momentary intervention to reduce marijuana use in youth","authors":"Lydia A Shrier, Pamela J Burke, Meredith Kells, Emily A Scherer, Vishnudas Sarda, Cassandra Jonestrask, Ziming Xuan, Sion Kim, Harris","year":"2018","venue":"Mhealth","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Simchowitz et al., 2021] Max Simchowitz, Christopher Tosh, Ak- shay Krishnamurthy, Daniel Hsu, Thodoris Lykouris, Miro Dudik, and Robert Schapire. Bayesian decision-making un- der misspecified priors with applications to meta-learning. Ad- vances in Neural Information Processing Systems, 34:26382- 26394, 2021.","title":"Bayesian decision-making under misspecified priors with applications to meta-learning","authors":"Simchowitz","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tewari and Murphy, 2017] Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile health: sensors, analytic methods, and applications, pages 495-517, 2017.","title":"From ads to interventions: Contextual bandits in mobile health","authors":"Ambuj Tewari, Susan A Murphy","year":"2017","venue":"Mobile health: sensors, analytic methods, and applications","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tomkins et al., 2021] Sabina Tomkins, Peng Liao, Predrag Klas- nja, and Susan Murphy. Intelligentpooling: Practical thomp- son sampling for mhealth. Machine learning, 110(9):2685-2727, 2021.","title":"Intelligentpooling: Practical thompson sampling for mhealth","authors":"Sabina Tomkins, Peng Liao, Predrag Klasnja, Susan Murphy","year":"2021","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Trella et al., 2022] Anna Trella, Kelly Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, and Susan Murphy. Designing reinforcement learning algorithms for digital interventions: pre- implementation guidelines. Algorithms, 15(8):255, 2022.","title":"Designing reinforcement learning algorithms for digital interventions: preimplementation guidelines","authors":"Anna Trella, Kelly Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, Susan Murphy","year":"2022","venue":"Algorithms","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Trella et al., 2023] Anna L. Trella, Kelly W. Zhang, Inbal Nahum- Shani, Vivek Shetty, Finale Doshi-Velez, and Susan A. Murphy. Reward design for an online reinforcement learning algorithm supporting oral self-care. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13):15724-15730, Sep. 2023.","title":"Reward design for an online reinforcement learning algorithm supporting oral self-care","authors":"Anna L Trella, Kelly W Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, Susan A Murphy","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Volkow et al., 2014] Nora D Volkow, Ruben D Baler, Wilson M Compton, and Susan RB Weiss. Adverse health effects of mar- ijuana use. New England Journal of Medicine, 370(23):2219- 2227, 2014.","title":"Adverse health effects of marijuana use","authors":"Nora D Volkow, Ruben D Baler, Wilson M Compton, Susan Rb Weiss","year":"2014","venue":"New England Journal of Medicine","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Walsh and Groarke, 2019] Jane C Walsh and Jenny M Groarke. In- tegrating behavioral science with mobile (mhealth) technology to optimize health behavior change interventions. European Psy- chologist, 2019.","title":"Integrating behavioral science with mobile (mhealth) technology to optimize health behavior change interventions","authors":"C Jane, Jenny M Walsh, Groarke","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wan et al., 2021] Runzhe Wan, Lin Ge, and Rui Song. Metadata- based multi-task bandits with bayesian hierarchical models. Ad- vances in Neural Information Processing Systems, 34:29655- 29668, 2021.","title":"Metadatabased multi-task bandits with bayesian hierarchical models","authors":"Wan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wan et al., 2023] Runzhe Wan, Lin Ge, and Rui Song. Towards scalable and robust structured bandits: A meta-learning frame- work. In International Conference on Artificial Intelligence and Statistics, pages 1144-1173. PMLR, 2023.","title":"Towards scalable and robust structured bandits: A meta-learning framework","authors":"Wan","year":"2023","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2005] Chih-Chun Wang, Sanjeev R Kulkarni, and H Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3):338-355, 2005.","title":"Bandit problems with side observations","authors":"Chih-Chun Wang, Sanjeev R Kulkarni, H Vincent Poor","year":"2005","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Yom-Tov et al., 2017] Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encourag- ing physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of medical Internet re- search, 19(10):e338, 2017.","title":"Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system","authors":"Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, Irit Hochberg","year":"2017","venue":"Journal of medical Internet research","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhang et al., 2022] Kelly W Zhang, Lucas Janson, and Susan A Murphy. Statistical inference after adaptive sampling for longi- tudinal data. arXiv preprint arXiv:2202.07098, 2022.","title":"Statistical inference after adaptive sampling for longitudinal data","authors":"Lucas Kelly W Zhang, Susan A Janson, Murphy","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhou et al., 2018] Mo Zhou, Yonatan Mintz, Yoshimi Fukuoka, Ken Goldberg, Elena Flowers, Philip Kaminsky, Alejandro Castillejo, and Anil Aswani. Personalizing mobile fitness apps using reinforcement learning. In CEUR workshop proceedings, volume 2068. NIH Public Access, 2018.","title":"Personalizing mobile fitness apps using reinforcement learning","authors":"Mo Zhou, Yonatan Mintz, Yoshimi Fukuoka, Ken Goldberg, Elena Flowers, Philip Kaminsky, Alejandro Castillejo, Anil Aswani","year":"2018","venue":"CEUR workshop proceedings","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhu and Kveton, 2022a] Rong Zhu and Branislav Kveton. Ran- dom effect bandits. In International Conference on Artificial In- telligence and Statistics, pages 3091-3107. PMLR, 2022.","title":"Random effect bandits","authors":"Kveton Zhu, Rong Zhu, Branislav Kveton","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhu and Kveton, 2022b] Rong Zhu and Branislav Kveton. Ro- bust contextual linear bandits. arXiv preprint arXiv:2210.14483, 2022.","title":"Robust contextual linear bandits","authors":"Kveton Zhu, Rong Zhu, Branislav Kveton","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}