{"type":"ScholarlyArticle","title":"Building a Personalized Messaging System for Health Intervention in Underprivileged Regions Using Reinforcement Learning","authors":[{"name":"Sarah Kinsey","slug":"sarah-kinsey","given":"Sarah","family":"Kinsey","affiliation":"University of Oregon"},{"name":"Jack Wolf","slug":"jack-wolf","given":"Jack","family":"Wolf","affiliation":"University of Oregon"},{"name":"Nalini Saligram","slug":"nalini-saligram","given":"Nalini","family":"Saligram","affiliation":"Arogya World"},{"name":"Varun Ramesan","slug":"varun-ramesan","given":"Varun","family":"Ramesan","affiliation":"Arogya World"},{"name":"Meeta Walavalkar","slug":"meeta-walavalkar","given":"Meeta","family":"Walavalkar","affiliation":"Arogya World"},{"name":"Nidhi Jaswal","slug":"nidhi-jaswal","given":"Nidhi","family":"Jaswal","affiliation":"Arogya World"},{"name":"Sandhya Ramalingam","slug":"sandhya-ramalingam","given":"Sandhya","family":"Ramalingam","affiliation":"Arogya World"},{"name":"Arunesh Sinha","slug":"arunesh-sinha","given":"Arunesh","family":"Sinha","affiliation":"Rutgers University"},{"name":"Thanh Nguyen","slug":"thanh-nguyen","given":"Thanh","family":"Nguyen","affiliation":"University of Oregon"}],"abstract":"This work builds an effective AI-based  message generation system for diabetes prevention in rural areas, where the diabetes rate has been increasing at an alarming rate. The messages contain information about diabetes causes and complications and the impact of nutrition and fitness on preventing diabetes. We propose to apply reinforcement learning (RL) to optimize our message selection policy over time, tailoring our messages to align with each individual participant's needs and preferences. We conduct an extensive field study in a large country in Asia which involves more than 1000 participants who are local villagers and they receive messages generated by our system, over a period of six months. Our analysis shows that with the use of AI, we can deliver significant improvements in the participants' diabetes-related knowledge, physical activity levels, and high-fat food avoidance, when compared to a static message set. Furthermore, we build a new neural network based behavior model to predict behavior changes of participants, trained on data collected during our study. By exploiting underlying characteristics of health-related behavior, we manage to significantly improve the prediction accuracy of our model compared to baselines.","doi":"10.24963/ijcai.2023/668","url":"https://ijcai.org/proceedings/2023/668","pdf":"https://ijcai.org/proceedings/2023/0668.pdf","markdown":"https://ijcai.org/proceedings/2023/668.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/668","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":668,"pages":{"first":6022,"last":6030},"keywords":["AI for Good: Multidisciplinary Topics and Applications","AI for Good: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Ahn and Park, 2011] Inkyung Ahn and Jooyoung Park. Drug scheduling of cancer chemotherapy based on natu- ral actor-critic approach. BioSystems, 106(2-3):121-129, 2011.","title":"Inkyung Ahn and Jooyoung Park. Drug scheduling of cancer chemotherapy based on natural actor-critic approach","authors":"Park Ahn","year":"2011","venue":"BioSystems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asoh et al., 2013] Hideki Asoh, Masanori Shiro1 Shotaro Akaho, Toshihiro Kamishima, Koiti Hasida, Eiji Aramaki, and Takahide Kohro. An application of inverse reinforce- ment learning to medical records of diabetes treatment. In ECMLPKDD2013 workshop on reinforcement learning with generalized feedback, 2013.","title":"An application of inverse reinforcement learning to medical records of diabetes treatment","authors":"Hideki Asoh, Masanori Shiro1, Shotaro Akaho, Toshihiro Kamishima, Koiti Hasida, Eiji Aramaki, Takahide Kohro","year":"2013","venue":"ECMLPKDD2013 workshop on reinforcement learning with generalized feedback","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bothe et al., 2013] Melanie K Bothe, Luke Dickens, Katrin Reichel, Arn Tellmann, Björn Ellger, Martin Westphal, and Ahmed A Faisal. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert review of medical devices, 10(5):661-673, 2013.","title":"The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas","authors":"Melanie K Bothe, Luke Dickens, Katrin Reichel, Arn Tellmann, Björn Ellger, Martin Westphal, Ahmed A Faisal","year":"2013","venue":"Expert review of medical devices","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chu et al., 2016] Tianshu Chu, Jie Wang, and Jiayu Chen. An adaptive online learning framework for practical breast cancer diagnosis. In Medical imaging 2016: Computer- aided diagnosis, volume 9785, pages 537-548. SPIE, 2016.","title":"An adaptive online learning framework for practical breast cancer diagnosis","authors":"Tianshu Chu, Jie Wang, Jiayu Chen","year":"2016","venue":"Computeraided diagnosis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Corbett and Anderson, 1994] Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisi- tion of procedural knowledge. User modeling and user- adapted interaction, 4:253-278, 1994.","title":"Knowledge tracing: Modeling the acquisition of procedural knowledge","authors":"Anderson ; Corbett, T Albert, John R Corbett, Anderson","year":"1994","venue":"User modeling and useradapted interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Daskalaki et al., 2013] Elena Daskalaki, Peter Diem, and Stavroula G Mougiakakou. Personalized tuning of a re- inforcement learning control algorithm for glucose reg- ulation. In 2013 35th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 3487-3490. IEEE, 2013.","title":"Personalized tuning of a reinforcement learning control algorithm for glucose regulation","authors":"Elena Daskalaki, Peter Diem, G Stavroula, Mougiakakou","year":"2013","venue":"35th Annual international conference of the IEEE engineering in medicine and biology society (EMBC)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gaweda et al., 2005] Adam E Gaweda, Mehmet K Muezzinoglu, George R Aronoff, Alfred A Jacobs, Jacek M Zurada, and Michael E Brier. Individualization of pharmacological anemia management using rein- forcement learning. Neural Networks, 18(5-6):826-834, 2005.","title":"Individualization of pharmacological anemia management using reinforcement learning","authors":"Adam E Gaweda, K Mehmet, George R Muezzinoglu, Alfred A Aronoff, Jacek M Jacobs, Michael E Zurada, Brier","year":"2005","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gaweda et al., 2006] Adam E Gaweda, Mehmet K Muezzinoglu, Alfred A Jacobs, George R Aronoff, and Michael E Brier. Model predictive control with reinforcement learning for drug delivery in renal anemia management. In 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pages 5177-5180. IEEE, 2006.","title":"Model predictive control with reinforcement learning for drug delivery in renal anemia management","authors":"Adam E Gaweda, K Mehmet, Alfred A Muezzinoglu, George R Jacobs, Michael E Aronoff, Brier","year":"2006","venue":"International Conference of the IEEE Engineering in Medicine and Biology Society","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghesu et al., 2017] Florin-Cristian Ghesu, Bogdan Georgescu, Yefeng Zheng, Sasa Grbic, Andreas Maier, Joachim Hornegger, and Dorin Comaniciu. Multi-scale deep reinforcement learning for real-time 3d-landmark detection in ct scans. IEEE transactions on pattern analysis and machine intelligence, 41(1):176-189, 2017.","title":"Multi-scale deep reinforcement learning for real-time 3d-landmark detection in ct scans","authors":"Florin-Cristian Ghesu, Bogdan Georgescu, Yefeng Zheng, Sasa Grbic, Andreas Maier, Joachim Hornegger, Dorin Comaniciu","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghosh et al., 2020] Aritra Ghosh, Neil Heffernan, and An- drew S Lan. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2330-2339, 2020.","title":"Context-aware attentive knowledge tracing","authors":"Aritra Ghosh, Neil Heffernan, Andrew S Lan","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hassani and others, 2010] Amin Hassani et al. Rein- forcement learning based control of tumor growth with chemotherapy. In 2010 International Conference on System Science and Engineering, pages 185-189. IEEE, 2010.","title":"Reinforcement learning based control of tumor growth with chemotherapy","authors":"Others ; Amin Hassani, Hassani","year":"2010","venue":"2010 International Conference on System Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochberg et al., 2016] Irit Hochberg, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Elad Yom-Tov. A reinforcement learning system to encour- age physical activity in diabetes patients. arXiv preprint arXiv:1605.04070, 2016.","title":"A reinforcement learning system to encourage physical activity in diabetes patients","authors":"Irit Hochberg, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, Elad Yom-Tov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter and Schmidhuber, 1996] Sepp Hochreiter and Jürgen Schmidhuber. Lstm can solve hard long time lag problems. In Proceedings of the 9th International Confer- ence on Neural Information Processing Systems, NIPS'96, page 473-479, Cambridge, MA, USA, 1996. MIT Press.","title":"Lstm can solve hard long time lag problems","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1996","venue":"Proceedings of the 9th International Conference on Neural Information Processing Systems, NIPS'96","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kao et al., 2018] Hao-Cheng Kao, Kai-Fu Tang, and Ed- ward Chang. Context-aware symptom checking for dis- ease diagnosis using hierarchical reinforcement learning. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018.","title":"Context-aware symptom checking for disease diagnosis using hierarchical reinforcement learning","authors":"Hao-Cheng Kao, Kai-Fu Tang, Edward Chang","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Martín-Guerrero et al., 2009] José D Martín-Guerrero, Faustino Gomez, Emilio Soria-Olivas, Jürgen Schmid- huber, Mónica Climente-Martí, and N Víctor Jiménez- Torres. A reinforcement learning approach for individu- alizing erythropoietin dosages in hemodialysis patients. Expert Systems with Applications, 36(6):9737-9742, 2009.","title":"A reinforcement learning approach for individualizing erythropoietin dosages in hemodialysis patients","authors":"Faustino José D Martín-Guerrero, Emilio Gomez, Jürgen Soria-Olivas, Mónica Schmidhuber, N Climente-Martí, Jiménez-Torres Víctor","year":"2009","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moore et al., 2014] Brett L Moore, Larry D Pyeatt, Vivekanand Kulkarni, Periklis Panousis, Kevin Padrez, and Anthony G Doufas. Reinforcement learning for closed-loop propofol anesthesia: a study in human vol- unteers. The journal of machine learning research, 15(1):655-696, 2014.","title":"Reinforcement learning for closed-loop propofol anesthesia: a study in human volunteers","authors":"Larry D Brett L Moore, Vivekanand Pyeatt, Periklis Kulkarni, Kevin Panousis, Anthony G Padrez, Doufas","year":"2014","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Noori et al., 2017] Amin Noori, Mohammad Ali Sadrnia, et al. Glucose level control using temporal difference methods. In 2017 Iranian Conference on Electrical En- gineering (ICEE), pages 895-900. IEEE, 2017.","title":"Glucose level control using temporal difference methods","authors":"Amin Noori, Mohammad Ali Sadrnia","year":"2017","venue":"Iranian Conference on Electrical Engineering (ICEE)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Organization, 2018] World Health Organization. Noncom- municable diseases country profiles 2018. World Health Organization, 2018.","title":"World Health Organization. Noncommunicable diseases country profiles 2018","authors":null,"year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Padmanabhan et al., 2017] Regina Padmanabhan, Nader Meskin, and Wassim M Haddad. Reinforcement learning- based control of drug dosing for cancer chemotherapy treatment. Mathematical biosciences, 293:11-20, 2017.","title":"Reinforcement learningbased control of drug dosing for cancer chemotherapy treatment","authors":"Regina Padmanabhan, Nader Meskin, Wassim M Haddad","year":"2017","venue":"Mathematical biosciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pandey and Karypis, 2019] Shalini Pandey and George Karypis. A self-attentive model for knowledge tracing. arXiv preprint arXiv:1907.06837, 2019.","title":"A self-attentive model for knowledge tracing","authors":"Shalini Pandey, George Karypis","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Parbhoo et al., 2017] Sonali Parbhoo, Jasmina Bogojeska, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Combining kernel and model based learning for hiv ther- apy selection. AMIA Summits on Translational Science Proceedings, 2017:239, 2017.","title":"Combining kernel and model based learning for hiv therapy selection","authors":"Sonali Parbhoo, Jasmina Bogojeska, Maurizio Zazzi, Roth Volker, Finale Doshi-Velez","year":"2017","venue":"AMIA Summits on Translational Science Proceedings","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pfammatter et al., 2016] Angela Pfammatter, Bonnie Spring, Nalini Saligram, Raj Davé, Arun Gowda, Linelle Blais, Monika Arora, Harish Ranjani, Om Ganda, Donald Hedeker, et al. mhealth intervention to improve diabetes risk behaviors in india: a prospective, parallel group cohort study. Journal of medical Internet research, 18(8):e207, 2016.","title":"mhealth intervention to improve diabetes risk behaviors in india: a prospective, parallel group cohort study","authors":"Angela Pfammatter, Bonnie Spring, Nalini Saligram, Raj Davé, Arun Gowda, Linelle Blais, Monika Arora, Harish Ranjani, Om Ganda, Donald Hedeker","year":"2016","venue":"Journal of medical Internet research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Piech et al., 2015] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge trac- ing. Advances in neural information processing systems, 28, 2015.","title":"Deep knowledge tracing","authors":"Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, Jascha Sohl-Dickstein","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ranjani et al., 2020] Harish Ranjani, Sharma Nitika, Ran- jit Mohan Anjana, Sandhya Ramalingam, Viswanathan Mohan, Nalini Saligram, et al. Impact of noncommuni- cable disease text messages delivered via an app in pre- venting and managing lifestyle diseases: Results of the \"myarogya\" worksite-based effectiveness study from in- dia. Journal of Diabetology, 11(2):90, 2020.","title":"Impact of noncommunicable disease text messages delivered via an app in preventing and managing lifestyle diseases: Results of the \"myarogya\" worksite-based effectiveness study from india","authors":"Harish Ranjani, Sharma Nitika, Ranjit Mohan Anjana, Sandhya Ramalingam, Viswanathan Mohan, Nalini Saligram","year":"2020","venue":"Journal of Diabetology","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sahba et al., 2008] Farhang Sahba, Hamid R Tizhoosh, and Magdy MA Salama. Application of reinforcement learn- ing for segmentation of transrectal ultrasound images. BMC medical imaging, 8:1-10, 2008.","title":"Application of reinforcement learning for segmentation of transrectal ultrasound images","authors":"Farhang Sahba, Hamid R Tizhoosh, M A Magdy, Salama","year":"2008","venue":"BMC medical imaging","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Saria, 2018] Suchi Saria. Individualized sepsis treat- ment using reinforcement learning. Nature medicine, 24(11):1641-1642, 2018.","title":"Individualized sepsis treatment using reinforcement learning","authors":"Suchi Saria","year":"2018","venue":"Nature medicine","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sinzinger and Moore, 2005] Eric D Sinzinger and Brett Moore. Sedation of simulated icu patients using reinforce- ment learning based control. International Journal on Ar- tificial Intelligence Tools, 14(01n02):137-156, 2005.","title":"Sedation of simulated icu patients using reinforcement learning based control","authors":"D Eric, Brett Sinzinger, Moore","year":"2005","venue":"International Journal on Artificial Intelligence Tools","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xiong et al., 2016] Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, and Joseph E Beck. Going deeper with deep knowledge tracing. International Educational Data Min- ing Society, 2016.","title":"Going deeper with deep knowledge tracing","authors":"Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, Joseph E Beck","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yeung and Yeung, 2018] Chun-Kit Yeung and Dit-Yan Ye- ung. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the fifth annual ACM conference on learning at scale, pages 1-10, 2018.","title":"Addressing two problems in deep knowledge tracing via prediction-consistent regularization","authors":"Yeung Yeung, Chun-Kit Yeung, Dit-Yan Yeung","year":"2018","venue":"Proceedings of the fifth annual ACM conference on learning at scale","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yom-Tov et al., 2017] Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning sys- tem. Journal of medical Internet research, 19(10):e338, 2017.","title":"Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning system","authors":"Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, Irit Hochberg","year":"2017","venue":"Journal of medical Internet research","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2019] Chao Yu, Yinzhao Dong, Jiming Liu, and Guoqi Ren. Incorporating causal factors into reinforce- ment learning for dynamic treatment regimes in hiv. BMC medical informatics and decision making, 19(2):19-29, 2019.","title":"Incorporating causal factors into reinforcement learning for dynamic treatment regimes in hiv","authors":"Chao Yu, Yinzhao Dong, Jiming Liu, Guoqi Ren","year":"2019","venue":"BMC medical informatics and decision making","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2021] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.","title":"Reinforcement learning in healthcare: A survey","authors":"Chao Yu, Jiming Liu, Shamim Nemati, Guosheng Yin","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2009] Yufan Zhao, Michael R Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in medicine, 28(26):3294-3315, 2009.","title":"Reinforcement learning design for cancer clinical trials","authors":"Zhao","year":"2009","venue":"Statistics in medicine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}