---
title: "ReBandit: Random Effects Based Online RL Algorithm for Reducing Cannabis Use"
authors:
  - "Susobhan Ghosh"
  - "Yongyi Guo"
  - "Pei-Yao Hung"
  - "Lara Coughlin"
  - "Erin Bonar"
  - "Inbal Nahum-Shani"
  - "Maureen Walton"
  - "Susan Murphy"
doi: "10.24963/ijcai.2024/805"
url: "https://ijcai.org/proceedings/2024/805"
pdf: "https://ijcai.org/proceedings/2024/0805.pdf"
venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence"
year: 2024
pages: "7278-7286"
track: "AI for Good"
keywords:
  - "Multidisciplinary Topics and Applications: General"
  - "Uncertainty in AI: General"
  - "Machine Learning: General"
---

# ReBandit: Random Effects Based Online RL Algorithm for Reducing Cannabis Use

Susobhan Ghosh, Yongyi Guo, Pei-Yao Hung, Lara Coughlin, Erin Bonar, Inbal Nahum-Shani, Maureen Walton, Susan Murphy

*Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 7278-7286. DOI: [10.24963/ijcai.2024/805](https://doi.org/10.24963/ijcai.2024/805)

## Abstract

The escalating prevalence of cannabis use, and associated cannabis-use disorder (CUD), poses a significant public health challenge globally. With a notably wide treatment gap, especially among emerging adults (EAs; ages 18-25), addressing cannabis use and CUD remains a pivotal objective within the 2030 United Nations Agenda for Sustainable Development Goals (SDG). In this work, we develop an online reinforcement learning (RL) algorithm called reBandit which will be utilized in a mobile health study to deliver personalized mobile health interventions aimed at reducing cannabis use among EAs. reBandit utilizes random effects and informative Bayesian priors to learn quickly and efficiently in noisy mobile health environments. Moreover, reBandit employs Empirical Bayes and optimization techniques to autonomously update its hyper-parameters online. To evaluate the performance of our algorithm, we construct a simulation testbed using data from a prior study, and compare against commonly used algorithms in mobile health studies. We show that reBandit performs equally well or better than all the baseline algorithms, and the performance gap widens as population heterogeneity increases in the simulation environment, proving its adeptness to adapt to diverse population of study participants.

## Full text

## 1 Introduction &Motivation

Addressing at-risk substance use, including cannabis use, is a pivotal objective within the 2030 UN Agenda for Sustainable Development Goals (SDG) 1 . Within this agenda, SDG 3 focuses on ensuring healthy lives and well-being across the lifespan, yet, increasing use of cannabis, third in global prevalence after alcohol and nicotine, threatens this goal [Peacock et al., 2018]. Hence, as highlighted in target 3.5 of the agenda, strengthening the prevention and treatment of cannabis use and cannabis use disorder (CUD) is crucial. Unfortunately, this coincides with a decreased public perception of the risks associated with cannabis use, likely influenced by ongoing decriminalization efforts and greater access to cannabis products [Carliner et al., 2017], further worsened by one of the largest treatment gaps of any medical condition, with one study showing only 5% of those with CUD receiving treatment [Lapham et al., 2019].

1 https://sdgs.un.org/2030agenda

In the US, the prevalence of cannabis use is highest among emerging adults (EAs; age 18-25) [SAMHSA, 2023], marking it as a significant concern within the growing landscape of cannabis use. Particularly worrisome is the fact that early initiation of cannabis use links to an array of physical and mental health repercussions, as well as escalated risk for developing CUD [Volkow et al., 2014; Hall, 2009; Chan et al., 2021; Hasin et al., 2016]. Given that cannabis use frequently commences during adolescence and peaks in emerging adulthood, this is a critical developmental period for early intervention strategies to prevent transitions into CUD.

Mobile health technologies, such as health apps and sensors, can potentially serve as support tools to help individuals manage their cannabis use. Using these tools, individuals can track their cannabis consumption, receive personalized interventions, and provide objective data for early detection of issues. These technologies enable the delivery of just-in-time adaptive interventions (JITAIs) [Nahum-Shani et al., 2018], which leverage rapidly changing information about a person's state and context to decide whether and how to intervene in daily life. JITAIs have been successful for many domains of behavioral health [Jaimes et al., 2015; Clarke et al., 2017; Golbus et al., 2021], whilst JITAIs for cannabis use among EAs are currently lacking evidence despite promising early data [Shrier et al., 2018].

In this work, we develop an RL algorithm called reBandit which will be utilized in the MiWaves pilot study (Section 1.1). MiWaves focuses on developing a JITAI for reducing cannabis use among emerging adults (EAs) (ages 18-25). This JITAI leverages reBandit to determine the likelihood of delivering an intervention message.

## 1.1 MiWaves Pilot Study

The MiWaves pilot study focuses on developing a personalizing Just-In-Time Adaptive Intervention (pJITAI), namely a JITAI that integrates an RL algorithm. In this study, EAs are randomized to receive a mobile-based intervention message or no message, twice daily. The RL algorithm is designed to learn from a participant's history, and personalize the likelihood of intervention delivery based on a participant's current context. By combining technology, behavioral science, and data-driven decision-making, MiWaves aims to empower emerging adults with the digital tools to help reduce their cannabis use. The MiWaves pilot study has been registered on ClinicalTrials.gov (NCT05824754), and is scheduled to start in March 2024. Figure 1 provides a visual overview of the MiWaves pilot study.

## 1.2 Challenges, Contributions and Overview

Deploying RL algorithms in mHealth studies like MiWaves present a multitude of challenges that must be addressed, which include:

- C1 Limited Data: Many sequential decision making problems in mHealth involve scarce data, forcing RL algorithms to learn and perform well under strict data constraints [Trella et al., 2022].
- C2 After-study analysis and evaluation: The RL algorithms deployed in mHealth studies need to be developed in a way to facilitate after-study analysis and offpolicy evaluation.
- C3 Autonomy and Stability: The intervention protocol in clinical studies is pre-specified. Since the RL algorithm is part of the intervention, scientists do not have the flexibility to change the RL algorithm while the study is running. RL algorithms must exhibit robustness in the face of noisy data, ensuring consistent and reliable performance throughout the study [Trella et al., 2022].
- C4 Explainability: It is imperative that RL algorithms are interpretable and comprehensible to behavioral scientists and medical professionals to enhance their ability to critique RL performance and to enhance the possibility of larger scale implementation.
- C5 Delayed Effects: In mobile health studies, each intervention message sent to the user has a delayed effect. Users may perceive burden upon receiving an intervention message, which influences their future behavior.
- C6 Reproducibility: Any algorithm used as part of the intervention in a clinical study needs to be reproducible in order for health scientists to evaluate and implement the intervention package in practice. Hence, the decisions taken by the RL algorithm must be reproducible, allowing for scrutiny and verification of their effectiveness.

To that end, we introduce reBandit, an online RL algorithm which utilizes random effects to address the challenges mentioned above. When used as part of an RL algorithm, random effects allow the algorithm to learn quickly and efficiently by making use of other participant's data in the population while simultaneously personalizing treatment for a given participant. Moreover, reBandit employs an informative Bayesian prior formulated from pre-existing data to act as a warmstart. Carefully designed priors incorporate previous (domain) knowledge, which help algorithms learn quickly and efficiently. Both random effects and informative priors can help reBandit to tackle challenge C1.

The most commonly used RL algorithms in mHealth settings are bandit algorithms. In mHealth settings, predictions of the value of next state (eg. using [Jiang et al., 2015]) can be very noisy. Bandit algorithms are, thus, preferred due to their performance in such noisy environments. Moreover, they are computationally less complex, and hence are able to run stably and reliably in an online environment. Further, linear models are often considered interpretable due to their simplicity of representing the role of various factors, and can also be stably updated. reBandit utilizes both these concepts - it uses a bandit framework, along with a linear model (with random effects) to model the reward. We derive the formula to update reBandit's parameters and hyper-parameters online (Sec 4.1). We show that we are able to autonomously update these parameters and hyper-parameters within a reasonable time-limit in an online environment. Moreover, to facilitate after-study analysis, we utilize a smooth variant of posterior sampling, and clip the probabilities of taking an action (Sec 4.2). This way, reBandit is able to overcome challenges C2, C3 and C4.

To address delayed effects (C5), one can use RL algorithms to model a full Markov Decision Process (MDP). However, in mobile health settings, such approaches are not feasible due to limited data and noisy outcomes [Trella et al., 2023]. On the other hand, the classical bandit framework alone is also insufficient, as it is designed to optimize for immediate reward, and thus, cannot account for the delayed effects of actions. To that end, we engineer the reward used to update reBandit's parameters and hyper-parameters (Sec 4.3), to account for delayed effects of actions.

Finally, to tackle challenge C6, we have made our implementation of reBandit publicly available 2 To ensure reproducibility, we employ a seeded pseudo-random number generator to make every stochastic decision in reBandit. Additionally, all intermediate results and values used to make decisions are programatically stored in a database for reproducing the results obtained during any run of the algorithm.

## 2 RL Framework and Notation

This section provides a brief overview of the Reinforcement Learning (RL) [Sutton and Barto, 2018] setup used in this work, and the specifics of the RL setup with respect to the MiWaves pilot study.

We approximate the pilot study environment as a bandit environment. We represent it as a Markov Decision Process (MDP) where in the RL algorithm (eg. the mobile app) interacts with the environment (eg. the user). The MDP is specified by the tuple ⟨S,A, r, P, T⟩, where S is the statespace of the algorithm, A is the action-space, r(s, a) is the reward function defined for a given state s ∈ S and action a ∈ A, P(s, a, s′ ) is the transition function for a given state s ∈ S, action a ∈ A and next state s′ ∈ S, and T is the total number of decision times. A user trajectory is given by H(T +1) i = {S(t) , a(t) ,R(t) }Tt=1 , where S(t) denotes the state at decision time t, a(t) the action assigned by the RL algorithm at time t, and R(t) the reward collected after selection of the action. In the case of MiWaves we have:

[2 https://github.com/StatisticalReinforcementLearningLab/ miwaves rl service](https://github.com/StatisticalReinforcementLearningLab/miwaves_rl_service)

Figure 1: Summary of the MiWaves pilot study. m = 120 EAs are expected to be recruited through social media ads. Each EA will be in the trial for 30 days, and will be asked to self-report twice daily - once in the morning and once in the evening. Upon completion or time expiration of the self-reporting, the RL algorithm will decide whether to send or not send an intervention message.

![Image](https://ijcai.org/proceedings/2024/fulltext/805/image_000000_30a42bf9aff11598c6e65132c5ee711c551cdcb7e9558ef9eb79a1ca55fe7a73.png)

Actions Binary action space, i.e. A = {0, 1} - to not send (0) or to send (1) an intervention message.

Decision points T decision points per user. The study is set to run for D = 30 days, and each day is supposed to have 2 decision points per day. Therefore, we expect to have 60 decision points per user, i.e. T = 60.

Reward We denote the reward for user i at decision time t by R(t) i . For the MiWaves pilot study, we have discrete rewards {0, 1, 2, 3}, which increase linearly with user engagement. We utilize engagement as our reward because engagement is critical to assess effectiveness of interventions after the study is over [Nahum-Shani et al., 2022].

States Let us denote the state observation of the user i at decision time t as S(t) i . A given state S = (S1, S2, S3) is defined as a 3-tuple of the following binary variables (omitting the user and time index for brevity):

- S1: Recent engagement - set to 1 if the average of past 3 observed rewards is greater than or equal to 2 (high engagement), and set to 0 otherwise (low engagement). At decision point t = 1, we set S1 to 0, as there is no engagement by the user at the start of the pilot study.
- S2: Time of day of the decision point - Morning (0) vs. Evening (1).
- S3: Recent cannabis use - set to 0 if the participant reported using cannabis during their self-monitoring, and 1 otherwise. If the user fails to self-report, we set S3 to be 0 because we expect the participant in the MiWaves pilot study to be using cannabis regularly (at least 3 times a week).

Overall, we represent the favorable states as 1 (not using cannabis, high engagement), and the unfavorable states as 0 (using cannabis, low engagement).

Number of users We expect m = 120 users to participate during the RL-powered MiWaves pilot study.

## 3 RelatedWork

Random effects (and mixed-effects) models have been wellstudied in the statistical literature [Laird, 2004; Laird and Ware, 1982; Robinson, 1991], mainly in the context of batch data analysis. Mixed-effects models comprise of fixed and random effects - hence termed mixed effects. Laird and Ware introduce the notion of random-effects models for longitudinal data, and describe an unified approach to fitting such models using empirical Bayes and maximum likelihood estimation using EM algorithm. Our work (which is in context of streaming / real-time data) draws inspiration from Laird and Ware to extend random-effects based models to real-time decision making through RL in sequential decision making problems.

There has been a myriad of works in optimizing intervention delivery in mHealth settings in recent years [Golbus et al., 2021; Kramer et al., 2019; Rabbi et al., 2019; Trella et al., 2022; Walsh and Groarke, 2019]. Bandit algorithms are the most commonly used RL algorithms used in such high stakes online settings [Langford and Zhang, 2007; Tewari and Murphy, 2017; Wang et al., 2005] due to their simplicity and stability, and ability to perform in noisy environments. Such algorithms have mainly used one of two approaches. The first approach is person specific (a.k.a. fully personalized) [Forman et al., 2019; Jaimes et al., 2015; Liao et al., 2019; Rabbi et al., 2015] where a separate model is deployed for each user in the trial. This approach is suitable when the population of users are highly heterogeneous, but suffers greatly when data is scarce and/or noisy. Note that fully personalized approaches are not feasible for the MiWaves pilot study, due to scarce data (the study runs for only 30 days). The second approach completely pools data (a.k.a. fully pooled) across all users in the population [Clarke et al., 2017; Paredes et al., 2014; Trella et al., 2022; Yom-Tov et al., 2017; Zhou et al., 2018]. Our algorithm, reBandit, strikes a balance between the two approaches - it adaptively pools data across users depending on the degree of heterogeneity in the population. Section 4.1 describes how we achieve that balance using random effects.

Tomkins et al. also use random effects in their ThompsonSampling [Russo and Van Roy, 2014; Thompson, 1933] contextual bandit algorithm [Li et al., 2010], IntelligentPooling.

IntelligentPooling updates its hyper-parameters by modeling the problem as a Gaussian Process (GP). However, IntelligentPooling fails to run autonomously and stably in an online environment (does not overcome C3) 3 . Here we deal with this problem by updating the hyper-parameters in reBandit using empirical Bayes (similar to [Laird and Ware, 1982]), and solve the optimization problem using projected gradient descent. reBandit runs autonomously and stably in an online environment while also having more users (12x) and more features (8x) as compared to the environment described in IntelligentPooling.

Recently, various approaches have been explored regarding the application of mixed effects models within a bandit framework [Zhu and Kveton, 2022a; Zhu and Kveton, 2022b; Aouali et al., 2023]. However, these works primarily focus on utilizing mixed effects to capture the dependence and heterogeneity of rewards associated with different actions, rather than addressing the similarity and heterogeneity among multiple users. For example, [Zhu and Kveton, 2022a] consider a (non-contextual) multi-arm bandit problem, where the agent chooses one of the K arms at each time t with the goal of maximizing cumulative reward. The authors assume that the reward for the arms are correlated with each other and can be expressed using a mixed-effects model, so that pulling an arm gives some information about the reward of other arms as well. A follow up work, [Zhu and Kveton, 2022b], adds context into the reward model, and results in a linear mixed effects model for the rewards of the arms. [Aouali et al., 2023] further generalizes the above works to a non-linear reward setting. Our approach diverges from these studies by utilizing mixed-effects to model user similarity and heterogeneity, while making decisions for each user at each time point.

In the broader RL literature, there has been much work on Thompson Sampling based bandit algorithms [Basu et al., 2021; Hong et al., 2022], especially in connection to multi-task learning and meta learning [Peleg et al., 2022; Simchowitz et al., 2021; Wan et al., 2021; Wan et al., 2023]. The multi-task learning based approaches quantify the similarity between arms and/or users from their policies - the extent to which one user's data influences or contributes to another user's policy is a function of some similarity measure. reBandit can be connected to multi-task learning, as it learns across multiple users (or tasks), and tries to maximize rewards across all users (or tasks). However, due to its unique application in mobile health, reBandit adopts a distinct set of assumptions on the structure of the similarity measures in comparison to the works mentioned above. The meta-learning based approaches exploit the underlying structure of similar tasks to improve performance on new (or unseen, but similar) tasks. While reBandit can be viewed as a form of meta-learning, where shared population parameters are learnt across users (or tasks), and user-specific parameters are learnt to personalize to tasks, reBandit does not try to improve performance on new or unseen users.

3 We were unable to run their code published on GitHub, and unable to parse the code or replicate it due to poor documentation

## 4 Bandit Algorithm: reBandit

This section details details the reBandit algorithm used in the MiWaves pilot study. Being an online RL algorithm, reBandit has two major components: (i) the online learning algorithm; and (ii) the action-selection procedure. Going forward, we describe reBandit's online learning algorithm in Section 4.1, and it's posterior sampling based action selection strategy in Section 4.2. Finally, to address delayed effects, we describe its reward engineering procedure in Section 4.3. The reBandit algorithm is summarized in Algorithm 1.

## 4.1 Online Learning Algorithm

This section details the online learning algorithm - specifically the algorithm's reward approximating function and its model update procedure.

## Reward Approximating Function

One of the key components of the online learning algorithm is its reward approximation function, through which it models the participant's reward. Recall that the reward function is the conditional mean of the reward given state and action. We chose a Bayesian Mixed Linear Model to model the reward. Mixed models allow the RL algorithm to adaptively pool and learn across users while simultaneously personalizing actions for each user.

Let us assume that for a given user i at decision time t, the RL algorithm receives the reward R(t) i after taking action a(t) i Then, the reward model is written as:

<!-- formula-not-decoded -->

where ϵ(t) i is the noise, assumed to be gaussian i.e. ϵ ∼ N (0, σ2 ϵ Imt), and m is the total number of users who have been or are currently part of the study. Also αi, βi, and γi are weights that the algorithmwants to learn. g(S) and f(S) are functions of the RL state defined in Section 2. To enhance robustness to misspecification of the baseline reward model when a(t) i = 0, g(S(t) i )T αi, we utilize action-centering [Greenewald et al., 2017] to learn an over-parameterized version of the above reward model:

<!-- formula-not-decoded -->

where π(t) i is the probability of taking action a(t) i = 1 in state S(t) i for participant i at decision time t. We refer to the term g(S(t) i )T αi as the baseline, and f(S(t) i )T βi as the advantage (i.e. the advantage of taking action 1 over action 0).

We re-write the reward model as follows:

<!-- formula-not-decoded -->

where ΦTit = Φ(S(t) i , a(t) i , π(t) i )T = [g(S(t) i )T , (a(t) i - π(t) i )f(S(t) i )T , (π(t) i )f(S(t) i )T ] is the designmatrix for given state and action, and θi = [αi, βi, γi]T is the joint weight vector that the algorithm wants to learn. We further break down the joint weight vector θi into two components:

<!-- formula-not-decoded -->

Here, θpop = [αpop, βpop, γpop]T is the population level term which is common across all the user's reward models and follows a normal prior distribution given by θpop ∼ N(µprior, Σprior). On the other hand, ui = [uα,i, uβ,i, uγ,i]T are the individual level parameters, or the random effects, for any given user i. Note that the individual level parameters are assumed to be normal by definition, i.e. ui ∼ N(0, Σu), and independent of ϵi. Refer to Appendix D 4 for more details.

## Online Model Update Procedure

Posterior Update: We vectorize the terms across the m users in a study, and re-write the model as:

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

As specified before, we assume a gaussian prior on the population level term θpop ∼ N(µprior,Σprior). The hyperparameters of the above model, given the definition above, are the noise variance σ2 ϵ and the random effects variance Σu. Now, at a given decision point t, using estimated values of the hyper-parameters (σ2 ϵ,t is the estimate of σ2 ϵ and Σu,t is the estimate of Σu), the posterior mean and covariance matrix of the parameter θ can be calculated as:

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

where

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

The action-selection procedure (described in Section 4.2) uses the Gaussian posterior distribution defined by the posterior mean µ(t) post and variance Σ(t) post to determine the action selection probability π(t+1) and the corresponding actions for the next time steps.

Hyper-parameter Update: The hyper-parameters in the algorithm's reward model are the noise variance σ2 ϵ and random effects variance Σu. In order to update these variance estimates at the end of decision time t, we use Empirical Bayes

[4 https://arxiv.org/abs/2402.17739](https://arxiv.org/abs/2402.17739)

[Morris, 1983] to maximize the marginal likelihood of observed rewards, marginalized over the parameters θ. So, in order to formΣu,t and σ2 ϵ,t , we solve the following optimization problem:

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

where,

<!-- formula-not-decoded -->

Note that, X = ˜Σ-1 θ,t (see Eq. 17) and y = σ-2 ϵ,t . We solve the optimization problem using gradient descent.

## 4.2 Action Selection Procedure

The action selection procedure utilizes a modified posterior sampling algorithm called the smooth posterior sampling algorithm. Recall from Section 4.1, our model for the reward is a Bayesian linear mixed model with action centering (refer Eq. 2) where π(t) i is the probability that the RL algorithm selects action a(t) i = 1 in state S(t) i for participant i at decision point t. The RL algorithm computes the probability π(t) i as follows:

<!-- formula-not-decoded -->

Notice that the last expectation above is over the draw of β from the posterior distribution parameterized by µ(t-1) post,i and Σ(t-1) post,i (see Eq. 12 and Eq. 13 for their definitions).

Classical posterior sampling sets ρ(x) = I(x &gt; 0). In this case, the posterior sampling algorithm sets randomization probabilities to the posterior probability that the treatment effect is positive. However, when using a pooled algorithm, Zhang et al. showed that between study statistical inference is enhanced if ρ is a smooth i.e. continuously differentiable function. Using a smooth function ensures that the randomization probabilities formed by the algorithm concentrate. Concentration enhances the replicability of the randomization probabilities if the study is repeated. Without concentration, the randomization probabilities might fluctuate greatly between repetitions of the study [Deshpande et al., 2018; Kalvit and Zeevi, 2021; Zhang et al., 2022]. In MiWaves , we choose ρ to be a generalized logistic function, defined as follows (details in Appendix E 4 ) :

<!-- formula-not-decoded -->

where c = 5, and b = 21. We set the lower and upper clipping probabilities as Lmin = 0.2 and Lmax = 0.8 (i.e., 0.2 ≤ π(t) i ≤ 0.8). The probabilities are clipped to facilitate afterstudy analysis and off-policy evaluation [Zhang et al., 2022].

## Algorithm 1: reBandit

```
Input : m, D, µ(0) post = µprior, Σ(0) post = Σprior, Σu,0, σ2 ϵ,0, ρ(x) for d = 1 to D do for j = 0 to 1 do Compute timestep τ = ((d - 1) × 2) + j for i = 1 to m do Observe state S(τ) i ; Get posteriors µ(d-1) post,i and Σ(d-1) post,i for user i from µ(d-1) post and Σ(d-1) post ; Compute action selection probability π(τ) i using Eq. 21 Sample action a(t) i = Bern(π(τ) i ) Collect reward R(τ) i end end Update Σu,d and σ2 ϵ,d using Eq. 18, with engineered rewards from Eq. 23; Update posteriors µ(d) post and Σ(d) post using Eq. 12 and Eq. 13, with engineered rewards from Eq. 23 end
```

## 4.3 Reward Engineering

To account for delayed effects in the bandit framework, we engineer the reward for the RL algorithm. Note that this engineered reward is only utilized to update the RL algorithm's parameters and hyper-parameters. We are still interested in maximizing the reward defined in Sec. 2, and use it to evaluate the algorithm's performance. The engineered reward ˆR(t) i for user i at decision time t is defined as:

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

where σi,obs is the standard deviation of the observed rewards for a given user i, and λ is a tuned non-negative hyper-parameter. Note that the reward is not penalized when a(t) i = 0. Intuitively, the cost function is designed to allow the RL algorithm to optimize for user engagement, while simultaneously accounting for the delayed effect of sending an intervention message, i.e. a(t) i = 1.

## 5 Experimental Results

In this section, we detail the design of a simulation testbed (Sec. 5.1) to help evaluate the performance of our algorithm. Our experimental setup and the corresponding results are discussed in Sec 5.2.

## 5.1 Simulation Testbed Design

We leverage data from the SARA [Rabbi et al., 2018] study, which trialed an mHealth app aimed at sustaining engagement of substance use data collection from participants. Since the SARA study focused on a similar demographic of EAs as the MiWaves pilot study, it appears ideal for constructing a simulation testbed. However, note that this data is impoverished. SARA had only 1 decision point per day, as compared to 2 per day in MiWaves . The goal of the messages sent to the participants in SARA was to increase survey completion in order to collect substance use data. In contrast, the goal of sending intervention messages in MiWaves pilot study is to reduce the participant's cannabis use through self-monitoring and mobile health engagement. Moreover, the daily cannabis use data in SARA was collected retro-actively at the end of each week, which often resulted in participant's noisy recollection of their cannabis use, and had missing cannabis use data if the participant chose to not respond. In contrast, participants in MiWaves are asked to self-report twice daily, which reduces the amount of missing data if they fail to selfreport once. We construct a base dataset of 42 users after cleaning and imputing the SARA data (please refer to appendix A.1 4 for more details).

Base Model For the base model of the environment, we fit Multinomial Logistic Regression (MLR) models on each of the 42 users in the base dataset. The learnt weights include weights for the baseline (when action is 0), and the advantage (added to the baseline when action is 1). These user models are overfit to learn the user behavior as well capture the noise in the environment. We choose MLR for our user models, as it is interpretable, and performs similar in comparison to a generic neural network (see Appendix A.6 4 ) .

Varying Treatment Effects (TE) The effect of the intervention message on a particular user is measured by their unique treatment effect size. Given that the intervention messages in SARA had minimal treatment effect [Nahum-Shani et al., 2021], we introduce higher levels of treatment effect into the user models by augmenting their weights. Higher levels of treatment effect increase the likelihood of obtaining higher rewards when taking action 1. To that end, we construct TE = low and TE = high treatment effect variants for each MLR user model. Refer to Appendix A.9 4 for more details.

Modeling Habituation (HB) To account for delayed effects in the environment, we introduce user habituation to repeated stimuli (multiple intervention messages sent to the user in a short span of time) by adding a negative effect in the baseline weights of the MLR user models. To that end, we define dosage for each user at each decision point as the weighted average of the number of intervention messages received in the previous six decision points. The weights are decreased with each past decision point, reflecting a diminishing impact of older intervention messages received by the user. Next, we impute baseline weights for dosage in the MLR user models in a way that higher dosage (more messages received) leads to higher likelihood of generating low rewards, and vice-versa. Note that this procedure simulates how the user may experience habituation; if the RL algorithm does not send many interventions to a user experiencing habituation, the user may dis-habituate and recover their baseline behavior. We construct two environment variants - HB = low and HB = high habituation effect - by varying the baseline weights for dosage. Additionally, we simulate the proportion of users who can experience habituation within a population - set at either P = 50% or P = 100%. Please refer to Appendix A.9 4 for more details.

Table 1: Average total reward per user per simulated trial, averaged across 500 simulated trials and 120 users per trial, along with their 95% confidence intervals (CI) for minimal and low treatment effect settings. HB refers to the level of habituation in the environment, while P is used to denote the proportion of the population who can experience habituation.

| Alg.     | Minimal Treatment Effect - HB=None   | Minimal Treatment Effect - HB=Low - P=50%   | Minimal Treatment Effect - HB=Low - P=100%   | Minimal Treatment Effect - HB=High - P=50%   | Minimal Treatment Effect - HB=High - P=100%   | Low Treatment Effect - HB=None   | Low Treatment Effect - HB=Low - P=50%   | Low Treatment Effect - HB=Low - P=100%   | Low Treatment Effect - HB=High - P=50%   | Low Treatment Effect - HB=High - P=100%   |
|----------|--------------------------------------|---------------------------------------------|----------------------------------------------|----------------------------------------------|-----------------------------------------------|----------------------------------|-----------------------------------------|------------------------------------------|------------------------------------------|-------------------------------------------|
| reBandit | 128.54±0.18                          | 127.23±0.18                                 | 126.01±0.18                                  | 123.22±0.19                                  | 119.55±0.20                                   | 129.44±0.17                      | 128.11±0.17                             | 126.80±0.18                              | 123.74±0.19                              | 120.12±0.20                               |
| BLR      | 127.78±0.16                          | 126.60±0.18                                 | 125.78±0.18                                  | 123.23±0.19                                  | 119.60±0.20                                   | 129.10±0.17                      | 127.85±0.17                             | 126.53±0.18                              | 123.75±0.19                              | 120.16±0.20                               |
| random   | 127.83±0.16                          | 126.52±0.18                                 | 125.22±0.18                                  | 119.03±0.21                                  | 110.29±0.23                                   | 128.97±0.17                      | 127.70±0.17                             | 126.45±0.18                              | 120.49±0.20                              | 112.06±0.22                               |

Table 2: Average total reward per user per simulated trial along with their 95%CIs for the high treatment effect settings.

| Alg.     | HB=None     | HB=Low - P=50%   | HB=Low - P=100%   | HB=High - P=50%   | HB=High - P=100%   |
|----------|-------------|------------------|-------------------|-------------------|--------------------|
| reBandit | 132.25±0.16 | 130.95±0.17      | 129.71±0.17       | 124.70±0.19       | 121.19±0.20        |
| BLR      | 132.21±0.16 | 130.94±0.17      | 129.63±0.17       | 124.70±0.19       | 121.22±0.19        |
| random   | 131.05±0.17 | 129.88±0.17      | 128.71±0.17       | 123.19±0.20       | 115.39±0.22        |

## 5.2 Simulation Results

We construct 15 simulation environment variants using a combination of techniques described in Sec. 5.1. For each environment, we simulate 500 studies with m = 120 users each, over a period of D = 30 days (T = 60). The m = 120 users are drawn with replacement from the 42 MLR user models learnt using SARA data.

We compare the performance of our algorithm to two common approaches in mobile health studies. First, is a full pooling algorithm called BLR. BLR utilizes Bayesian Linear Regression [Liao et al., 2019] to pool data and learn a single model across all the users in a study, and select actions according to the action selection procedure mentioned in Sec. 4.2. We use engineered rewards (Sec. 4.3) to update BLR's parameters and hyper-parameters. We also update BLR hyper-parameters using Empirical Bayes, similar to the approach described in Sec. 4.1, for a fair comparison. For both reBandit and BLR, we update the posteriors at the end of each simulated day (every 2 decision points), and the hyper-parameters at the end of each week (every 14 decision points). We refer the reader to Appendix B 4 for more details about BLR's implementation. In addition to BLR, we also compare against the random algorithm, which utilizes an action selection probability of π(t) i = 0.5.

For each algorithm and simulation environment pair, we calculate the average total reward per user per simulated trial, averaged across the 500 simulated trials and 120 users in each trial. We also compute their 95% confidence intervals (CIs). We summarize our findings in Table 1 and 2. In all the simulation environments, reBandit performs no worse than the baseline algorithms. We highlight the environments where reBandit significantly outperforms other algorithms (CIs do not overlap) in green. In the environments where the CIs for the average total reward overlap for reBandit and BLR, we individually compare each of the 500 seeded simulations, and count the number of times reBandit achieved an average total reward per user as compared to BLR. If this number is greater than 50% of the simulations, i.e. greater than 250, we highlight those environments in yellow, otherwise they are highlighted in blue.

The primary takeaway from our simulation results in Tables 1 and 2 is that reBandit is impactful - it performs better than BLR in most environments, and even in the blue highlighted environments where it performs slightly worse than BLR, the performance is still comparable. It is important to note that our procedures to artificially introduce treatment effects and user habituation into the user models reduces the heterogeneity among the user models. This is due to the fact that our procedure to artificially inject treatment effect establishes a non-negative effect of taking an action across all the users in the user models. The same applies to the procedure for introducing user habituation, as it establishes a clear negative effect with respect to dosage across all users in the user models. However, in practice, higher levels of treatment effect or user habituation effect may lead to more heterogeneity in the population. Given that limitation, it is easy to observe that in our simulations, as levels of treatment effect or user habituation effect are increased, the performance gap between reBandit and BLR decreases. In simulation environments characterized by more pronounced heterogeneity due to lower levels of treatment and/or habituation effects, reBandit excels by adeptly identifying and leveraging the heterogeneity within the user population to personalize the likelihood of intervention message delivery and accrues greater rewards.

## 6 Conclusion

In this paper, we introduced reBandit, an online RL algorithm which will be a part of the upcoming mobile health study named MiWaves aimed at reducing cannabis use among emerging adults. We addressed the unique challenges inherent in mobile health studies, including limited data, and requirement for algorithmic autonomy and stability, while designing reBandit. We showed that reBandit utilizes randomeffects and informative Bayesian priors to learn quickly and efficiently in noisy environments which are common in mobile health studies. The introduction of random effects allows reBandit to leverage the heterogeneity in the study population and deliver personalized interventions. To benchmark our algorithm, we detailed the design of a simulation testbed using prior data, and showed that reBandit performs equally well or better than two common approaches used in mobile health studies. In the future, we aim to analyze the effectiveness of the interventions in the MiWaves pilot study. In addition, we aim to investigate the contribution of an individual's data and the study population data towards learning the individual's parameters in the random effects model (see Appendix F 4 ).

## Acknowledgements

Research reported in this paper was supported by NIH/NIDA P50DA054039, and NIH/NIBIB and OD P41EB028242. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

## References

1. [Aouali et al., 2023] Imad Aouali, Branislav Kveton, and Sumeet Katariya. Mixed-effect thompson sampling. In International Conference on Artificial Intelligence and Statistics, pages 2087- 2115. PMLR, 2023.
2. [Basu et al., 2021] Soumya Basu, Branislav Kveton, Manzil Za- heer, and Csaba Szepesvári. No regrets for learning the prior in bandits. Advances in neural information processing systems, 34:28029-28041, 2021.
3. [Carliner et al., 2017] Hannah Carliner, Qiana L Brown, Aaron L Sarvet, and Deborah S Hasin. Cannabis use, attitudes, and legal status in the us: A review. Preventive medicine, 104:13-23, 2017.
4. [Chan et al., 2021] Gary CK Chan, Denise Becker, Peter Butter- worth, Lindsey Hines, Carolyn Coffey, Wayne Hall, and George Patton. Young-adult compared to adolescent onset of regular cannabis use: A 20-year prospective cohort study of later con- sequences. Drug and Alcohol Review, 40(4):627-636, 2021.
5. [Clarke et al., 2017] Shanice Clarke, Luis G Jaimes, and Miguel A Labrador. mstress: A mobile recommender system for just-in- time interventions for stress. In 2017 14th IEEE annual consumer communications & networking conference (CCNC), pages 1-5. IEEE, 2017.
6. [Deshpande et al., 2018] Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive lin- ear models. In International Conference on Machine Learning, pages 1194-1203. PMLR, 2018.
7. [Forman et al., 2019] Evan M Forman, Stephanie G Kerrigan, Meghan L Butryn, Adrienne S Juarascio, Stephanie M Manasse, Santiago Ontañón, Diane H Dallal, Rebecca J Crochiere, and Danielle Moskow. Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss? Journal of behavioral medicine, 42:276-290, 2019.
8. [Golbus et al., 2021] Jessica R Golbus, Walter Dempsey, Eliza- beth A Jackson, Brahmajee K Nallamothu, and Predrag Klasnja. Microrandomized trial design for evaluating just-in-time adap- tive interventions through mobile health technologies for cardio- vascular disease. Circulation: Cardiovascular Quality and Out- comes, 14(2):e006760, 2021.
9. [Greenewald et al., 2017] Kristjan Greenewald, Ambuj Tewari, Su- san Murphy, and Predag Klasnja. Action centered contextual bandits. Advances in neural information processing systems, 30, 2017.
10. [Hall, 2009] Wayne Hall. The adverse health effects of cannabis use: what are they, and what are their implications for policy? International Journal of drug policy, 20(6):458-466, 2009.
11. [Hasin et al., 2016] Deborah Hasin, Bradley Kerridge, Tulshi Saha, Boji Huang, Roger Pickering, Sharon Smith, Jeesun Jung, Haitao Zhang, and Bridget Grant. Prevalence and correlates of dsm- 5 cannabis use disorder, 2012-2013: Findings from the na- tional epidemiologic survey on alcohol and related conditions-iii. American Journal of Psychiatry, 173(6):588-599, 2016.
12. [Hong et al., 2022] Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical bayesian bandits. In International Conference on Artificial Intelligence and Statistics, pages 7724-7741. PMLR, 2022.
13. [Jaimes et al., 2015] Luis G Jaimes, Martin Llofriu, and Andrew Raij. Preventer, a selection mechanism for just-in-time preven- tive interventions. IEEE Transactions on Affective Computing, 7(3):243-257, 2015.
14. [Jiang et al., 2015] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Con- ference on Autonomous Agents and Multiagent Systems, pages 1181-1189, 2015.
15. [Kalvit and Zeevi, 2021] Anand Kalvit and Assaf Zeevi. A closer look at the worst-case behavior of multi-armed bandit algorithms. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8807-8819.
16. [Kramer et al., 2019] Jan-Niklas Kramer, Florian Künzler, Varun Mishra, Bastien Presset, David Kotz, Shawna Smith, Urte Scholz, Tobias Kowatsch, et al. Investigating intervention components and exploring states of receptivity for a smartphone app to pro- mote physical activity: protocol of a microrandomized trial. JMIR research protocols, 8(1):e11540, 2019.
17. [Laird and Ware, 1982] Nan Laird and James Ware. Random- effects models for longitudinal data. Biometrics, pages 963-974, 1982.
18. [Laird, 2004] Nan Laird. Random effects and the linear mixed model. In Analysis of Longitudinal and Cluster-Correlated Data, volume 8, pages 79-96. Institute of Mathematical Statistics, 2004.
19. [Langford and Zhang, 2007] John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Ad- vances in neural information processing systems, 20(1):96-1, 2007.
20. [Lapham et al., 2019] Gwen T Lapham, Cynthia I Campbell, Bobbi Jo H Yarborough, Rulin C Hechter, Brian K Ahmedani, Irina V Haller, Andrea H Kline-Simon, Derek D Satre, Amy M Loree, Constance Weisner, et al. The prevalence of healthcare effective- ness data and information set (hedis) initiation and engagement in treatment among patients with cannabis use disorders in 7 us health systems. Substance Abuse, 40(3):268-277, 2019.
21. [Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news ar- ticle recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.
22. [Liao et al., 2019] Peng Liao, Kristjan H. Greenewald, Predrag V. Klasnja, and Susan A. Murphy. Personalized heartsteps: A re- inforcement learning algorithm for optimizing physical activity. CoRR, abs/1909.03539, 2019.
23. [Morris, 1983] Carl N Morris. Parametric empirical bayes infer- ence: theory and applications. Journal of the American statistical Association, 78(381):47-55, 1983.
24. [Nahum-Shani et al., 2018] Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. Just-in-time adaptive interven- tions (jitais) in mobile health: key components and design prin- ciples for ongoing health behavior support. Annals of Behavioral Medicine, pages 1-17, 2018.
25. [Nahum-Shani et al., 2021] Inbal Nahum-Shani, Mashfiqui Rabbi, Jamie Yap, Meredith L Philyaw-Kotov, Predrag Klasnja, Erin E Bonar, Rebecca M Cunningham, Susan A Murphy, and Mau- reen A Walton. Translating strategies for promoting engage- ment in mobile health: A proof-of-concept microrandomized trial. Health Psychology, 40(12):974, 2021.
26. [Nahum-Shani et al., 2022] Inbal Nahum-Shani, Steven D Shaw, Stephanie M Carpenter, Susan A Murphy, and Carolyn Yoon. En- gagement in digital interventions. American Psychologist, 2022.
27. [Paredes et al., 2014] Pablo Paredes, Ran Gilad-Bachrach, Mary Czerwinski, Asta Roseway, Kael Rowan, and Javier Hernandez. Poptherapy: Coping with stress through pop-culture. In Proceed- ings of the 8th international conference on pervasive computing technologies for healthcare, pages 109-117, 2014.
28. [Peacock et al., 2018] Amy Peacock, Janni Leung, Sarah Larney, Samantha Colledge, Matthew Hickman, Jürgen Rehm, Gary A Giovino, Robert West, Wayne Hall, Paul Griffiths, et al. Global statistics on alcohol, tobacco and illicit drug use: 2017 status re- port. Addiction, 113(10):1905-1926, 2018.
29. [Peleg et al., 2022] Amit Peleg, Naama Pearl, and Ron Meir. Met- alearning linear bandits by prior update. In International Confer- ence on Artificial Intelligence and Statistics, pages 2885-2926. PMLR, 2022.
30. [Rabbi et al., 2015] Mashfiqui Rabbi, Min Hane Aung, Mi Zhang, and Tanzeem Choudhury. Mybehavior: automatic personalized health feedback from user behaviors and preferences using smart- phones. In Proceedings of the 2015 ACM international joint con- ference on pervasive and ubiquitous computing, pages 707-718, 2015.
31. [Rabbi et al., 2018] Mashfiqui Rabbi, Meredith Philyaw Kotov, Re- becca Cunningham, Erin E Bonar, Inbal Nahum-Shani, Predrag Klasnja, Maureen Walton, Susan Murphy, et al. Toward increas- ing engagement in substance use data collection: development of the substance abuse research assistant app and protocol for a mi- crorandomized trial using adolescents and emerging adults. JMIR research protocols, 7(7):e9850, 2018.
32. [Rabbi et al., 2019] Mashfiqui Rabbi, Predrag Klasnja, Tanzeem Choudhury, Ambuj Tewari, and Susan Murphy. Optimizing mhealth interventions with a bandit. Digital Phenotyping and Mobile Sensing: New Developments in Psychoinformatics, pages 277-291, 2019.
33. [Robinson, 1991] George K Robinson. That blup is a good thing: the estimation of random effects. Statistical science, pages 15- 32, 1991.
34. [Russo and Van Roy, 2014] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Op- erations Research, 39(4):1221-1243, 2014. [SAMHSA, 2023] SAMHSA. 2021 NSDUH De- tailed Tables. https://www.samhsa.gov/data/report/ 2021-nsduh-detailed-tables, 2023. Accessed: 2024-02-14.
35. [Shrier et al., 2018] Lydia A Shrier, Pamela J Burke, Meredith Kells, Emily A Scherer, Vishnudas Sarda, Cassandra Jonestrask, Ziming Xuan, and Sion Kim Harris. Pilot randomized trial of moment, a motivational counseling-plus-ecological momentary intervention to reduce marijuana use in youth. Mhealth, 4, 2018.
36. [Simchowitz et al., 2021] Max Simchowitz, Christopher Tosh, Ak- shay Krishnamurthy, Daniel Hsu, Thodoris Lykouris, Miro Dudik, and Robert Schapire. Bayesian decision-making un- der misspecified priors with applications to meta-learning. Ad- vances in Neural Information Processing Systems, 34:26382- 26394, 2021.
37. [Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
38. [Tewari and Murphy, 2017] Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile health: sensors, analytic methods, and applications, pages 495-517, 2017.
39. [Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
40. [Tomkins et al., 2021] Sabina Tomkins, Peng Liao, Predrag Klas- nja, and Susan Murphy. Intelligentpooling: Practical thomp- son sampling for mhealth. Machine learning, 110(9):2685-2727, 2021.
41. [Trella et al., 2022] Anna Trella, Kelly Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, and Susan Murphy. Designing reinforcement learning algorithms for digital interventions: pre- implementation guidelines. Algorithms, 15(8):255, 2022.
42. [Trella et al., 2023] Anna L. Trella, Kelly W. Zhang, Inbal Nahum- Shani, Vivek Shetty, Finale Doshi-Velez, and Susan A. Murphy. Reward design for an online reinforcement learning algorithm supporting oral self-care. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13):15724-15730, Sep. 2023.
43. [Volkow et al., 2014] Nora D Volkow, Ruben D Baler, Wilson M Compton, and Susan RB Weiss. Adverse health effects of mar- ijuana use. New England Journal of Medicine, 370(23):2219- 2227, 2014.
44. [Walsh and Groarke, 2019] Jane C Walsh and Jenny M Groarke. In- tegrating behavioral science with mobile (mhealth) technology to optimize health behavior change interventions. European Psy- chologist, 2019.
45. [Wan et al., 2021] Runzhe Wan, Lin Ge, and Rui Song. Metadata- based multi-task bandits with bayesian hierarchical models. Ad- vances in Neural Information Processing Systems, 34:29655- 29668, 2021.
46. [Wan et al., 2023] Runzhe Wan, Lin Ge, and Rui Song. Towards scalable and robust structured bandits: A meta-learning frame- work. In International Conference on Artificial Intelligence and Statistics, pages 1144-1173. PMLR, 2023.
47. [Wang et al., 2005] Chih-Chun Wang, Sanjeev R Kulkarni, and H Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3):338-355, 2005.
48. [Yom-Tov et al., 2017] Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encourag- ing physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of medical Internet re- search, 19(10):e338, 2017.
49. [Zhang et al., 2022] Kelly W Zhang, Lucas Janson, and Susan A Murphy. Statistical inference after adaptive sampling for longi- tudinal data. arXiv preprint arXiv:2202.07098, 2022.
50. [Zhou et al., 2018] Mo Zhou, Yonatan Mintz, Yoshimi Fukuoka, Ken Goldberg, Elena Flowers, Philip Kaminsky, Alejandro Castillejo, and Anil Aswani. Personalizing mobile fitness apps using reinforcement learning. In CEUR workshop proceedings, volume 2068. NIH Public Access, 2018.
51. [Zhu and Kveton, 2022a] Rong Zhu and Branislav Kveton. Ran- dom effect bandits. In International Conference on Artificial In- telligence and Statistics, pages 3091-3107. PMLR, 2022.
52. [Zhu and Kveton, 2022b] Rong Zhu and Branislav Kveton. Ro- bust contextual linear bandits. arXiv preprint arXiv:2210.14483, 2022.