--- title: "Hyperparameter Optimization Can Even Be Harmful in Off-Policy Learning and How to Deal with It" authors: - "Yuta Saito" - "Masahiro Nomura" doi: "10.24963/ijcai.2024/537" url: "https://ijcai.org/proceedings/2024/537" pdf: "https://ijcai.org/proceedings/2024/0537.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "4860-4867" track: "Main Track" keywords: - "Machine Learning: ML: Hyperparameter optimization" - "Uncertainty in AI: UAI: Causality, structural causal models and causal inference" - "Machine Learning: ML: Multi-armed bandits" - "Machine Learning: ML: Causality" --- # Hyperparameter Optimization Can Even Be Harmful in Off-Policy Learning and How to Deal with It Yuta Saito, Masahiro Nomura *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 4860-4867. DOI: [10.24963/ijcai.2024/537](https://doi.org/10.24963/ijcai.2024/537) ## Abstract There has been a growing interest in off-policy evaluation in the literature such as recommender systems and personalized medicine. We have so far seen significant progress in developing estimators aimed at accurately estimating the effectiveness of counterfactual policies based on biased logged data. However, there are many cases where those estimators are used not only to evaluate the value of decision making policies but also to search for the best hyperparameters from a large candidate space. This work explores the latter hyperparameter optimization (HPO) task for off-policy learning. We empirically show that naively applying an unbiased estimator of the generalization performance as a surrogate objective in HPO can cause an unexpected failure, merely pursuing hyperparameters whose generalization performance is greatly overestimated. We then propose simple and computationally efficient corrections to the typical HPO procedure to deal with the aforementioned issues simultaneously. Empirical investigations demonstrate the effectiveness of our proposed HPO algorithm in situations where the typical procedure fails severely. ## Full text ## 1 Introduction Interactive decision making systems, such as recommender systems, produce logged data valuable for optimizing future decision making. For example, the logs of an e-commerce recommender system record which product was recommended and whether the users purchased it, giving the system designer a rich logged dataset useful for evaluating and improving the decision making quality. This type of historical data is often called logged bandit data and is one of the most ubiquitous forms of data available in many real-life applications [Swaminathan and Joachims, 2015a; Su et al., 2020; Kiyohara et al., 2021; Saito et al., 2024]. Off-Policy Learning (OPL) aims to train a new decision making policy using only the logged bandit data. OPL is useful in that it can improve the decision making system continuously in a batch manner without requiring a risky exploration. Owing to the ubiquity of logged bandit data in the real-world, significant attention has been paid to OPL of contextual bandits [Strehl et al., 2010; Swaminathan and Joachims, 2015a; Swaminathan and Joachims, 2015b; Wang et al., 2017; Kallus et al., 2021; Kiyohara et al., 2023; Kiyohara et al., 2024]. The fundamental problem in OPL is that the outcome is only observed for the action chosen by the system in the past. Thus, estimating the generalization performance of a policy is nontrivial because we cannot naively apply the empirical risk as done in typical supervised machine learning (ML). Therefore, a variety of estimators have been developed in the field of off-policy evaluation (OPE), such as Inverse Propensity Score (IPS) [Precup et al., 2000] and Doubly Robust (DR) [Dud´ık et al., 2014]. Then, a feasible approach to OPL is to maximize one such estimator as a surrogate objective using only the logged data. Hyperparameter optimization (HPO) can also be performed based on one of the estimators on a validation set of the logged data [Paine et al., 2020]. In this study, we investigate how well automatic HPO algorithms work for OPL using only the available logged data. In particular, we empirically find two critical issues in HPO that have yet to be investigated in the literature, but can have a significant adverse impact on the effectiveness of the OPL pipeline. The first issue is optimistic bias, which implies that the hyperparameter values selected by an HPO procedure are often the ones whose performance is greatly overestimated. In HPO, we often use an unbiased estimator as a strategy to optimize the generalization performance (primary objective) using only validation data. The problem is that, when optimizing the validation performance as a surrogate objective, HPO can identify a set of hyperparameters whose validation performance looks good but its generalization performance is detrimental. As a result, the typical HPO procedure often produces a highly sub-optimal solution, even with an unbiased estimator of the generalization performance. The second issue is unsafe behavior, which suggests that the typical HPO procedure can output a solution, which underperforms the logging (data collection) policy, even when we set the logging policy as an initial solution. This is problematic because a logging policy is often a baseline policy to improve upon in OPL. If an HPO procedure aggravates the performance of the logging policy, there is no need to implement it in practice. After formulating the problem in Section 2, Section 3 provides clear empirical evidence of optimistic bias and unsafe behavior. We observe these phenomena even when we use an unbiased surrogate objective and a popular adaptive HPO algorithm. We also explain these observations theoretically, demonstrating that ignoring the fact that HPO optimizes the validation performance as a surrogate of the generalization performance can lead to a worse regret of HPO algorithms. More specifically, we identify that a heavy-tailed distribution of overestimation bias during HPO can cause an unexpected gap between the generalization and validation regret. These empirical and theoretical observations result in our proposed corrections to the typical HPO procedure, which we describe in Section 4. Finally, Section 5 conducts comprehensive experiments and demonstrates that our simple corrections can deal with the aforementioned issues and improve the typical procedure, particularly for cases where the typical procedure becomes unsafe and underperforms the logging policy. 1 ## 2 Preliminaries We use x ∈ X to denote a context vector and a ∈ A to denote a (discrete) action such as a playlist recommendation in a music streaming service. Let r ∈ [0, rmax] denote a reward variable, which is sampled identically and independently from an unknown conditional distribution p(r|x, a). A decision making policy is modeled as a distribution over the action space, i.e., π : X → ∆(A) where ∆(·) is a probability simplex. We can then represent the probability of action a being taken by policy π given context x as π(a|x). ## 2.1 Off-Policy Evaluation and Learning In OPE, we are given logged bandit data D := {(xi, ai, ri)}ni=1 consisting of n independent draws from the logging policy π0. Using this logged dataset, OPE aims to estimate the generalization performance of a given evaluation policy πe, which is often different from π0: This is the ground-truth performance of the evaluation policy when deployed in an environment of interest. OPE uses an estimator ˆ V to estimate V (πe) based only on D as V (πe) ≈ ˆ V (πe;D). A typical choice of ˆ V is IPS: where πe(ai|xi)/π0(ai|xi) is called the importance weight. Under some assumptions for identification such as full support (πe(a|x) > 0 → π0(a|x) > 0, ∀(x, a)), IPS provides an unbiased estimate of the generalization policy performance, i.e., ED[ ˆ VIPS(πe; D)] = V (πe). Beyond IPS, significant efforts have been made to enable a more accurate OPE from the logged data [Dud´ık et al., 2014; Wang et al., 2017; Su et al., 2020; Saito et al., 2023]. In OPL, we aim to learn an optimal decision making policy π∗ := arg max π V (π) from the logged data. As in supervised 1 Appendix B provides a comprehensive survey of related work. Note that the arXiv version of the paper, available at https://arxiv.org/abs/2404.15084, includes the full text with appendices. Algorithm 1 Typical HPO with IPS as a surrogate (Baseline) Input: A, Θ, T , Dtr, Dval Output: ˆ θ - 1: π∗ ← π0 , S0 ← ∅ - 2: for t = 1, 2, . . . , T do - 3: θt ←A(θ | St-1) // sample candidate hyperparameters - 4: πt ← ˆπ(· | θt,Dtr) // train a policy (lower-level) - 5: if ˆ VIPS (πt; Dval) > ˆ VIPS (π∗ ;Dval) then - 6: ˆ θ ← θt, π∗ ← πt // update the solution - 7: end if - 8: St ←St-1 ∪ {(θt, ˆ VIPS(πt;Dval))} // store the result - 9: end for ML, we cannot directly use the generalization policy performance. Instead, we use its estimator as a surrogate: where Π is a policy class, which might be a linear class [Swaminathan and Joachims, 2015a] or deep neural nets [Joachims et al., 2018]. R(·) regularizes the complexity of the policy π, and λ(≥ 0) is a hyperparamter that controls the effect of regularization. ## 2.2 Hyperparameter Optimization OPE involves many hyperparameters to be properly tuned from those defining the policy class Π to the regularization parameter λ. In a typical HPO procedure for OPL, we first split the original logged bandit data D into training (Dtr) and validation (Dval) sets. Then, we wish to solve the following bi-level optimization: where Θ is a pre-defined hyperparamter search space. ˆπ(·; θ, Dtr) is a policy parameterized by a set of hyperparameters θ. The model parameter of ˆπ(·; θ,Dtr) is trained on the training set Dtr (lower-level optimization). The problem here is that the generalization performance of ˆπ(·; θ,Dtr) is unknown and needs to be estimated. A feasible HPO procedure based on an estimated policy performance is: where the generalization performance of ˆπ(·; θ,Dtr) is estimated by an estimator ˆ V on the validation set Dval. 2 A common choice of ˆ V is an unbiased estimator that satisfies E[ ˆ V (π; Dval)] = V (π), ∀π ∈ Π such as IPS. Then, one can apply grid search, random search [Bergstra and Bengio, 2012], or adaptive methods such as tree-structured Parzen estimator (TPE) [Bergstra et al., 2011] to solve the higher-level optimization in Eq. (3) efficiently. Algorithm 1 describes this typical HPO procedure for OPL, which starts from the logging policy π0 as its initial solution and adaptively samples promising hyperparameters via an arbitrary HPO algorithm (denoted here as A) [Tang and Wiens, 2021]. 2 For brevity of notation, we sometimes use V (θ) and ˆ V (θ;D) to denote the generalization and validation performances of the policy induced by θ. Figure 1: Empirical Evidence of Optimistic Bias and Unsafe Behavior in HPO for OPL (w/ TPE). The results are averaged over 25 runs with different seeds and then normalized by V (π0). The shaded regions indicate 95% confidence intervals. ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000000_37f25dd5fb2169cc23a704f94bec50ea1d31743783dfd1aee00e5f0deeacf3bd.png) ## 3 Unexpected Failure in HPO for OPL This section studies the effectiveness of HPO when applied to OPL from both empirical and theoretical perspectives. ## 3.1 Empirical Analysis First, we conduct a synthetic experiment and provide empirical evidence of surprising failure of HPO in OPL. ## Synthetic Data Our empirical analysis is based on OpenBanditPipeline (OBP) 3 , an open-source toolkit for OPE and OPL, which includes synthetic data generation modules and a range of estimators [Saito et al., 2021]. We synthesize context vectors x by sampling them from a 10-dimensional standard normal distribution. We then set |A| = 10, where each action a ∈ A is characterized by a 10-dimensional representation vector ea. The reward function µ(x, a) := E[r | x, a] is defined as: where σ(z) := 1/(1 + exp(-z)) is the sigmoid function. M, ηx, and ηa are parameter matrices or vectors for defining the synthetic reward function. These parameters are sampled from a uniform distribution with range [-1, 1]. After generating the synthetic reward function, we sample binary rewards from a Bernoulli distribution with parameter µ(x, a). We then define the logging policy π0 by applying the softmax function to the reward function µ(x, a) as follows. where β0 is an inverse temperature parameter to control the optimality and entropy of the logging policy. A large positive value of β0 leads to a near-deterministic and near-optimal logging policy. When β0 = 0, π0 is uniform. ## Policy Class and HPO Algorithms To train a new policy π from only the logged data, we first estimate µ(x, a) by a supervised ML method, where the resulting estimator is denoted as ˆµ(x, a; Dtr). We then form a stochastic policy by applying the softmax rule as: 3 https://github.com/st-tech/zr-obp where β is an inverse temperature parameter to define a new policy. θ is a set of hyperparameters, which consists of β, supervised ML model to construct ˆµ, and the hyperparameters of ˆµ. The hyperparameter search space Θ is summarized in Table 1 in Appendix E. As an HPO algorithm, we use TPE [Bergstra et al., 2011], which is a popular adaptive method in the HPO community [Akiba et al., 2019]. TPE has been shown to work well for HPO of supervised ML, however, whether it also works for OPL has never been thoroughly investigated. ## Observations In this synthetic experiment, we set β0 ∈ {0, 3, 20} and |Dtr| = |Dval| = 1, 000. The number of trials (T in Algorithm 1) for HPO is set to 1,000. Figure 1 shows the validation performance ( ˆ VIPS(π;Dval); what HPO algorithm maximizes from the logged data) and the generalization performance (V (π); the primary objective of OPL) during the HPO procedure. We obtain the following key observations in this experiment. 1. Optimistic Bias: For all β0, TPE succeeds in maximizing the validation performance, monotonically improving the blue lines. However, there is a substantial gap between validation and generalization, and the validation performance becomes an extremely optimistic proxy of the generalization performance. For example, when β0 = 3, TPE does not bring any impact on the generalization performance, although the validation performance is greatly improved. This result suggests that implementing HPO is indeed a waste of time and resources for this setting. 2. Unsafe Behavior: When β0 = 20 (where π0 is already much better than uniform random), TPE outputs a solution that is significantly worse than the logging policy with respect to the generalization performance. This is problematic, as the solution at the final trial seems to provide a substantial improvement over the logging policy with respect to the unbiased validation performance (blue lines). In reality, we have no access to the generalization performance (red lines), making it impossible to detect this performance degradation, possibly deploying an unsafe policy in the field without even noticing it. These observations suggest that optimizing an unbiased surrogate objective is not an ideal strategy and is even harmful in some cases regarding the optimization of the generalization performance. Note that we obtain similar results when random search (RS) is used as an HPO algorithm and DR is used as an OPE estimator as reported in Appendix E. In particular, comparing RS with TPE in terms of the generalization performance, we find that there are no particular differences between the two algorithms for β0 = 0, 3. Even more surprisingly, when β0 = 20, TPE is outperformed by RS, even if TPE is better at optimizing the validation performance. These results further suggest that merely optimizing an unbiased surrogate objective is not a suitable approach for optimizing the generalization performance in HPO of OPL. ## 3.2 Theoretical Analysis Next, we investigate the mechanism causing the somewhat surprising issues observed in the previous section. 4 First, we explain the phenomena from a statistical perspective. Proposition 3.1. Given that ˆ V is unbiased, we have the following inequalities. where ED[·] takes expectation over every randomness in the logged data D, and ED[ ˆ V (ˆ θ(D); D)] - ED[V (ˆ θ(D))] is the amount of optimistic bias. Note that, in Eq. (2), V (θ∗ ) is defined as the best generalization performance we could achieve with HPO. Thus, the first inequality in Eq. (7) suggests that the validation performance of the HPO solution ˆ θ(Dval) is better than the best achievable generalization performance in expectation, suggesting that the performance estimation of the HPO solution is optimistic in general. In addition, the second inequality in Eq. (7) implies that the generalization performance of ˆ θ(Dval) is worse than the best achievable generalization performance in expectation, even though the validation performance of ˆ θ(Dval) is likely to be better. As a result, we will often be disappointed with the performance of the HPO solution ˆ θ even with an unbiased surrogate (validation) objective. Overall, Proposition 3.1 explains the substantial gap between the blue (E[ ˆ V (ˆ θ(Dval))]) and red (E[V (ˆ θ(Dval))]) lines observed in Figure 1. Next, we analyze "regret" to understand what causes the optimistic bias in Proposition 3.1 and how we can deal with it. For this, we define two variants of regret, which measure the difference between the validation or generalization performances of the optimal hyperparameter and HPO solution. 4 Appendix C provides proofs omitted in the main text. Figure 2: Distributions of Overestimation Bias (β0 = 3) ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000001_efd4666b08c88fd7577f11d64e72b22104412874e002868d7f592cab9acdbe06.png) where θ∗ := arg max θ∈Θ V (θ) is the optimal hyperparame- ter with respect to the generalization performance. ˆ θ∗ = arg max θ∈Θ ˆ VIPS(θ; D) denotes the optimal hyperparameter with respect to the validation performance, and ˆ θT,A(D) is the solution of Algorithm 1 given budget T and algorithm A. We also define the overestimation bias for a specific hyperparameter θ as τ(θ;D) = ˆ VIPS(θ;D) -V (θ). Then, the following implies that a heavy-tailed distribution of overestimation bias during HPO can produce an unexpected gap between the generalization and validation regret. Proposition 3.2. Given HPO algorithm A, budget T, and logged data D, the generalization regret can be written as Only the first two terms of the RHS in Eq. (10) depend on the HPO solution ˆ θT,A(D), and are thus critical for analyzing the HPO performance. The first term rval is the validation regret. Under some mild conditions, we can achieve no-regret (rval(T ; A, D) = o(1)) with optimization methods such as GP-UCB [Srinivas et al., 2010], as we can target the validation performance directly using available data. The second term∆τ(ˆ θT,A(D), θ∗ ;D) is the difference in the extent of overestimation between ˆ θT,A(D) and θ∗ . When the extent of overestimation of ˆ θT,A(D) is larger than that of θ∗ , ∆τ (ˆ θT,A(D), θ∗ ;D) becomes large. Therefore, Proposition 3.2 suggests that the overestimation bias of ˆ θT,A(D) can exacerbate the generalization regret of HPO algorithms. More specifically, if an HPO algorithm is likely to sample many hyperparameters whose performance is overestimated ( ˆ VIPS(θ)-V (θ) > 0) and the overestimation bias has a heavytailed distribution, the second term of Eq. (10) tends to become large, so does the generalization regret rgen. Given this regret analysis, we investigate the distributions of overestimation bias observed in the empirical analysis in Figure 2. This figure implies that TPE more frequently samples hyperparameters incurring a large overestimation bias than RS. According to Proposition 3.2, this is why we do not find the advantage of TPE with respect to the generalization performance. RS has a worse validation regret than TPE, while overestimation bias of RS is not very problematic compared to TPE. As a result, RS performs similarly to or slightly better than TPE in terms of the generalization performance. In this way, the heavy-tailed distribution of overestimation bias makes the generalization regret of HPO algorithms (in particular TPE) worse than its validation regret, resulting in optimistic bias and possibly unsafe behavior. ## 4 How ShouldWe Deal with the Issues? In this section, we propose two simple corrections, namely (i) conservative surrogate objective and (ii) adaptive imitation regularization, to deal with the critical issues in HPO. We also describe the resulting HPO procedure, which we call Conservative and Imitation-Regularized HPO (CIR-HPO). ## 4.1 Conservative Surrogate Objective (CSO) First, we address the heavy-tailed distribution of overestimation bias ( ˆ VIPS(π) - V (π)) during HPO, as suggested in Figure 2. Proposition 3.2 implies that the overestimation of the value of hyperparameters sampled during HPO can exacerbate the generalization regret of an HPO algorithm. To deal with this issue, we introduce conservative surrogate objective, which penalizes the validation performance of hyperparameters whose performance has a large uncertainty to avoid the issue of overestimation bias during HPO. Specifically, we propose to use a high probability lower bound of the generalization performance (denoted as ˆ V-(·)) as an alternative surrogate objective, which is given as: P(V (π) ≥ ˆ V-(π; D, δ)) ≥ 1-δ where δ ∈ (0, 1) specifies a confidence level. A prevalent strategy to construct ˆ V-(·) in OPE is to apply a concentration inequality such as Hoeffding and Bernstein [Thomas et al., 2015b; Thomas et al., 2015a]. A problem is that these inequalities are often overly conservative as they make no assumptions about underlying distribution. Thus, we use an alternative strategy to construct ˆ V-(·) based on the Student's t-distribution as follows. where t1-δ,ν is the T-value given confidence level δ and degrees of freedom ν. The upside of Eq. (11) is that it produces a tighter lower bound than aforementioned concentration inequalities. This is because Eq. (11) introduces the additional assumption that the mean of importance weighted rewards (π/π0)r is normally distributed. This assumption is reasonable with growing data sizes. However, (π/π0)r often follows a distribution with heavy upper tails, which may make the assumption invalid in a small sample setting. Nonetheless, Appendix E empirically verifies that Eq. (11) is reasonably tight compared with other popular concentration inequalities. ## 4.2 Adaptive Imitation Regularization (AIR) The second technique we propose is adaptive imitation regularization, which tackles the unsafe behavior of the typical procedure. The issue of unsafe behavior suggests that, if logging ## Algorithm 2 Conservative and Imitation-Regularized HPO Input: A, δ, γ, Θ, T, π0, Dtr, Dval Output: ˆ θ - 1: S0 ← ∅ - 2: for t = 1, 2, . . . , T do - 3: θt ←A(θ | St-1) // sample candidate hyperparameters - 4: ˆπt ← ˆπ(· | θt,Dtr) // train a policy (lower-level) - 5: πt ← (1-αt) · ˆπt +αt · π0 // regularization (Eq. (14)) - 7: ˆ θ ← θt, π∗ ← πt // update the solution - 8: end if - 9: St ←St-1 ∪ {(θt, ˆ V t-(πt;Dval, δ))} // store the result 10: end for policy π0 is better than uniform random or is near-optimal, Algorithm 1 can produce a solution whose performance is much worse than that of the logging policy. Avoiding this problem is non-trivial, because we do not have access to the generalization performance and do not know the optimality of the logging policy in practice. For example, simply setting π0 as an initial solution does not solve the issue at all, as suggested in Section 3.1. An instant idea might be to imitate the logging policy to some extent: where θt is a set of hyperparameters sampled at the t-th trial. α(∈ [0, 1]) is a regularization parameter, which mixes the policy induced by θt and π0 to construct a policy to evaluate. A large value of α makes πt closer to the logging policy, possibly avoiding the unsafe behavior. However, if the logging policy is detrimental, we should use a small α so that we can avoid an unnecessary performance degradation. So, a natural question to ask here is: how should we set the regularization parameter α? Again, this problem is non-trivial, as the optimality of the logging policy is unknown when performing HPO. To overcome this difficulty in correctly setting α, we propose adaptively tuning this parameter over the course of HPO. Based on the previous discussion, we should apply a strong regularization if π0 performs well, otherwise we should not imitate π0. A key idea here is that we can reason about the optimality of the logging policy by comparing it with solutions sampled during HPO, i.e., {ˆπ(a|x; θt,Dtr)}Tt=1 . If most of the sampled solutions underperform π0, we can infer that the logging policy is well-performing. To make a valid comparison between the sampled solutions and the logging policy, we apply a Student's t-test based on the following T-value. where ∆ ˆ VIPS (π1, π2) := ˆ VIPS(π1) - ˆ VIPS(π2) is the performance difference between the two policies estimated by IPS. Given a null hypothesis (∆ˆ VIPS (π1, π2) = 0) and a normality assumption, T(π1, π2) follows a t-distribution with ν degrees of freedom. We then calculate the optimality score of π0 at the t-th trial as follows. Figure 3: Comparing CIR-HPO (our proposal) and Baseline by their generalization performance. The results are averaged over 25 runs with different seeds and then normalized by V (π0). The shaded regions indicate 95% confidence intervals. ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000002_8399a4c1a36f1d9f2b3ba0c282dde644e6679be1e7a53752281c7b3337d01a96.png) Figure 4: Behavior of adaptive regularization parameter (αt) of CIRHPO with varying values of β0 ∈ {-3, 0, 3, 10, 20}. ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000003_847873f5983b8d7f39aab916c77f313ee04cd72f0e3a7da6af4ed4eaf92115a4.png) st indicates whether π0 is better or worse than πt in a significant level. If π0 is better than πt, then st = 1. Instead, st = -1 if π0 is tested to be worse. If there is no significant difference between π0 and πt, the score is zero. Using the sequence of scores up to the t-th trial, i.e., {st′}tt′=1 , we define adaptive regularization parameter as: where αinit ∈ [0, 1] is an initial regularization parameter and γ (> 0) is a scheduling parameter for adaptive regularization. For example, suppose that st = 1, ∀t = 1, 2, . . . , T , meaning that π0 is always better than πt in a significant level. Then, following Eq. (14), αT = 1 and the HPO procedure outputs π0, because it should be near-optimal. On the other hand, if st = -1, ∀t = 1, 2, . . . , T , meaning that π0 is always worse than πt in a significant level, then αT = 0 and the HPO procedure does not imitate the logging policy at all, because it should be a bad policy. ## 4.3 The CIR-HPOAlgorithm Algorithm 2 describes the CIR-HPO algorithm, which leverages conservative surrogate objective (lines 6 and 9) and adaptive imitation regularization (line 5). δ and γ are meta hyperparameters. δ controls how conservative we would like to be during HPO, and γ controls the scheduling of the adaptive regularization. In Section 5, we show that these configurations have some impact on the behavior of CIR-HPO, but we also demonstrate that the default values (δ = 0.1 and γ = 0.01) work reasonably well in a range of experiment settings. The other inputs are the same as those of Algorithm 1. Note that our algorithm is easy to implement with a few additional lines of code and there is no additional computational overhead compared to the typical procedure in Algorithm 1. ## 5 Empirical Evaluation This section empirically compares Baseline (Algorithm 1) and CIR-HPO (Algorithm 2), employing the same synthetic data and policy class as in Section 3.1. Note that we compare CIRHPO against only Baseline because there is no other method proposed for HPO using logged bandit data (comprehensive summary of related work can be found in Appendix B). ## 5.1 Baseline vs CIR-HPO Figure 3 compares the performance of Baseline and CIRHPO with varying logging policies (β0 ∈ {0, 3, 20}). First, when β0 = 0 where the logging policy is uniform random, both Baseline and CIR-HPO work reasonably well and succeed in finding a set of hyperparameters that leads to a policy much better than the logging policy. What is notable for this setting is that CIR-HPO is inefficient and slow to converge compared to Baseline due to adaptive imitation regularization, even though it reaches far above the black horizontal line (V (π0)). At the initial stage of HPO, we do not know how close the logging policy is to the optimal policy. Therefore, the proposed procedure gradually learns the optimality of the logging policy, potentially leading to a slower convergence if the logging policy is far from optimal (such as uniform random). Next, when β0 = 3 where the logging policy is better than uniform random, but is not close to the optimal, both Baseline and CIR-HPO slightly improve the logging policy. However, the confidence intervals indicate that CIR-HPO is much more stable than Baseline. In particular, Baseline is much more likely to underperform the logging policy, even though it outperforms the logging policy on average. Finally, when β0 = 20 where the logging policy is near-optimal, Baseline outputs a solution that is substantially worse than the logging policy, even though it starts from the logging policy as its initial solution. In contrast, CIR-HPO learns that the logging policy is near-optimal during HPO and strengthens the imitation regularization adaptively. As a result, it prevents the solution from being significantly worse than the (already near-optimal) logging policy, which is compelling, because we do not know the optimality of the logging policy in advance. Figure 4 illustrates the behavior of adaptive imitation regularization, which suggests that it succeeds in controlling the strength of regularization depending on the optimality of the logging policy. Figure 5: Sensitivity of the generalization performance of CIR-HPO regarding the choice of δ. ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000004_0f4abafde11b8fb3b2bba0faf8bf58e8bd72d4317bcae7148e348fe3deb65972.png) Figure 6: Sensitivity of the generalization performance of CIR-HPO regarding the choice of γ. ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000005_b369f78c00d32b2b83d6f42dccd0fe2d1bcb3aac5503af207167b10a692b429b.png) ## 5.2 Choice of Meta Hyperparameters Next, we evaluate the sensitivity of CIR-HPO to the choice of its meta hyperparameters. Figure 5 shows that the effectiveness of CIR-HPO with different values of δ. The result demonstrates that there is no significant difference among the three values, suggesting that we do not have to care too much about which value to use for δ. In addition, Figure 6 evaluates different values of γ, which controls the scheduling of adaptive imitation regularization. This result implies that, for a sub-optimal logging policy (β0 = 0), the choice of γ has no significant effect on the behavior of CIR-HPO. For a near-optimal logging policy (β0 = 20), however, a smaller γ Figure 7: Ablation study of CIR-HPO (β0 = 20). ![Image](https://ijcai.org/proceedings/2024/fulltext/537/image_000006_bb083683aaf575c517e354c91738c7573c10648cd9889b5e902d9267de11b373.png) leads to a faster convergence, although all values achieve the same level of performance in the final stage. ## 5.3 Ablation Study We also conduct an ablation study to evaluate the contribution of conservative surrogate objective (CSO) and adaptive imitation regularization (AIR) to the effectiveness of CIRHPO. To this end, we compare CIR-HPO to CIR-HPO (w/o CSO) and CIR-HPO (w/o AIR) in Figure 7. The result demonstrates that both CSO and AIR clearly contribute to the performance of CIR-HPO, while AIR has a more appealing effect (CSO and AIR provide 1.6% and 23.6% improvements, respectively, in terms of the final generalization performance). ## 5.4 AReal-World Experiment In addition to the synthetic experiments, we apply CIR-HPO to the Open Bandit Dataset [Saito et al., 2021], a publicly available logged bandit dataset collected on a large-scale fashion e-commerce platform. The results suggest that CIR-HPO leads to a better policy compared to the Baseline procedure in terms of the generalization performance, providing a further argument regarding its real-world applicability. The experiment detail and results can be found in Appendix A. ## 6 Conclusion This work studies the effectiveness of the typical HPO procedure in the OPL setup from both empirical and theoretical perspectives and found that it can fail and even be harmful. In particular, we investigated two surprising issues, namely optimistic bias and unsafe behavior, and showed that a heavytailed distribution of overestimation can cause an unexpected gap between validation and generalization. In response, we made two extremely simple corrections to the typical HPO procedure, resulting in the CIR-HPO algorithm, to deal with the issues. Extensive experiments demonstrated that CIR-HPO can be advantageous, particularly when the conventional procedure collapses and causes a significant and undetectable deterioration in the generalization performance. ## References 1. [Akiba et al., 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pages 2623-2631, New York, NY, USA, 2019. ACM. 2. [Bergstra and Bengio, 2012] James Bergstra and Yoshua Ben- gio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012. 3. [Bergstra et al., 2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011. 4. [Dudík et al., 2014] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly Robust Policy Evaluation and Optimization. Statistical Science, 29:485-511, 2014. 5. [Joachims et al., 2018] Thorsten Joachims, Adith Swami- nathan, and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations, 2018. 6. [Kallus et al., 2021] Nathan Kallus, Yuta Saito, and Masatoshi Uehara. Optimal off-policy evaluation from multiple logging policies. In International Conference on Machine Learning, pages 5247-5256. PMLR, 2021. 7. [Kiyohara et al., 2021] Haruka Kiyohara, Kosuke Kawakami, and Yuta Saito. Accelerating offline reinforcement learning application in real-time bidding and recommendation: Po- tential use of simulation. arXiv preprint arXiv:2109.08331, 2021. 8. [Kiyohara et al., 2023] Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. Towards assessing and benchmarking risk- return tradeoff of off-policy evaluation. In The Twelfth In- ternational Conference on Learning Representations, 2023. 9. [Kiyohara et al., 2024] Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. Off-policy evaluation of slate ban- dit policies via optimizing abstraction. arXiv preprint arXiv:2402.02171, 2024. 10. [Paine et al., 2020] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter se- lection for offline reinforcement learning. arXiv preprint arXiv:2007.09055, 2020. 11. [Precup et al., 2000] Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility Traces for Off-Policy Policy Evaluation. In Proceedings of the 17th International Con- ference on Machine Learning, pages 759-766, 2000. 12. [Saito et al., 2021] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. Open bandit dataset and pipeline: Towards realistic and reproducible off-policy eval- uation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 13. [Saito et al., 2023] Yuta Saito, Ren Qingyang, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. In International Conference on Machine Learning, pages 29734-29759. PMLR, 2023. 14. [Saito et al., 2024] Yuta Saito, Jihan Yao, and Thorsten Joachims. Potec: Off-policy learning for large action spaces via two-stage policy decomposition. arXiv preprint arXiv:2402.06151, 2024. 15. [Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Confer- ence on International Conference on Machine Learning, pages 1015-1022, 2010. 16. [Strehl et al., 2010] Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from Logged Implicit Exploration Data. In Advances in Neural Information Pro- cessing Systems, volume 23, pages 2217-2225, 2010. 17. [Su et al., 2020] Yi Su, Maria Dimakopoulou, Akshay Krish- namurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning, volume 119, pages 9167-9176. PMLR, 2020. 18. [Swaminathan and Joachims, 2015a] Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755, 2015. 19. [Swaminathan and Joachims, 2015b] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems, volume 28, pages 3231-3239, 2015. 20. [Tang and Wiens, 2021] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practi- cal considerations for healthcare settings. arXiv preprint arXiv:2107.11003, 2021. 21. [Thomas et al., 2015a] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High- confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015. 22. [Thomas et al., 2015b] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In Proceedings of the 32th International Conference on Machine Learning, volume 37, pages 2380-2388, 2015. 23. [Wang et al., 2017] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and Adaptive Off-policy Evalua- tion in Contextual Bandits. In Proceedings of the 34th In- ternational Conference on Machine Learning, pages 3589- 3597, 2017.