{"type":"ScholarlyArticle","title":"Hyperparameter Optimization Can Even Be Harmful in Off-Policy Learning and How to Deal with It","authors":[{"name":"Yuta Saito","slug":"yuta-saito","given":"Yuta","family":"Saito","affiliation":"Cornell University , 2 CyberAgent ,"},{"name":"Masahiro Nomura","slug":"masahiro-nomura","given":"Masahiro","family":"Nomura","affiliation":"Cornell University , 2 CyberAgent ,"}],"abstract":"There has been a growing interest in off-policy evaluation in the literature such as recommender systems and personalized medicine. We have so far seen significant progress in developing estimators aimed at accurately estimating the effectiveness of counterfactual policies based on biased logged data. However, there are many cases where those estimators are used not only to evaluate the value of decision making policies but also to search for the best hyperparameters from a large candidate space. This work explores the latter hyperparameter optimization (HPO) task for off-policy learning. We empirically show that naively applying an unbiased estimator of the generalization performance as a surrogate objective in HPO can cause an unexpected failure, merely pursuing hyperparameters whose generalization performance is greatly overestimated. We then propose simple and computationally efficient corrections to the typical HPO procedure to deal with the aforementioned issues simultaneously. Empirical investigations demonstrate the effectiveness of our proposed HPO algorithm in situations where the typical procedure fails severely.","doi":"10.24963/ijcai.2024/537","url":"https://ijcai.org/proceedings/2024/537","pdf":"https://ijcai.org/proceedings/2024/0537.pdf","markdown":"https://ijcai.org/proceedings/2024/537.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/537","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":537,"pages":{"first":4860,"last":4867},"keywords":["Machine Learning: ML: Hyperparameter optimization","Uncertainty in AI: UAI: Causality, structural causal models and causal inference","Machine Learning: ML: Multi-armed bandits","Machine Learning: ML: Causality"],"videos":[],"references":[{"position":1,"raw":"[Akiba et al., 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pages 2623-2631, New York, NY, USA, 2019. ACM.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bergstra and Bengio, 2012] James Bergstra and Yoshua Ben- gio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.","title":"Random search for hyper-parameter optimization","authors":"Bengio Bergstra","year":"2012","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergstra et al., 2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.","title":"Algorithms for hyperparameter optimization","authors":"James Bergstra, Rémi Bardenet, Yoshua Bengio, Balázs Kégl","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dudík et al., 2014] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly Robust Policy Evaluation and Optimization. Statistical Science, 29:485-511, 2014.","title":"Doubly Robust Policy Evaluation and Optimization","authors":"Dudík","year":"2014","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Joachims et al., 2018] Thorsten Joachims, Adith Swami- nathan, and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations, 2018.","title":"Deep learning with logged bandit feedback","authors":"Thorsten Joachims, Adith Swaminathan, Maarten De Rijke","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kallus et al., 2021] Nathan Kallus, Yuta Saito, and Masatoshi Uehara. Optimal off-policy evaluation from multiple logging policies. In International Conference on Machine Learning, pages 5247-5256. PMLR, 2021.","title":"Optimal off-policy evaluation from multiple logging policies","authors":"Nathan Kallus, Yuta Saito, Masatoshi Uehara","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kiyohara et al., 2021] Haruka Kiyohara, Kosuke Kawakami, and Yuta Saito. Accelerating offline reinforcement learning application in real-time bidding and recommendation: Po- tential use of simulation. arXiv preprint arXiv:2109.08331, 2021.","title":"Accelerating offline reinforcement learning application in real-time bidding and recommendation: Potential use of simulation","authors":"Haruka Kiyohara, Kosuke Kawakami, Yuta Saito","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kiyohara et al., 2023] Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. Towards assessing and benchmarking risk- return tradeoff of off-policy evaluation. In The Twelfth In- ternational Conference on Learning Representations, 2023.","title":"Towards assessing and benchmarking riskreturn tradeoff of off-policy evaluation","authors":"Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, Yuta Saito","year":"2023","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kiyohara et al., 2024] Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. Off-policy evaluation of slate ban- dit policies via optimizing abstraction. arXiv preprint arXiv:2402.02171, 2024.","title":"Off-policy evaluation of slate bandit policies via optimizing abstraction","authors":"Haruka Kiyohara, Masahiro Nomura, Yuta Saito","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Paine et al., 2020] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter se- lection for offline reinforcement learning. arXiv preprint arXiv:2007.09055, 2020.","title":"Hyperparameter selection for offline reinforcement learning","authors":"Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, Nando De Freitas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Precup et al., 2000] Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility Traces for Off-Policy Policy Evaluation. In Proceedings of the 17th International Con- ference on Machine Learning, pages 759-766, 2000.","title":"Eligibility Traces for Off-Policy Policy Evaluation","authors":"Doina Precup, Richard S Sutton, Satinder Singh","year":"2000","venue":"Proceedings of the 17th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Saito et al., 2021] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. Open bandit dataset and pipeline: Towards realistic and reproducible off-policy eval- uation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.","title":"Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evaluation","authors":"Yuta Saito, Shunsuke Aihara, Megumi Matsutani, Yusuke Narita","year":"2021","venue":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Saito et al., 2023] Yuta Saito, Ren Qingyang, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. In International Conference on Machine Learning, pages 29734-29759. PMLR, 2023.","title":"Off-policy evaluation for large action spaces via conjunct effect modeling","authors":"Yuta Saito, Ren Qingyang, Thorsten Joachims","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Saito et al., 2024] Yuta Saito, Jihan Yao, and Thorsten Joachims. Potec: Off-policy learning for large action spaces via two-stage policy decomposition. arXiv preprint arXiv:2402.06151, 2024.","title":"Potec: Off-policy learning for large action spaces via two-stage policy decomposition","authors":"Yuta Saito, Jihan Yao, Thorsten Joachims","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Confer- ence on International Conference on Machine Learning, pages 1015-1022, 2010.","title":"Gaussian process optimization in the bandit setting: No regret and experimental design","authors":"Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger","year":"2010","venue":"Proceedings of the 27th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Strehl et al., 2010] Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from Logged Implicit Exploration Data. In Advances in Neural Information Pro- cessing Systems, volume 23, pages 2217-2225, 2010.","title":"Learning from Logged Implicit Exploration Data","authors":"Alex Strehl, John Langford, Lihong Li, M Sham, Kakade","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Su et al., 2020] Yi Su, Maria Dimakopoulou, Akshay Krish- namurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning, volume 119, pages 9167-9176. PMLR, 2020.","title":"Doubly robust off-policy evaluation with shrinkage","authors":"Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, Miroslav Dudík","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Swaminathan and Joachims, 2015a] Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755, 2015.","title":"Batch learning from logged bandit feedback through counterfactual risk minimization","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Swaminathan and Joachims, 2015b] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems, volume 28, pages 3231-3239, 2015.","title":"The self-normalized estimator for counterfactual learning","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang and Wiens, 2021] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practi- cal considerations for healthcare settings. arXiv preprint arXiv:2107.11003, 2021.","title":"Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings","authors":"Wiens Tang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thomas et al., 2015a] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High- confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.","title":"Highconfidence off-policy evaluation","authors":"Thomas","year":"2015","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thomas et al., 2015b] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In Proceedings of the 32th International Conference on Machine Learning, volume 37, pages 2380-2388, 2015.","title":"High confidence policy improvement","authors":"Thomas","year":"2015","venue":"Proceedings of the 32th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and Adaptive Off-policy Evalua- tion in Contextual Bandits. In Proceedings of the 34th In- ternational Conference on Machine Learning, pages 3589- 3597, 2017.","title":"Optimal and Adaptive Off-policy Evaluation in Contextual Bandits","authors":"Yu-Xiang Wang, Alekh Agarwal, Miroslav Dudik","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}