{"type":"ScholarlyArticle","title":"More for Less: Safe Policy Improvement with Stronger Performance Guarantees","authors":[{"name":"Patrick Wienhöft","slug":"patrick-wienhoft","given":"Patrick","family":"Wienhöft","affiliation":"Department of Computer Science , Technische Universität Dresden , Dresden , Germany; Centre for Tactile Internet with Human -in-the-Loop (CeTI)"},{"name":"Marnix Suilen","slug":"marnix-suilen","given":"Marnix","family":"Suilen","affiliation":"Department of Software Science , Radboud University , Nijmegen , The Netherlands"},{"name":"Thiago D. Simão","slug":"thiago-d-simao","given":"Thiago D.","family":"Simão","affiliation":"Department of Software Science , Radboud University , Nijmegen , The Netherlands"},{"name":"Clemens Dubslaff","slug":"clemens-dubslaff","given":"Clemens","family":"Dubslaff","affiliation":"Eindhoven University of Technology , Eindhoven , The Netherlands; Centre for Tactile Internet with Human -in-the-Loop (CeTI)"},{"name":"Christel Baier","slug":"christel-baier","given":"Christel","family":"Baier","affiliation":"Department of Computer Science , Technische Universität Dresden , Dresden , Germany; Centre for Tactile Internet with Human -in-the-Loop (CeTI)"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Department of Software Science , Radboud University , Nijmegen , The Netherlands"}],"abstract":"In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated.\r\nState-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance.\r\nWe present a novel approach to the SPI problem that provides the means to require less data for such guarantees. \r\nSpecifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI.\r\nOur empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.","doi":"10.24963/ijcai.2023/490","url":"https://ijcai.org/proceedings/2023/490","pdf":"https://ijcai.org/proceedings/2023/0490.pdf","markdown":"https://ijcai.org/proceedings/2023/490.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/490","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":490,"pages":{"first":4406,"last":4415},"keywords":["Machine Learning: ML: Reinforcement learning","Planning and Scheduling: PS: Learning in planning and scheduling","Planning and Scheduling: PS: Planning under uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Barrett and Narayanan, 2008] Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In ICML, pages 41-47. ACM, 2008.","title":"Learning all optimal policies with multiple criteria","authors":"Narayanan Barrett, Leon Barrett, Srini Narayanan","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brandfonbrener et al., 2022] David Brandfonbrener, Remi Tachet des Combes, and Romain Laroche. Incorporating explicit uncertainty estimates into deep offline reinforce- ment learning. arXiv preprint arXiv:2206.01085, 2022.","title":"Incorporating explicit uncertainty estimates into deep offline reinforcement learning","authors":"David Brandfonbrener, Remi Tachet Des Combes, Romain Laroche","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castellini et al., 2023] Alberto Castellini, Federico Bianchi, Edoardo Zorzi, Thiago D. Simão, Alessandro Farinelli, and Matthijs T. J. Spaan. Scalable Safe Policy Improve- ment via Monte Carlo Tree Search. In ICML, 2023.","title":"Scalable Safe Policy Improvement via Monte Carlo Tree Search","authors":"Alberto Castellini, Federico Bianchi, Edoardo Zorzi, Thiago D Simão, Alessandro Farinelli, T J Matthijs, Spaan","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chandak et al., 2020] Yash Chandak, Scott M. Jordan, Georgios Theocharous, Martha White, and Philip S. Thomas. Towards safe policy improvement for non- stationary MDPs. In NeurIPS, pages 9156-9168.","title":"Towards safe policy improvement for nonstationary MDPs","authors":"Yash Chandak, Scott M Jordan, Georgios Theocharous, Martha White, Philip S Thomas","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cubuktepe et al., 2021] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Ahmadreza Marandi, Marnix Suilen, and Ufuk Topcu. Robust finite-state controllers for uncertain POMDPs. In AAAI, pages 11792-11800. AAAI Press, 2021.","title":"Robust finite-state controllers for uncertain POMDPs","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Ahmadreza Junges, Marnix Marandi, Ufuk Suilen, Topcu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hans and Udluft, 2009] Alexander Hans and Steffen Ud- luft. Efficient uncertainty propagation for reinforcement learning with limited data. In ICANN (1), pages 70-79. Springer, 2009.","title":"Efficient uncertainty propagation for reinforcement learning with limited data","authors":"Alexander Hans, Steffen Udluft","year":"2009","venue":"ICANN (1)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaynes, 2003] E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge University Press, 2003.","title":"Probability Theory: The Logic of Science","authors":"E T Jaynes","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Junges et al., 2018] Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, and Bernd Becker. Finite-state controllers of POMDPs using parameter synthesis. In UAI, pages 519- 529. AUAI Press, 2018.","title":"Finite-state controllers of POMDPs using parameter synthesis","authors":"Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, Bernd Becker","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Kaelbling","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kidambi et al., 2020] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. In NeurIPS, pages 21810-21823.","title":"MOReL: Model-based offline reinforcement learning","authors":"Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In NeurIPS, pages 1179- 1191.","title":"Conservative Q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lange et al., 2012] Sascha Lange, Thomas Gabel, and Mar- tin A. Riedmiller. Batch reinforcement learning. In Rein- forcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 45-73. Springer, 2012.","title":"Batch reinforcement learning","authors":"Sascha Lange, Thomas Gabel, Martin A Riedmiller","year":"2012","venue":"Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laroche et al., 2019] Romain Laroche, Paul Trichelair, and Remi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In ICML, pages 3652-3661.","title":"Safe policy improvement with baseline bootstrapping","authors":"Romain Laroche, Paul Trichelair, Remi Tachet Des Combes","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"PMLR, 2019.","title":null,"authors":"Pmlr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mayr and Munday, 2023] Richard Mayr and Eric Munday. Strategy Complexity of Point Payoff, Mean Payoff and To- tal Payoff Objectives in Countable MDPs. Logical Meth- ods in Computer Science, Volume 19, Issue 1, March 2023.","title":"Strategy Complexity of Point Payoff, Mean Payoff and Total Payoff Objectives in Countable MDPs","authors":"Munday Mayr, Richard Mayr, Eric Munday","year":"2023","venue":"Logical Methods in Computer Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nat","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nadjahi et al., 2019] Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In ECML/PKDD (3), pages 53-68. Springer, 2019.","title":"Safe policy improvement with soft baseline bootstrapping","authors":"Kimia Nadjahi, Romain Laroche, Rémi Tachet Des Combes","year":"2019","venue":"ECML/PKDD (3)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Petrik et al., 2016] Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe policy im- provement by minimizing robust baseline regret. In NIPS, pages 2298-2306.","title":"Safe policy improvement by minimizing robust baseline regret","authors":"Marek Petrik, Mohammad Ghavamzadeh, Yinlam Chow","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":"Wiley Series in Probability and Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ramakrishnan et al., 2020] Ramya Ramakrishnan, Ece Ka- mar, Debadeepta Dey, Eric Horvitz, and Julie Shah. Blind spot detection for safe sim-to-real transfer. J. Artif. Intell. Res., 67:191-234, 2020.","title":"Blind spot detection for safe sim-to-real transfer","authors":"Ramya Ramakrishnan, Ece Kamar, Debadeepta Dey, Eric Horvitz, Julie Shah","year":"2020","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rigter et al., 2022] Marc Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based of- fline reinforcement learning. In NeurIPS, pages 16082- 16097.","title":"RAMBO-RL: Robust adversarial model-based offline reinforcement learning","authors":"Marc Rigter, Bruno Lacerda, Nick Hawes","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Curran Associates, Inc., 2022.","title":null,"authors":"Curran Associates, Inc","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rockafellar and Uryasev, 2000] R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at- risk. Journal of risk, 2:21-42, 2000.","title":"Optimization of conditional value-atrisk","authors":"Rockafellar Tyrrell, Stanislav Uryasev","year":"2000","venue":"Journal of risk","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Satija et al., 2021] Harsh Satija, Philip S. Thomas, Joelle Pineau, and Romain Laroche. Multi-objective SPIBB: sel- donian offline policy improvement with safety constraints in finite MDPs. In NeurIPS, pages 2004-2017, 2021.","title":"Multi-objective SPIBB: seldonian offline policy improvement with safety constraints in finite MDPs","authors":"Harsh Satija, Philip S Thomas, Joelle Pineau, Romain Laroche","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Scholl et al., 2022] Philipp Scholl, Felix Dietrich, Clemens Otte, and Steffen Udluft. Safe policy improvement ap- proaches on discrete Markov decision processes. In ICAART (2), pages 142-151. SCITEPRESS, 2022.","title":"Safe policy improvement approaches on discrete Markov decision processes","authors":"Philipp Scholl, Felix Dietrich, Clemens Otte, Steffen Udluft","year":"2022","venue":"ICAART (2)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algo- rithm that masters chess, shogi, and Go through self-play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, Demis Hassabis","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Simão and Spaan, 2019a] Thiago D. Simão and Matthijs T. J. Spaan. Safe policy improvement with baseline boot- strapping in factored environments. In AAAI, pages 4967- 4974. AAAI Press, 2019.","title":"Safe policy improvement with baseline bootstrapping in factored environments","authors":"Spaan Simão, D Thiago, Simão, T J Matthijs, Spaan","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Simão and Spaan, 2019b] Thiago D. Simão and Matthijs T. J. Spaan. Structure learning for safe policy improve- ment. In IJCAI, pages 3453-3459. ijcai.org, 2019.","title":"Structure learning for safe policy improvement","authors":"D Simão And Spaan ; B] Thiago, Simão, T J Matthijs, Spaan","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/479"},{"position":35,"raw":"[Simão et al., 2020] Thiago D. Simão, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with an estimated baseline policy. In AAMAS, pages 1269- 1277. IFAAMAS, 2020.","title":"Safe policy improvement with an estimated baseline policy","authors":"D Thiago, Romain Simão, Rémi Laroche, Tachet Des Combes","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Simão et al., 2023] Thiago D. Simão, Marnix Suilen, and Nils Jansen. Safe policy improvement for POMDPs via finite-state controllers. In AAAI. AAAI Press, 2023.","title":"Safe policy improvement for POMDPs via finite-state controllers","authors":"D Thiago, Marnix Simão, Nils Suilen, Jansen","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Smit et al., 2021] Jordi Smit, Canmanie Ponnambalam, Matthijs T.J. Spaan, and Frans A. Oliehoek. PEBL: Pes- simistic ensembles for offline deep reinforcement learning. In IJCAI Workshop on Robust and Reliable Autonomy in the Wild (R2AW), 2021.","title":"PEBL: Pessimistic ensembles for offline deep reinforcement learning","authors":"Jordi Smit, Canmanie Ponnambalam, T J Matthijs, Frans A Spaan, Oliehoek","year":"2021","venue":"IJCAI Workshop on Robust and Reliable Autonomy in the Wild (R2AW)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Spaan, 2012] Matthijs T. J. Spaan. Partially observable Markov decision processes. In Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 387-414. Springer, 2012.","title":"Partially observable Markov decision processes","authors":"T J Matthijs, Spaan","year":"2012","venue":"Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning -an introduction. Adaptive computation and machine learning. MIT Press, 1998.","title":"Reinforcement learning -an introduction. Adaptive computation and machine learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Temme, 1992] N.M. Temme. Asymptotic inversion of the incomplete beta function. Journal of Computational and Applied Mathematics, 41(1):145-157, 1992.","title":"Asymptotic inversion of the incomplete beta function","authors":"N M Temme","year":"1992","venue":"Journal of Computational and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Thomas et al., 2015] Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In ICML, pages 2380-2388. PMLR, 2015.","title":"High confidence policy improvement","authors":"Philip S Thomas, Georgios Theocharous, Mohammad Ghavamzadeh","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Weissman et al., 2003] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdú, and Marcelo J. Weinberger. Inequalities for the L1 deviation of the empirical distribu- tion. Hewlett-Packard Labs, Tech. Rep, 2003.","title":"Inequalities for the L1 deviation of the empirical distribution","authors":"Weissman","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wienhöft et al., 2023] Patrick Wienhöft, Marnix Suilen, Thiago D. Simão, Clemens Dubslaff, Christel Baier, and Nils Jansen. More for less: Safe policy improvement with stronger performance guarantees. arXiv preprint arXiv:2305.07958, 2023.","title":"More for less: Safe policy improvement with stronger performance guarantees","authors":"Wienhöft","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy opti- mization. In NeuRIPS, pages 14129-14142.","title":"MOPO: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"NeuRIPS","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yu et al., 2021] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy opti- mization. In NeurIPS, pages 28954-28967.","title":"COMBO: conservative offline model-based policy optimization","authors":"Yu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhao et al., 2020] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep rein- forcement learning for robotics: a survey. In SSCI, pages 737-744. IEEE, 2020.","title":"Sim-to-real transfer in deep reinforcement learning for robotics: a survey","authors":"Zhao","year":"2020","venue":"SSCI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}