DACBench: A Benchmark Library for Dynamic Algorithm Configuration

Eimer, Theresa; Biedenkapp, André; Reimer, Maximilian; Adriansen, Steven; Hutter, Frank; Lindauer, Marius

doi:10.24963/ijcai.2021/230

DACBench: A Benchmark Library for Dynamic Algorithm Configuration

Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriansen, Frank Hutter, Marius Lindauer

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 1668-1674.

https://doi.org/10.24963/ijcai.2021/230

Keywords

Heuristic Search and Game Playing: Meta-Reasoning and Meta-Heuristics
Heuristic Search and Game Playing: Heuristic Search and Machine Learning
Heuristic Search and Game Playing: Evaluation and Analysis

Abstract

Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.

Several theoretical and empirical results have demonstrated the benefits of dynamically controlling hyperparameters in domains like evolutionary computation, AI Planning or deep learning.

Replicating these results, as well as studying new methods for DAC, however, is difficult since existing benchmarks are often specialized and incompatible with the same interfaces.

To facilitate benchmarking and thus research on DAC, we propose DACBench, a benchmark library that seeks to collect and standardize existing DAC benchmarks from different AI domains, as well as provide a template for new ones.

For the design of DACBench, we focused on important desiderata, such as (i) flexibility, (ii) reproducibility, (iii) extensibility and (iv) automatic documentation and visualization.

To show the potential, broad applicability and challenges of DAC, we explore how a set of six initial benchmarks compare in several dimensions of difficulty.

Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriansen, Frank Hutter, and Marius Lindauer. 2021. DACBench: A Benchmark Library for Dynamic Algorithm Configuration. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 1668–1674. https://doi.org/10.24963/ijcai.2021/230

@inproceedings{ijcai2021p230,
  title     = {DACBench: A Benchmark Library for Dynamic Algorithm Configuration},
  author    = {Eimer, Theresa and Biedenkapp, André and Reimer, Maximilian and Adriansen, Steven and Hutter, Frank and Lindauer, Marius},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {1668--1674},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/230},
  url       = {https://doi.org/10.24963/ijcai.2021/230},
}

TY  - CPAPER
TI  - DACBench: A Benchmark Library for Dynamic Algorithm Configuration
AU  - Eimer, Theresa
AU  - Biedenkapp, André
AU  - Reimer, Maximilian
AU  - Adriansen, Steven
AU  - Hutter, Frank
AU  - Lindauer, Marius
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 1668
EP  - 1674
DO  - 10.24963/ijcai.2021/230
UR  - https://doi.org/10.24963/ijcai.2021/230
ER  -