Active Evaluation of Ranking Functions Based on Graded Relevance (Extended Abstract)

Sawade, Christoph; Bickel, Steffen; von Oertzen, Timo; Scheffer, Tobias; Landwehr, Niels

Active Evaluation of Ranking Functions Based on Graded Relevance (Extended Abstract)

Christoph Sawade, Steffen Bickel, Timo von Oertzen, Tobias Scheffer, Niels Landwehr

Download BibTeX Cite

Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. Pages 3072-3075.

Abstract

Evaluating the quality of ranking functions is a core task in web search and other information retrieval domains. Because query distributions and item relevance change over time, ranking models often cannot be evaluated accurately on held-out training data. Instead, considerable effort is spent on manually labeling the relevance of query results for test queries in order to track ranking performance. We address the problem of estimating ranking performance as accurately as possible on a fixed labeling budget. Estimates are based on a set of most informative test queries selected by an active sampling distribution. Query labeling costs depend on the number of result items and item-specific attributes such as document length. We derive cost-optimal sampling distributions for commonly used ranking performance measures. Experiments on web search engine data illustrate significant reductions in labeling costs.

Christoph Sawade, Steffen Bickel, Timo von Oertzen, Tobias Scheffer, and Niels Landwehr. 2013. Active Evaluation of Ranking Functions Based on Graded Relevance (Extended Abstract). In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI-13. IJCAI, 3072–3075. https://ijcai.org/proceedings/2013/459

@inproceedings{ijcai2013p459,
  title     = {Active Evaluation of Ranking Functions Based on Graded Relevance (Extended Abstract)},
  author    = {Sawade, Christoph and Bickel, Steffen and von Oertzen, Timo and Scheffer, Tobias and Landwehr, Niels},
  booktitle = {Proceedings of the Twenty-Third International Joint Conference on
               Artificial Intelligence, {IJCAI-13}},
  publisher = {AAAI Press / International Joint Conferences on Artificial Intelligence},
  editor    = {Francesca Rossi},
  pages     = {3072--3075},
  year      = {2013},
  month     = {8},
  url       = {https://ijcai.org/proceedings/2013/459},
}

TY  - CPAPER
TI  - Active Evaluation of Ranking Functions Based on Graded Relevance (Extended Abstract)
AU  - Sawade, Christoph
AU  - Bickel, Steffen
AU  - von Oertzen, Timo
AU  - Scheffer, Tobias
AU  - Landwehr, Niels
T2  - Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI-13
PB  - AAAI Press / International Joint Conferences on Artificial Intelligence
PY  - 2013
SP  - 3072
EP  - 3075
UR  - https://ijcai.org/proceedings/2013/459
ER  -