{"type":"ScholarlyArticle","title":"The Information Retrieval Experiment Platform (Extended Abstract)","authors":[{"name":"Maik Fröbe","slug":"maik-frobe","given":"Maik","family":"Fröbe","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Jan Heinrich Reimer","slug":"jan-heinrich-reimer","given":"Jan Heinrich","family":"Reimer","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Sean MacAvaney","slug":"sean-macavaney","given":"Sean","family":"MacAvaney","affiliation":"University of Glasgow"},{"name":"Niklas Deckers","slug":"niklas-deckers","given":"Niklas","family":"Deckers","affiliation":"Leipzig University; ScaDS.AI"},{"name":"Simon Reich","slug":"simon-reich","given":"Simon","family":"Reich","affiliation":"Leipzig University"},{"name":"Janek Bevendorff","slug":"janek-bevendorff","given":"Janek","family":"Bevendorff","affiliation":"Bauhaus-Universität Weimar"},{"name":"Benno Stein","slug":"benno-stein","given":"Benno","family":"Stein","affiliation":"Bauhaus-Universität Weimar"},{"name":"Matthias Hagen","slug":"matthias-hagen","given":"Matthias","family":"Hagen","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Martin Potthast","slug":"martin-potthast","given":"Martin","family":"Potthast","affiliation":"ScaDS.AI; University of Kassel; hessian . AI"}],"abstract":"We have built TIREx, the information retrieval experiment platform, to promote standardized, reproducible, scalable, and blinded retrieval experiments. Standardization is achieved through integration with PyTerrier's interfaces and compatibility with ir_datasets and ir_measures. Reproducibility and scalability are based on the underlying TIRA framework, which runs dockerized software in a cloud-native execution environment. Using Docker images of 50 standard retrieval approaches, we evaluated all of them on 32 tasks (i.e., 1,600 runs) in less than a week on a midsize cluster (1,620 CPU cores and 24 GPUs), demonstrating multi-task scalability. Importantly, TIRA also enables blind evaluation of AI experiments, as the test data can be hidden from public access and the tested approaches run in a sandbox that prevents data leaks. Keeping the test data hidden from public access ensures that it cannot be used by third parties for LLM training, preventing future training-test leaks.","doi":"10.24963/ijcai.2024/931","url":"https://ijcai.org/proceedings/2024/931","pdf":"https://ijcai.org/proceedings/2024/0931.pdf","markdown":"https://ijcai.org/proceedings/2024/931.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/931","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":931,"pages":{"first":8405,"last":8410},"keywords":["Natural Language Processing: NLP: Information retrieval and text mining","Natural Language Processing: NLP: Tools","Natural Language Processing: NLP: Resources and evaluation"],"videos":[],"references":[{"position":1,"raw":"[Arguello et al., 2015] J. Arguello, F. Diaz, J. Lin, and A. Trotman. SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). In SIGIR 2015, pages 1147-1148, 2015.","title":"SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR)","authors":"J Arguello, F Diaz, J Lin, A Trotman","year":"2015","venue":"SIGIR 2015","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Armstrong et al., 2009a] T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. EvaluatIR: An online tool for evaluating and comparing IR systems. In SIGIR 2009, page 833, 2009.","title":"EvaluatIR: An online tool for evaluating and comparing IR systems","authors":"Armstrong","year":"2009","venue":"SIGIR 2009","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Armstrong et al., 2009b] T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. Improvements that don't add up: Ad-hoc retrieval results since 1998. In CIKM 2009, pages 601-610, 2009.","title":"Improvements that don't add up: Ad-hoc retrieval results since 1998","authors":"Armstrong","year":"2009","venue":"CIKM 2009","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bevendorff et al., 2018] J. Bevendorff, B. Stein, M. Hagen, and M. Potthast. Elastic ChatNoir: Search engine for the ClueWeb and the Common Crawl. In ECIR 2018, 2018.","title":"Elastic ChatNoir: Search engine for the ClueWeb and the Common Crawl","authors":"J Bevendorff, B Stein, M Hagen, M Potthast","year":"2018","venue":"ECIR 2018","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bondarenko et al., 2020] A. Bondarenko, M. Fröbe, M. Be- loucif, L. Gienapp, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, and M. Hagen. Overview of Touché 2020: Argument retrieval. In CLEF 2020, pages 384-395, 2020.","title":"Overview of Touché 2020: Argument retrieval","authors":"A Bondarenko, M Fröbe, M Beloucif, L Gienapp, Y Ajjour, A Panchenko, C Biemann, B Stein, H Wachsmuth, M Potthast, M Hagen","year":"2020","venue":"CLEF 2020","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bondarenko et al., 2021] A. Bondarenko, L. Gienapp, M. Fröbe, M. Beloucif, Y. Ajjour, A. Panchenko, C. Bie- mann, B. Stein, H. Wachsmuth, M. Potthast, and M. Hagen. Overview of Touché 2021: Argument retrieval. In CLEF 2021, pages 450-467, 2021.","title":"Overview of Touché 2021: Argument retrieval","authors":"A Bondarenko, L Gienapp, M Fröbe, M Beloucif, Y Ajjour, A Panchenko, C Biemann, B Stein, H Wachsmuth, M Potthast, M Hagen","year":"2021","venue":"CLEF 2021","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bondarenko et al., 2022] A. Bondarenko, M. Fröbe, J. Kiesel, S. Syed, T. Gurcke, M. Beloucif, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, and M. Hagen. Overview of Touché 2022: Argument retrieval. In CLEF 2022, 2022.","title":"Overview of Touché 2022: Argument retrieval","authors":"A Bondarenko, M Fröbe, J Kiesel, S Syed, T Gurcke, M Beloucif, A Panchenko, C Biemann, B Stein, H Wachsmuth, M Potthast, M Hagen","year":"2022","venue":"CLEF 2022","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Boytsov and Nyberg, 2020] L. Boytsov and E. Nyberg. Flex- ible retrieval with NMSLIB and FlexNeuART. In NLP-OSS 2020, pages 32-43, 2020.","title":"Flexible retrieval with NMSLIB and FlexNeuART","authors":"Boytsov, ] L Nyberg, E Boytsov, Nyberg","year":"2020","venue":"NLP-OSS 2020","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Breuer et al., 2019] T. Breuer, P. Schaer, N. Tavakolpour- saleh, J. Schaible, B. Wolff, and B. Müller. STELLA: To- wards a framework for the reproducibility of online search experiments. In OSIRRC at SIGIR 2019, pages 8-11, 2019.","title":"STELLA: Towards a framework for the reproducibility of online search experiments","authors":"T Breuer, P Schaer, N Tavakolpoursaleh, J Schaible, B Wolff, B Müller","year":"2019","venue":"OSIRRC at SIGIR 2019","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Breuer et al., 2020] T. Breuer, N. Ferro, N. Fuhr, M. Maistro, T. Sakai, P. Schaer, and I. Soboroff. How to measure the re- producibility of system-oriented IR experiments. In SIGIR 2020, pages 349-358, 2020.","title":"How to measure the reproducibility of system-oriented IR experiments","authors":"T Breuer, N Ferro, N Fuhr, M Maistro, T Sakai, P Schaer, I Soboroff","year":"2020","venue":"SIGIR 2020","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Breuer et al., 2021] T. Breuer, N. Ferro, M. Maistro, and P. Schaer. repro_eval: A python interface to reproducibility measures of system-oriented IR experiments. In ECIR 2021, pages 481-486, 2021.","title":"repro_eval: A python interface to reproducibility measures of system-oriented IR experiments","authors":"T Breuer, N Ferro, M Maistro, P Schaer","year":"2021","venue":"ECIR 2021","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Breuer et al., 2022] T. Breuer, J. Keller, and P. Schaer. ir_metadata: An extensible metadata schema for IR ex- periments. In SIGIR 2022, pages 3078-3089, 2022.","title":"ir_metadata: An extensible metadata schema for IR experiments","authors":"T Breuer, J Keller, P Schaer","year":"2022","venue":"SIGIR 2022","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chung et al., 2022] H. W. Chung, L. Hou, S. Longpre, et al. Scaling instruction-finetuned language models. arXiv:2210.11416, 2022.","title":"Scaling instruction-finetuned language models","authors":"H W Chung, L Hou, S Longpre","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Clancy et al., 2019] R. Clancy, N. Ferro, C. Hauff, J. Lin, T. Sakai, and Z. Z. Wu. Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). In OSIRRC at SIGIR 2019, pages 1-7, 2019.","title":"Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)","authors":"R Clancy, N Ferro, C Hauff, J Lin, T Sakai, Z Z Wu","year":"2019","venue":"OSIRRC at SIGIR 2019","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Cleverdon, 1967] C. Cleverdon. The Cranfield tests on index language devices. In ASLIB Proceedings, pages 173-192, 1967.","title":"The Cranfield tests on index language devices","authors":"C Cleverdon","year":"1967","venue":"ASLIB Proceedings","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Costello et al., 2022] C. Costello, E. Yang, D. Lawrie, and J. Mayfield. Patapsco: A python framework for cross- language information retrieval experiments. In ECIR 2022, 2022.","title":"Patapsco: A python framework for crosslanguage information retrieval experiments","authors":"C Costello, E Yang, D Lawrie, J Mayfield","year":"2022","venue":"ECIR 2022","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ferro et al., 2016] N. Ferro, N. Fuhr, K. Järvelin, N. Kando, M. Lippold, and J. Zobel. Increasing reproducibility in IR: Findings from the Dagstuhl seminar on \"Reproducibility of data-oriented experiments in e-science\". SIGIR Forum, 50(1):68-82, 2016.","title":"Increasing reproducibility in IR: Findings from the Dagstuhl seminar on \"Reproducibility of data-oriented experiments in e-science","authors":"N Ferro, N Fuhr, K Järvelin, N Kando, M Lippold, J Zobel","year":"2016","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ferro et al., 2018] N. Ferro, M. Maistro, T. Sakai, and I. Soboroff. Overview of Centre@CLEF 2018: A first tale in the systematic reproducibility realm. In CLEF 2018, 2018.","title":"Overview of Centre@CLEF 2018: A first tale in the systematic reproducibility realm","authors":"N Ferro, M Maistro, T Sakai, I Soboroff","year":"2018","venue":"CLEF 2018","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ferro et al., 2019] N. Ferro, N. Fuhr, M. Maistro, T. Sakai, and I. Soboroff. Overview of Centre@CLEF 2019: Sequel in the systematic reproducibility realm. In CLEF 2019, pages 287-300, 2019.","title":"Overview of Centre@CLEF 2019: Sequel in the systematic reproducibility realm","authors":"N Ferro, N Fuhr, M Maistro, T Sakai, I Soboroff","year":"2019","venue":"CLEF 2019","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Fröbe et al., 2023a] M. Fröbe, T. Gollub, B. Stein, M. Ha- gen, and M. Potthast. SemEval-2023 Task 5: Clickbait spoiling. In SemEval-2023, pages 2278-2289, 2023.","title":"SemEval-2023 Task 5: Clickbait spoiling","authors":"Fröbe","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Fröbe et al., 2023b] M. Fröbe, J. H. Reimer, S. MacAvaney, N. Deckers, S. Reich, J. Bevendorff, B. Stein, M. Hagen, and M. Potthast. The information retrieval experiment platform. In SIGIR 2023, pages 2826-2836, 2023.","title":"The information retrieval experiment platform","authors":"Fröbe","year":"2023","venue":"SIGIR 2023","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Fröbe et al., 2023c] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, and M. Potthast. Continuous integration for reproducible shared tasks with TIRA.io. In ECIR 2023, 2023.","title":"Continuous integration for reproducible shared tasks with TIRA","authors":"Fröbe","year":"2023","venue":"ECIR 2023","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Fuhr, 2017] N. Fuhr. Some common mistakes in IR evalua- tion, and how they can be avoided. SIGIR Forum, 51(3):32- 41, 2017.","title":"Some common mistakes in IR evaluation, and how they can be avoided","authors":"N Fuhr","year":"2017","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Fuhr, 2020] N. Fuhr. Proof by experimentation?: Towards better IR research. SIGIR Forum, 54(2):2:1-2:4, 2020.","title":"Proof by experimentation?: Towards better IR research","authors":"N Fuhr","year":"2020","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Gollub et al., 2012] T. Gollub, B. Stein, S. Burrows, and D. Hoppe. TIRA: Configuring, executing, and dissemi- nating information retrieval experiments. In TIR 2012 at DEXA, pages 151-155, 2012.","title":"TIRA: Configuring, executing, and disseminating information retrieval experiments","authors":"T Gollub, B Stein, S Burrows, D Hoppe","year":"2012","venue":"TIR 2012 at DEXA","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hopfgartner et al., 2015] F. Hopfgartner, T. Brodt, J. Seiler, et al. Benchmarking news recommendations: The CLEF NewsREEL use case. SIGIR Forum, 49(2):129-136, 2015.","title":"Benchmarking news recommendations: The CLEF NewsREEL use case","authors":"F Hopfgartner, T Brodt, J Seiler","year":"2015","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Hopfgartner et al., 2018] F. Hopfgartner, A. Hanbury, H. Müller, et al. Evaluation-as-a-service for the computa- tional sciences: Overview and outlook. Journal of Data and Information Quality, 10(4):15:1-15:32, 2018.","title":"Evaluation-as-a-service for the computational sciences: Overview and outlook","authors":"F Hopfgartner, A Hanbury, H Müller","year":"2018","venue":"Journal of Data and Information Quality","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Jagerman et al., 2018] R. Jagerman, K. Balog, and M. de Ri- jke. OpenSearch: Lessons learned from an online evalua- tion campaign. Journal of Data and Information Quality, 10(3):13:1-13:15, 2018.","title":"OpenSearch: Lessons learned from an online evaluation campaign","authors":"R Jagerman, K Balog, M De Rijke","year":"2018","venue":"Journal of Data and Information Quality","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Jose et al., 2021] K. M. Jose, T. Nguyen, S. MacAvaney, J. Dalton, and A. Yates. DiffIR: Exploring differences in ranking models' behavior. In SIGIR 2021, pages 2595- 2599, 2021.","title":"DiffIR: Exploring differences in ranking models' behavior","authors":"K M Jose, T Nguyen, S Macavaney, J Dalton, A Yates","year":"2021","venue":"SIGIR 2021","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kiesel et al., 2023] J. Kiesel, M. Alshomary, N. Mirzakhme- dova, M. Heinrich, N. Handke, H. Wachsmuth, and B. Stein. SemEval-2023 Task 4: ValueEval: Identification of human values behind arguments. In SemEval 2023, pages 2290- 2306, 2023.","title":"SemEval-2023 Task 4: ValueEval: Identification of human values behind arguments","authors":"J Kiesel, M Alshomary, N Mirzakhmedova, M Heinrich, N Handke, H Wachsmuth, B Stein","year":"2023","venue":"SemEval 2023","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lin and Zhang, 2020] J. Lin and Q. Zhang. Reproducibility is a process, not an achievement: The replicability of IR reproducibility experiments. In ECIR 2020, pages 43-49, 2020.","title":"Reproducibility is a process, not an achievement: The replicability of IR reproducibility experiments","authors":"Lin, Zhang, ] J Lin, Q Zhang","year":"2020","venue":"ECIR 2020","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lin et al., 2021] J. Lin, X. Ma, S. Lin, J. Yang, R. Pradeep, and R. Nogueira. Pyserini: A python toolkit for repro- ducible information retrieval research with sparse and dense representations. In SIGIR 2021, pages 2356-2362, 2021.","title":"Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations","authors":"J Lin, X Ma, S Lin, J Yang, R Pradeep, R Nogueira","year":"2021","venue":"SIGIR 2021","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lin et al., 2022] J. Lin, D. Campos, N. Craswell, B. Mitra, and E. Yilmaz. Fostering coopetition while plugging leaks: The design and implementation of the MS MARCO leader- boards. In SIGIR 2022, pages 2939-2948, 2022.","title":"Fostering coopetition while plugging leaks: The design and implementation of the MS MARCO leaderboards","authors":"J Lin, D Campos, N Craswell, B Mitra, E Yilmaz","year":"2022","venue":"SIGIR 2022","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lin, 2018] J. Lin. The neural hype and comparisons against weak baselines. SIGIR Forum, 52(2):40-51, 2018.","title":"The neural hype and comparisons against weak baselines","authors":"J Lin","year":"2018","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[MacAvaney et al., 2020] S. MacAvaney, A. Yates, S. Feld- man, D. Downey, A. Cohan, and N. Goharian. OpenNIR: A complete neural ad-hoc ranking pipeline. In WSDM 2020, pages 845-848, 2020.","title":"OpenNIR: A complete neural ad-hoc ranking pipeline","authors":"S Macavaney, A Yates, S Feldman, D Downey, A Cohan, N Goharian","year":"2020","venue":"WSDM 2020","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[MacAvaney et al., 2021] S. MacAvaney, A. Yates, S. Feld- man, D. Downey, A. Cohan, and N. Goharian. Simplified data wrangling with ir_datasets. In SIGIR 2021, pages 2429-2436, 2021.","title":"Simplified data wrangling with ir_datasets","authors":"S Macavaney, A Yates, S Feldman, D Downey, A Cohan, N Goharian","year":"2021","venue":"SIGIR 2021","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[MacAvaney et al., 2022] S. MacAvaney, C. Macdonald, and I. Ounis. Streamlining evaluation with ir-measures. In ECIR 2022, pages 305-310, 2022.","title":"Streamlining evaluation with ir-measures","authors":"S Macavaney, C Macdonald, I Ounis","year":"2022","venue":"ECIR 2022","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Macdonald et al., 2021] C. Macdonald, N. Tonellotto, S. MacAvaney, and I. Ounis. Pyterrier: Declarative experimentation in python from BM25 to dense retrieval. In CIKM 2021, pages 4526-4533, 2021.","title":"Pyterrier: Declarative experimentation in python from BM25 to dense retrieval","authors":"C Macdonald, N Tonellotto, S Macavaney, I Ounis","year":"2021","venue":"CIKM 2021","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Mallia et al., 2019] A. Mallia, M. Siedlaczek, J. M. Macken- zie, and T. Suel. PISA: Performant indexes and search for academia. In OSIRRC at SIGIR 2019, pages 50-56, 2019.","title":"PISA: Performant indexes and search for academia","authors":"A Mallia, M Siedlaczek, J M Mackenzie, T Suel","year":"2019","venue":"OSIRRC at SIGIR 2019","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Moffat, 2022] A. Moffat. Batch evaluation metrics in infor- mation retrieval: Measures, scales, and meaning. IEEE Access, 10:105564-105577, 2022.","title":"Batch evaluation metrics in information retrieval: Measures, scales, and meaning","authors":"A Moffat","year":"2022","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Piwowarski, 2020] B. Piwowarski. Experimaestro and Data- maestro: Experiment and dataset managers (for IR). In SIGIR 2020, pages 2173-2176, 2020.","title":"Experimaestro and Datamaestro: Experiment and dataset managers (for IR)","authors":"B Piwowarski","year":"2020","venue":"SIGIR 2020","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Potthast et al., 2019] M. Potthast, T. Gollub, M. Wiegmann, and B. Stein. TIRA integrated research architecture. In Information Retrieval Evaluation in a Changing World. 2019.","title":"TIRA integrated research architecture","authors":"M Potthast, T Gollub, M Wiegmann, B Stein","year":"2019","venue":"Information Retrieval Evaluation in a Changing World","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Potthast et al., 2021] M. Potthast, S. Günther, J. Bevendorff, J. P. Bittner, A. Bondarenko, M. Fröbe, C. Kahmann, A. Niekler, M. Völske, B. Stein, and M. Hagen. The information retrieval anthology. In SIGIR 2021, pages 2550-2555, 2021.","title":"The information retrieval anthology","authors":"M Potthast, S Günther, J Bevendorff, J P Bittner, A Bondarenko, M Fröbe, C Kahmann, A Niekler, M Völske, B Stein, M Hagen","year":"2021","venue":"SIGIR 2021","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Sainz et al., 2023] O. Sainz, J. A. Campos, I. García-Ferrero, et al. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In EMNLP 2023, pages 10776-10787, 2023.","title":"NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark","authors":"O Sainz, J A Campos, I García-Ferrero","year":"2023","venue":"EMNLP 2023","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Sakai et al., 2019] T. Sakai, N. Ferro, I. Soboroff, Z. Zeng, P. Xiao, and M. Maistro. Overview of the NTCIR-14 Centre task. In NTCIR 2019, 2019.","title":"Overview of the NTCIR-14 Centre task","authors":"T Sakai, N Ferro, I Soboroff, Z Zeng, P Xiao, M Maistro","year":"2019","venue":"NTCIR 2019","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Sakai et al., 2020] T. Sakai, S. Tao, Z. Zeng, Y. Zheng, J. Mao, Z. Chu, Y. Liu, M. Maistro, Z. Dou, N. Ferro, et al. Overview of the NTCIR-15 We Want Web with Centre (WWW-3) task. NTCIR 2020, 2020.","title":"Overview of the NTCIR-15 We Want Web with Centre (WWW-3) task","authors":"T Sakai, S Tao, Z Zeng, Y Zheng, J Mao, Z Chu, Y Liu, M Maistro, Z Dou, N Ferro","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Sakai, 2020] T. Sakai. On Fuhr's guideline for IR evaluation. SIGIR Forum, 54(1):12:1-12:8, 2020.","title":"On Fuhr's guideline for IR evaluation","authors":"T Sakai","year":"2020","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Thakur et al., 2021] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks 2021, 2021.","title":"BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models","authors":"N Thakur, N Reimers, A Rücklé, A Srivastava, I Gurevych","year":"2021","venue":"NeurIPS Datasets and Benchmarks","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Tsatsaronis et al., 2015] G. Tsatsaronis, G. Balikas, P. Malakasiotis, et al. An overview of the BIOASQ large- scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16:138:1-138:28, 2015.","title":"An overview of the BIOASQ largescale biomedical semantic indexing and question answering competition","authors":"G Tsatsaronis, G Balikas, P Malakasiotis","year":"2015","venue":"BMC Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Vanschoren et al., 2013] J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo. OpenML: Networked science in machine learning. SIGKDD Explor., 15(2):49-60, 2013.","title":"OpenML: Networked science in machine learning","authors":"J Vanschoren, J N Van Rijn, B Bischl, L Torgo","year":"2013","venue":"SIGKDD Explor","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Voorhees et al., 2022] E. M. Voorhees, I. Soboroff, and J. Lin. Can old TREC collections reliably evaluate modern neural retrieval models? arXiv:2201.11086, 2022.","title":"Can old TREC collections reliably evaluate modern neural retrieval models?","authors":"E M Voorhees, I Soboroff, J Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Voorhees, 2001] E. M. Voorhees. The philosophy of infor- mation retrieval evaluation. In CLEF 2001, pages 355-370, 2001.","title":"The philosophy of information retrieval evaluation","authors":"E M Voorhees","year":"2001","venue":"CLEF 2001","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Voorhees, 2019] E. M. Voorhees. The evolution of Cranfield. In Information Retrieval Evaluation in a Changing World, pages 45-69. 2019.","title":"The evolution of Cranfield","authors":"E M Voorhees","year":"2019","venue":"Information Retrieval Evaluation in a Changing World","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Yadav et al., 2019] D. Yadav, R. Jain, H. Agrawal, et al. EvalAI: Towards better evaluation systems for AI agents. arXiv:1902.03570, 2019.","title":"EvalAI: Towards better evaluation systems for AI agents","authors":"D Yadav, R Jain, H Agrawal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Yang et al., 2017] P. Yang, H. Fang, and J. Lin. Anserini: En- abling the use of Lucene for information retrieval research. In SIGIR 2017, pages 1253-1256, 2017.","title":"Anserini: Enabling the use of Lucene for information retrieval research","authors":"P Yang, H Fang, J Lin","year":"2017","venue":"SIGIR 2017","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Yates et al., 2020] A. Yates, S. Arora, X. Zhang, W. Yang, K. M. Jose, and J. Lin. Capreolus: A toolkit for end-to-end neural ad hoc retrieval. In WSDM 2020, pages 861-864, 2020.","title":"Capreolus: A toolkit for end-to-end neural ad hoc retrieval","authors":"A Yates, S Arora, X Zhang, W Yang, K M Jose, J Lin","year":"2020","venue":"WSDM 2020","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Zhang et al., 2022] X. Zhang, N. Thakur, O. Ogundepo, et al. Making a MIRACL: Multilingual information retrieval across a continuum of languages. arXiv:2210.09984, 2022.","title":"Making a MIRACL: Multilingual information retrieval across a continuum of languages","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Zobel, 2023] J. Zobel. When measurement misleads: The limits of batch assessment of retrieval systems. In ACM SIGIR Forum, volume 56, pages 1-20, 2023.","title":"When measurement misleads: The limits of batch assessment of retrieval systems","authors":"J Zobel","year":"2023","venue":"ACM SIGIR Forum","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}