{"type":"ScholarlyArticle","title":"Experimental Design under the Bradley-Terry Model","authors":[{"name":"Yuan Guo","slug":"yuan-guo","given":"Yuan","family":"Guo","affiliation":"ECE Department , Northeastern University , Boston , MA , USA."},{"name":"Peng Tian","slug":"peng-tian","given":"Peng","family":"Tian","affiliation":"ECE Department , Northeastern University , Boston , MA , USA."},{"name":"Jayashree Kalpathy-Cramer","slug":"jayashree-kalpathy-cramer","given":"Jayashree","family":"Kalpathy-Cramer","affiliation":"Department of Radiology , Massachusetts General Hospital , Charlestown , MA , USA."},{"name":"Susan Ostmo","slug":"susan-ostmo","given":"Susan","family":"Ostmo","affiliation":"Dept of Ophthalmology , Casey Eye Institute , Oregon Health & Science University , Portland , OR , USA."},{"name":"J.Peter Campbell","slug":"j-peter-campbell","given":"J.Peter","family":"Campbell","affiliation":"Dept of Ophthalmology , Casey Eye Institute , Oregon Health & Science University , Portland , OR , USA."},{"name":"Michael F.Chiang","slug":"michael-f-chiang","given":"Michael","family":"F.Chiang","affiliation":"Dept of Ophthalmology , Casey Eye Institute , Oregon Health & Science University , Portland , OR , USA."},{"name":"Deniz Erdogmus","slug":"deniz-erdogmus","given":"Deniz","family":"Erdogmus","affiliation":"ECE Department , Northeastern University , Boston , MA , USA."},{"name":"Jennifer Dy","slug":"jennifer-dy","given":"Jennifer","family":"Dy","affiliation":"ECE Department , Northeastern University , Boston , MA , USA."},{"name":"Stratis Ioannidis","slug":"stratis-ioannidis","given":"Stratis","family":"Ioannidis","affiliation":"ECE Department , Northeastern University , Boston , MA , USA."}],"abstract":"Labels generated by human experts via comparisons exhibit smaller variance compared to traditional sample labels.  Collecting comparison labels is challenging over large datasets, as the number of comparisons grows quadratically with the dataset size. We study the following experimental design problem: given a budget of expert comparisons, and a set of existing sample labels, we determine the comparison labels to collect that lead to the highest classification improvement. We study several experimental design objectives motivated by the Bradley-Terry model. The resulting optimization problems amount to maximizing submodular functions. We experimentally evaluate the performance of these methods over synthetic and real-life datasets.","doi":"10.24963/ijcai.2018/304","url":"https://ijcai.org/proceedings/2018/304","pdf":"https://ijcai.org/proceedings/2018/0304.pdf","markdown":"https://ijcai.org/proceedings/2018/304.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/304","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":304,"pages":{"first":2198,"last":2204},"keywords":["Machine Learning: Active Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Boyd and Vandenberghe, 2004] S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.","title":"Convex optimization","authors":"S Boyd, L Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bradley and Terry, 1952] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pages 324-345, 1952.","title":"Rank analysis of incomplete block designs: I. the method of paired comparisons","authors":"R A Bradley, M E Terry","year":"1952","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Busetto et al., 2013] A. G. Busetto, A. Hauser, G. Krummenacher, M. Sunnåker, S. Dimopoulos, C. S. Ong, Jö. Stelling, and J. M Buhmann. Near-optimal experimental design for model selection in systems biology. Bioinformatics, pages 2625-2632, 2013.","title":"Near-optimal experimental design for model selection in systems biology","authors":"A G Busetto, A Hauser, G Krummenacher, M Sunnåker, S Dimopoulos, C S Ong, Jö Stelling, J M Buhmann","year":"2013","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cavagnaro et al., 2010] D. R. Cavagnaro, J. I. Myung, M. A. Pitt, and J. V. Kujala. Adaptive design optimization: A mutual information-based approach to model discrimination in cognitive science. Neural computation, pages 887-905, 2010.","title":"Adaptive design optimization: A mutual information-based approach to model discrimination in cognitive science","authors":"D R Cavagnaro, J I Myung, M A Pitt, J V Kujala","year":"2010","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chaloner and Verdinelli, 1995] K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science, pages 273-304, 1995.","title":"Bayesian experimental design: A review","authors":"K Chaloner, I Verdinelli","year":"1995","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2015] L. Chen, P. Zhang, and B. Li. Fusing pointwise and pairwise labels for supporting user-adaptive image retrieval. In ICMR, pages 67-74, 2015.","title":"Fusing pointwise and pairwise labels for supporting user-adaptive image retrieval","authors":"L Chen, P Zhang, B Li","year":"2015","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cohn et al., 1996] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. JAIR, pages 129-145, 1996.","title":"Active learning with statistical models","authors":"D A Cohn, Z Ghahramani, M I Jordan","year":"1996","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Delzell et al., 2012] D. A. Delzell, R. F. Gunst, et al. Key properties of d-optimal designs for event-related functional MRI experiments with application to nonlinear models. Statistics in medicine, 2012.","title":"Key properties of d-optimal designs for event-related functional MRI experiments with application to nonlinear models","authors":"D A Delzell, R F Gunst","year":"2012","venue":"Statistics in medicine","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Drovandi and Pettitt, 2013] C. C. Drovandi and A. N. Pettitt. Bayesian experimental design for models with intractable likeli- hoods. Biometrics, pages 937-948, 2013.","title":"Bayesian experimental design for models with intractable likelihoods","authors":"C C Drovandi, A N Pettitt","year":"2013","venue":"Biometrics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Drovandi et al., 2014] C.C. Drovandi, J.M. McGree, and A.N. Pet- titt. A sequential monte carlo algorithm to incorporate model uncertainty in bayesian sequential design. J. C. Graph. Stat, 2014.","title":"Pettitt. A sequential monte carlo algorithm to incorporate model uncertainty in bayesian sequential design","authors":"C C Drovandi, J M Mcgree, A N","year":"2014","venue":"J. C. Graph. Stat","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Elo, 1978] A. E. Elo. The rating of chessplayers, past and present. Arco Pub., 1978.","title":"The rating of chessplayers, past and present","authors":"A E Elo","year":"1978","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Flaherty et al., 2006] P. Flaherty, A. Arkin, and M. I. Jordan. Ro- bust design of biological experiments. In NIPS, pages 363-370, 2006.","title":"Robust design of biological experiments","authors":"P Flaherty, A Arkin, M I Jordan","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Glickman and Jensen, 2005] Mark E Glickman and Shane T Jensen. Adaptive paired comparison design. Journal of statistical planning and inference, pages 279-293, 2005.","title":"Adaptive paired comparison design","authors":"E Mark, Shane T Glickman, Jensen","year":"2005","venue":"Journal of statistical planning and inference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Graßhoff and Schwabe, 2008] U. Graßhoff and R. Schwabe. Opti- mal design for the bradley-terry paired comparison model. Statis- tical Methods and Applications, pages 275-289, 2008.","title":"Optimal design for the bradley-terry paired comparison model","authors":"U Graßhoff, R Schwabe","year":"2008","venue":"Statistical Methods and Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Grasshoff et al., 2003] Ulrike Grasshoff, Heiko Großmann, Heinz Holling, and Rainer Schwabe. Optimal paired comparison designs for first-order interactions. Statistics, pages 373-386, 2003.","title":"Optimal paired comparison designs for first-order interactions","authors":"Ulrike Grasshoff, Heiko Großmann, Heinz Holling, Rainer Schwabe","year":"2003","venue":"Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Guillory and Bilmes, 2011] A. Guillory and J. Bilmes. Active semi- supervised learning using submodular functions. In UAI, pages 274-282, 2011.","title":"Active semisupervised learning using submodular functions","authors":"A Guillory, J Bilmes","year":"2011","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Guo et al., 2018] Y. Guo, P. Tian, J. Kalpathy-Cramer, S. Ostmo, J. P. Campbell, M. F Chiang, D. Erdogmus, Jennifer Dy, and S. Ioannidis. Experimental design under the Bradley-Terry model. Spiral tech. rep, 2018. https://bit.ly/2rqCCzi.","title":"Experimental design under the Bradley-Terry model","authors":"Y Guo, P Tian, J Kalpathy-Cramer, S Ostmo, J P Campbell, M Chiang, D Erdogmus, Jennifer Dy, S Ioannidis","year":"2018","venue":"Spiral tech. rep","doi":null,"url":"https://bit.ly/2rqCCzi","internal_url":null},{"position":18,"raw":"[Harville, 1997] D. A. Harville. Matrix algebra from a statistician's perspective, volume 1. Springer, 1997.","title":"Matrix algebra from a statistician's perspective","authors":"D A Harville","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hoi et al., 2006] S. C. Hoi, R. Jin, J. Zhu, and M. R. Lyu. Batch mode active learning and its application to medical image classifi- cation. In ICML, pages 417-424, 2006.","title":"Batch mode active learning and its application to medical image classification","authors":"S C Hoi, R Jin, J Zhu, M R Lyu","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Horel et al., 2014] T. Horel, S. Ioannidis, and S. Muthukrishnan. Budget feasible mechanisms for experimental design. In LATIN, pages 719-730, 2014.","title":"Budget feasible mechanisms for experimental design","authors":"T Horel, S Ioannidis, S Muthukrishnan","year":"2014","venue":"LATIN","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Huan and Marzouk, 2013] X. Huan and Y. M. Marzouk. Simu- lation based optimal bayesian experimental design for nonlinear systems. Journal of Computational Physics, pages 288-317, 2013.","title":"Simulation based optimal bayesian experimental design for nonlinear systems","authors":"Marzouk Huan, X Huan, Y M Marzouk","year":"2013","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jordan et al., 1999] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphi- cal models. Machine learning, pages 183-233, 1999.","title":"An introduction to variational methods for graphical models","authors":"M I Jordan, Z Ghahramani, T S Jaakkola, L K Saul","year":"1999","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kalpathy-Cramer et al., 2016] J. Kalpathy-Cramer, J. P. Campbell, D. Erdogmus, et al. Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quanti- tative image analysis. Ophthalmology, pages 2345-2351, 2016.","title":"Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis","authors":"J Kalpathy-Cramer, J P Campbell, D Erdogmus","year":"2016","venue":"Ophthalmology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kamishima et al., 2009] T. Kamishima, M. Hamasaki, and S. Akaho. A simple transfer learning method and its applica- tion to personalization in collaborative tagging. In ICDM, pages 219-228, 2009.","title":"A simple transfer learning method and its application to personalization in collaborative tagging","authors":"T Kamishima, M Hamasaki, S Akaho","year":"2009","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Krause and Guestrin, 2012] A. Krause and C. E Guestrin. Near- optimal nonmyopic value of information in graphical models. arXiv preprint arXiv:1207.1394, 2012.","title":"Nearoptimal nonmyopic value of information in graphical models","authors":"A Krause, C E Guestrin","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lewis and Gale, 1994] D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR, pages 3-12, 1994.","title":"A sequential algorithm for training text classifiers","authors":"D D Lewis, W A Gale","year":"1994","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liepe et al., 2013] J. Liepe, S. Filippi, M. Komorowski, and M. P. Stumpf. Maximizing the information content of experiments in systems biology. PLOS Comput. Biol, page e1002888, 2013.","title":"Maximizing the information content of experiments in systems biology","authors":"J Liepe, S Filippi, M Komorowski, M P Stumpf","year":"2013","venue":"PLOS Comput. Biol","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Myerson, 1981] R. B. Myerson. Optimal auction design. Mathe- matics of operations research, pages 58-73, 1981.","title":"Optimal auction design","authors":"R B Myerson","year":"1981","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Nemhauser et al., 1978] G. L. Nemhauser, L. A. Wolsey, and M. L Fisher. An analysis of approximations for maximizing submodular set functions. Mathematical Programming, pages 265-294, 1978.","title":"An analysis of approximations for maximizing submodular set functions","authors":"G L Nemhauser, L A Wolsey, M L Fisher","year":"1978","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rao, 1992] C Radhakrishna Rao. Information and the accuracy attainable in the estimation of statistical parameters. In Break- throughs in statistics, pages 235-247. Springer, 1992.","title":"Information and the accuracy attainable in the estimation of statistical parameters","authors":"C Radhakrishna Rao","year":"1992","venue":"Breakthroughs in statistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Robert, 2014] Christian Robert. Machine learning, a probabilistic perspective, 2014.","title":"Machine learning, a probabilistic perspective","authors":"Christian Robert","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ryan, 2003] K. J. Ryan. Estimating expected information gains for experimental designs with application to the random fatigue-limit model. J. Comput. Graph. Stat, pages 585-603, 2003.","title":"Estimating expected information gains for experimental designs with application to the random fatigue-limit model","authors":"K J Ryan","year":"2003","venue":"J. Comput. Graph. Stat","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sculley, 2010] D. Sculley. Combined regression and ranking. In SIGKDD, pages 979-988, 2010.","title":"Combined regression and ranking","authors":"D Sculley","year":"2010","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sharma and Bilgic, 2017] M. Sharma and M. Bilgic. Evidence- based uncertainty sampling for active learning. DMKDFD, pages 164-202, 2017.","title":"Evidencebased uncertainty sampling for active learning","authors":"M Sharma, M Bilgic","year":"2017","venue":"DMKDFD","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Takahama et al., 2016] R. Takahama, T. Kamishima, and H. Kashima. Progressive comparison for ranking estimation. In IJCAI, pages 3882-3888, 2016.","title":"Progressive comparison for ranking estimation","authors":"R Takahama, T Kamishima, H Kashima","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/546"},{"position":36,"raw":"[Tsilifis et al., 2017] P. Tsilifis, R. G. Ghanem, and P. Hajali. Ef- ficient bayesian experimentation using an expected information gain lower bound. SIAM/ASA JUQ, pages 30-62, 2017.","title":"Efficient bayesian experimentation using an expected information gain lower bound","authors":"P Tsilifis, R G Ghanem, P Hajali","year":"2017","venue":"SIAM/ASA JUQ","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2016] Y. Wang, S. Wang, J. Tang, H. Liu, and B. Li. PPP: Joint pointwise and pairwise image label prediction. In CVPR, pages 6005-6013, 2016.","title":"PPP: Joint pointwise and pairwise image label prediction","authors":"Y Wang, S Wang, J Tang, H Liu, B Li","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wei et al., 2015] Kai Wei, R. Iyer, and J. Bilmes. Submodularity in data subset selection and active learning. In ICML, pages 1954-1963, 2015.","title":"Submodularity in data subset selection and active learning","authors":"Kai Wei, R Iyer, J Bilmes","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang and Oles, 2000] T. Zhang and F. Oles. The value of unla- beled data for classification problems. In ICML, pages 1191-1198, 2000.","title":"The value of unlabeled data for classification problems","authors":"T Zhang, F Oles","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}