{"type":"ScholarlyArticle","title":"Minimizing Time-to-Rank: A Learning and Recommendation Approach","authors":[{"name":"Haoming Li","slug":"haoming-li","given":"Haoming","family":"Li","affiliation":"Duke University"},{"name":"Sujoy Sikdar","slug":"sujoy-sikdar","given":"Sujoy","family":"Sikdar","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Rohit Vaish","slug":"rohit-vaish","given":"Rohit","family":"Vaish","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Junming Wang","slug":"junming-wang","given":"Junming","family":"Wang","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Lirong Xia","slug":"lirong-xia","given":"Lirong","family":"Xia","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Chaonan Ye","slug":"chaonan-ye","given":"Chaonan","family":"Ye","affiliation":"Stanford University"}],"abstract":"Consider the following problem faced by an online voting platform: A user is provided with a list of alternatives, and is asked to rank them in order of preference using only drag-and-drop operations. The platform's goal is to recommend an initial ranking that minimizes the time spent by the user in arriving at her desired ranking. We develop the first optimization framework to address this problem, and make theoretical as well as practical contributions. On the practical side, our experiments on the Amazon Mechanical Turk platform provide two interesting insights about user behavior: First, that users' ranking strategies closely resemble selection or insertion sort, and second, that the time taken for a drag-and-drop operation depends linearly on the number of positions moved. These insights directly motivate our theoretical model of the optimization problem. We show that computing an optimal recommendation is NP-hard, and provide exact and approximation algorithms for a variety of special cases of the problem. Experimental evaluation on MTurk shows that, compared to a random recommendation strategy, the proposed approach reduces the (average) time-to-rank by up to 50%.","doi":"10.24963/ijcai.2019/195","url":"https://ijcai.org/proceedings/2019/195","pdf":"https://ijcai.org/proceedings/2019/0195.pdf","markdown":"https://ijcai.org/proceedings/2019/195.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/195","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":195,"pages":{"first":1408,"last":1414},"keywords":["Humans and AI: Personalization and User Modeling","Multidisciplinary Topics and Applications: Recommender Systems"],"videos":[],"references":[{"position":1,"raw":"[Agarwal, 2016] Shivani Agarwal. On Ranking and Choice Models. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 4050- 4053, 2016.","title":"On Ranking and Choice Models","authors":"Shivani Agarwal","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/598"},{"position":2,"raw":"[Albert and Tullis, 2013] William Albert and Thomas Tullis. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Newnes, 2013.","title":"Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics","authors":"Albert, Tullis, William Albert, Thomas Tullis","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Awasthi et al., 2014] Pranjal Awasthi, Avrim Blum, Or Shef- fet, and Aravindan Vijayaraghavan. Learning Mixtures of Ranking Models. In Proceedings of Advances in Neural Information Processing Systems, pages 2609-2617, 2014.","title":"Learning Mixtures of Ranking Models","authors":"Pranjal Awasthi, Avrim Blum, Or Sheffet, Aravindan Vijayaraghavan","year":"2014","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bevan et al., 2015] Nigel Bevan, James Carter, and Susan Harker. ISO 9241-11 revised: What have we Learnt about Usability Since 1998? In International Conference on Human-Computer Interaction, pages 143-151, 2015.","title":"ISO 9241-11 revised: What have we Learnt about Usability Since 1998","authors":"Nigel Bevan, James Carter, Susan Harker","year":"2015","venue":"International Conference on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blum et al., 2004] Avrim Blum, Jeffrey Jackson, Tuomas Sandholm, and Martin Zinkevich. Preference Elicitation and Query Learning. Journal of Machine Learning Re- search, 5:649-667, 2004.","title":"Preference Elicitation and Query Learning","authors":"Avrim Blum, Jeffrey Jackson, Tuomas Sandholm, Martin Zinkevich","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boutilier, 2013] Craig Boutilier. Computational Decision Support: Regret-Based Models for Optimization and Pref- erence Elicitation. In P. H. Crowley and T. R. Zentall, edi- tors, Comparative Decision Making: Analysis and Support Across Disciplines and Applications. Oxford University Press, 2013.","title":"Computational Decision Support: Regret-Based Models for Optimization and Preference Elicitation","authors":"Craig Boutilier","year":"2013","venue":"Comparative Decision Making: Analysis and Support Across Disciplines and Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Braverman and Mossel, 2008] Mark Braverman and Elchanan Mossel. Noisy Sorting Without Resampling. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 268-276, 2008.","title":"Noisy Sorting Without Resampling","authors":"Mark Braverman, Elchanan Mossel","year":"2008","venue":"Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Busa-Fekete et al., 2014] Róbert Busa-Fekete, Eyke Hüllermeier, and Balázs Szörényi. Preference-Based Rank Elicitation using Statistical Models: The Case of Mallows. In Proceedings of the 31st International Conference on International Conference on Machine Learning, pages II:1071-1079, 2014.","title":"Preference-Based Rank Elicitation using Statistical Models: The Case of Mallows","authors":"Róbert Busa-Fekete, Eyke Hüllermeier, Balázs Szörényi","year":"2014","venue":"Proceedings of the 31st International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Conitzer and Sandholm, 2002] Vincent Conitzer and Tuo- mas Sandholm. Vote Elicitation: Complexity and Strategy- Proofness. In Eighteenth National Conference on Artificial Intelligence, pages 392-397, 2002.","title":"Vote Elicitation: Complexity and Strategy-Proofness","authors":"Sandholm Conitzer","year":"2002","venue":"Eighteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guiver and Snelson, 2009] John Guiver and Edward Snel- son. Bayesian Inference for Plackett-Luce Ranking Models. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML-09, pages 377-384, 2009.","title":"Bayesian Inference for Plackett-Luce Ranking Models","authors":"Snelson Guiver, John Guiver, Edward Snelson","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML-09","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kendall, 1938] Maurice G Kendall. A New Measure of Rank Correlation. Biometrika, 30(1/2):81-93, 1938.","title":"A New Measure of Rank Correlation","authors":"Maurice G Kendall","year":"1938","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khetan and Oh, 2016] Ashish Khetan and Sewoong Oh. Data-Driven Rank Breaking for Efficient Rank Aggrega- tion. Journal of Machine Learning Research, 17(193):1-54, 2016.","title":"Data-Driven Rank Breaking for Efficient Rank Aggregation","authors":"Ashish Khetan, Sewoong Oh","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Laskowski et al., 2004] Sharon J Laskowski, Marguerite Autry, John Cugini, and William Killam. Improving the Usability and Accessibility of Voting Systems and Products. NIST Special Publication, 500:256, 2004.","title":"Improving the Usability and Accessibility of Voting Systems and Products","authors":"Sharon J Laskowski, Marguerite Autry, John Cugini, William Killam","year":"2004","venue":"NIST Special Publication","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lerman and Hogg, 2014] Kristina Lerman and Tad Hogg. Leveraging Position Bias to Improve Peer Recommenda- tion. PloS one, 9(6):e98914, 2014.","title":"Leveraging Position Bias to Improve Peer Recommendation","authors":"Lerman, Hogg, Kristina Lerman, Tad Hogg","year":"2014","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019] Haoming Li, Sujoy Sikdar, Rohit Vaish, Jun- ming Wang, Lirong Xia, and Chaonan Ye. Minimiz- ing Time-to-Rank: A Learning and Recommendation Ap- proach. arXiv preprint arXiv:1905.11984, 2019.","title":"Minimizing Time-to-Rank: A Learning and Recommendation Approach","authors":"Haoming Li, Sujoy Sikdar, Rohit Vaish, Junming Wang, Lirong Xia, Chaonan Ye","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu, 2011] Tie-Yan Liu. Learning to Rank for Information Retrieval. Springer, 2011.","title":"Learning to Rank for Information Retrieval","authors":"Tie-Yan Liu, Liu","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu and Boutilier, 2014] Tyler Lu and Craig Boutilier. Ef- fective Sampling and Learning for Mallows Models with Pairwise-Preference Data. Journal of Machine Learning Research, 15:3963-4009, 2014.","title":"Effective Sampling and Learning for Mallows Models with Pairwise-Preference Data","authors":"Lu, Boutilier, Tyler Lu, Craig Boutilier","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luce, 1959] Robert Duncan Luce. Individual Choice Behav- ior: A Theoretical Analysis. Wiley, 1959.","title":"Individual Choice Behavior: A Theoretical Analysis","authors":"Robert Duncan, Luce","year":"1959","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mallows, 1957] Colin L. Mallows. Non-Null Ranking Model. Biometrika, 44(1/2):114-130, 1957.","title":"Non-Null Ranking Model","authors":"Colin L Mallows","year":"1957","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Negahban et al., 2017] Sahand Negahban, Sewoong Oh, and Devavrat Shah. Rank Centrality: Ranking from Pairwise Comparisons. Operations Research, 65(1):266-287, 2017.","title":"Devavrat Shah. Rank Centrality: Ranking from Pairwise Comparisons","authors":"Negahban","year":"2017","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Plackett, 1975] Robin L. Plackett. The Analysis of Permu- tations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193-202, 1975.","title":"The Analysis of Permutations","authors":"Robin L Plackett","year":"1975","venue":"Journal of the Royal Statistical Society. Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Procaccia and Shah, 2016] Ariel D Procaccia and Nisarg Shah. Optimal Aggregation of Uncertain Preferences. In Thirtieth AAAI Conference on Artificial Intelligence, pages 608-614, 2016.","title":"Optimal Aggregation of Uncertain Preferences","authors":"Shah Procaccia, Ariel D Procaccia, Nisarg Shah","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Soufiani et al., 2013] Hossein Azari Soufiani, David C Parkes, and Lirong Xia. Preference Elicitation For General Random Utility Models. In Proceedings of the Twenty- Ninth Conference on Uncertainty in Artificial Intelligence, pages 596-605, 2013.","title":"Preference Elicitation For General Random Utility Models","authors":"Hossein Azari Soufiani, David C Parkes, Lirong Xia","year":"2013","venue":"Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tversky and Kahneman, 1981] Amos Tversky and Daniel Kahneman. The Framing of Decisions and the Psychology of Choice. Science, 211(4481):453-458, 1981.","title":"The Framing of Decisions and the Psychology of Choice","authors":"Kahneman ; Tversky, Daniel Tversky, Kahneman","year":"1981","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xia, 2019] Lirong Xia. Learning and Decision-Making from Rank Data. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2019.","title":"Learning and Decision-Making from Rank Data","authors":"Lirong Xia","year":"2019","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao and Xia, 2018] Zhibing Zhao and Lirong Xia. Com- posite Marginal Likelihood Methods for Random Utility Models. In Proceedings of the 35th International Confer- ence on Machine Learning, pages 5922-5931, 2018.","title":"Composite Marginal Likelihood Methods for Random Utility Models","authors":"Xia Zhao, Zhibing Zhao, Lirong Xia","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2016] Zhibing Zhao, Peter Piech, and Lirong Xia. Learning Mixtures of Plackett-Luce Models. In Pro- ceedings of the 33rd International Conference on Machine Learning (ICML-16), 2016.","title":"Learning Mixtures of Plackett-Luce Models","authors":"Zhao","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning (ICML-16)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2018] Zhibing Zhao, Haoming Li, Junming Wang, Jeffrey Kephart, Nicholas Mattei, Hui Su, and Lirong Xia. A Cost-Effective Framework for Preference Elicitation and Aggregation. In Proceedings of the 2018 Conference on Uncertainty in Artificial Intelligence (UAI), 2018.","title":"A Cost-Effective Framework for Preference Elicitation and Aggregation","authors":"Zhibing Zhao, Haoming Li, Junming Wang, Jeffrey Kephart, Nicholas Mattei, Hui Su, Lirong Xia","year":"2018","venue":"Proceedings of the 2018 Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}