{"type":"ScholarlyArticle","title":"Quantifying Algorithmic Improvements over Time","authors":[{"name":"Lars Kotthoff","slug":"lars-kotthoff","given":"Lars","family":"Kotthoff","affiliation":"University of Wyoming ,"},{"name":"Alexandre Fréchette","slug":"alexandre-frechette","given":"Alexandre","family":"Fréchette","affiliation":"University of British Columbia ,"},{"name":"Tomasz Michalak","slug":"tomasz-michalak","given":"Tomasz","family":"Michalak","affiliation":"University of Warsaw ,"},{"name":"Talal Rahwan","slug":"talal-rahwan","given":"Talal","family":"Rahwan","affiliation":"Masdar Institute of Science and Technology ,"},{"name":"Holger H. Hoos","slug":"holger-h-hoos","given":"Holger H.","family":"Hoos","affiliation":"Universiteit Leiden; University of British Columbia ,"},{"name":"Kevin Leyton-Brown","slug":"kevin-leyton-brown","given":"Kevin","family":"Leyton-Brown","affiliation":"University of British Columbia ,"}],"abstract":"Assessing the progress made in AI and contributions to the state of the art is of major concern to\r\nthe community. Recently, Frechette et al. [2016]\r\nadvocated performing such analysis via the Shapley\r\nvalue, a concept from coalitional game theory. In\r\nthis paper, we argue that while this general idea is\r\nsound, it unfairly penalizes older algorithms that\r\nadvanced the state of the art when introduced, but\r\nwere then outperformed by modern counterparts.\r\nDriven by this observation, we introduce the temporal Shapley value, a measure that addresses this\r\nproblem while maintaining the desirable properties\r\nof the (classical) Shapley value. We use the tempo-\r\nral Shapley value to analyze the progress made in\r\n(i) the different versions of the Quicksort algorithm;\r\n(ii) the annual SAT competitions 2007–2014; (iii)\r\nan annual competition of Constraint Programming,\r\nnamely the MiniZinc challenge 2014–2016. Our\r\nanalysis reveals novel insights into the development\r\nmade in these important areas of research over time.","doi":"10.24963/ijcai.2018/716","url":"https://ijcai.org/proceedings/2018/716","pdf":"https://ijcai.org/proceedings/2018/0716.pdf","markdown":"https://ijcai.org/proceedings/2018/716.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/716","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Evolution of the contours of AI","track":"Evolution of the contours of AI","number":716,"pages":{"first":5165,"last":5171},"keywords":["Heuristic Search and Game Playing: Meta-Reasoning and Meta-heuristics","Heuristic Search and Game Playing: Evaluation and Analysis"],"videos":[],"references":[{"position":1,"raw":"[Aadithya et al., 2011] K.V. Aadithya, T.P. Michalak, and N.R. Jennings. Representation of coalitional games with algebraic decision diagrams. In 10th International Joint Conference on Autonomous Agents and Multi-Agent Sys- tems, pages 1121-1122, 2011.","title":"Representation of coalitional games with algebraic decision diagrams","authors":"K V Aadithya, T P Michalak, N R Jennings","year":"2011","venue":"10th International Joint Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bentley and McIlroy, 1993] Jon L. Bentley and M. Douglas McIlroy. Engineering a sort function. Software: Practice and Experience, 23(11):1249-1265, 1993.","title":"Engineering a sort function","authors":"Jon L Bentley, M Douglas Mcilroy","year":"1993","venue":"Software: Practice and Experience","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carlsson et al., 1997] Mats Carlsson, Greger Ottosson, and Björn Carlson. An open-ended finite domain constraint solver. In Programming Languages: Implementations, Log- ics, and Programs, 9th International Symposium, pages 191-206, 1997.","title":"An open-ended finite domain constraint solver","authors":"Carlsson","year":"1997","venue":"Programming Languages: Implementations, Logics, and Programs, 9th International Symposium","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chalkiadakis et al., 2011] G. Chalkiadakis, E. Elkind, and M. Wooldridge. Computational Aspects of Cooperative Game Theory. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2011.","title":"Synthesis Lectures on Artificial Intelligence and Machine Learning","authors":"G Chalkiadakis, E Elkind, M Wooldridge","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elkind et al., 2009] E. Elkind, L.A. Goldberg, P.W. Gold- berg, and M. Wooldridge. A tractable and expressive class of marginal contribution nets and its applications. Mathe- matical Logic Quarterly, 55(4):362-376, 2009.","title":"A tractable and expressive class of marginal contribution nets and its applications","authors":"E Elkind, L A Goldberg, P W Goldberg, M Wooldridge","year":"2009","venue":"Mathematical Logic Quarterly","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Faigle and Kern, 1992] Ulrich Faigle and Walter Kern. The Shapley value for cooperative games under precedence con- straints. International Journal of Game Theory, 21(3):249- 266, 1992.","title":"The Shapley value for cooperative games under precedence constraints","authors":"Kern Faigle, Ulrich Faigle, Walter Kern","year":"1992","venue":"International Journal of Game Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fréchette et al., 2016] Alexandre Fréchette, Lars Kotthoff, Talal Rahwan, Holger H. Hoos, Kevin Leyton-Brown, and Tomasz P. Michalak. Using the Shapley value to analyze algorithm portfolios. In 30th AAAI Conference on Artificial Intelligence, pages 3397-3403, February 2016.","title":"Using the Shapley value to analyze algorithm portfolios","authors":"Alexandre Fréchette, Lars Kotthoff, Talal Rahwan, H Holger, Kevin Hoos, Tomasz P Leyton-Brown, Michalak","year":"2016","venue":"30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gebser et al., 2011] Martin Gebser, Roland Kaminski, Ben- jamin Kaufmann, Torsten Schaub, Marius Thomas Schnei- der, and Stefan Ziller. A portfolio solver for answer set programming: preliminary report. In 11th International Conference on Logic Programming and Nonmonotonic Rea- soning, pages 352-357. Springer, 2011.","title":"A portfolio solver for answer set programming: preliminary report","authors":"Martin Gebser, Roland Kaminski, Benjamin Kaufmann, Torsten Schaub, Marius Thomas Schneider, Stefan Ziller","year":"2011","venue":"11th International Conference on Logic Programming and Nonmonotonic Reasoning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gomes and Selman, 2001] Carla P. Gomes and Bart Selman. Algorithm portfolios. Artificial Intelligence, 126(1-2):43- 62, 2001.","title":"Algorithm portfolios","authors":"Carla P Gomes, Bart Selman","year":"2001","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Helmert et al., 2011] Malte Helmert, Gabriele Röger, and Erez Karpas. Fast downward stone soup: A baseline for building planner portfolios. In ICAPS-2011 Workshop on Planning and Learning (PAL), pages 28-35, 2011.","title":"Fast downward stone soup: A baseline for building planner portfolios","authors":"Malte Helmert, Gabriele Röger, Erez Karpas","year":"2011","venue":"ICAPS-2011 Workshop on Planning and Learning (PAL)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoare, 1961] C. A. R. Hoare. Algorithm 63: Partition. Com- mun. ACM, 4(7):321, July 1961.","title":"Algorithm 63: Partition. Commun","authors":"C A R Hoare","year":"1961","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huberman et al., 1997] Bernardo A. Huberman, Rajan M. Lukose, and Tad Hogg. An economics approach to hard computational problems. Science, 275:51-54, January 3 1997.","title":"An economics approach to hard computational problems","authors":"Bernardo A Huberman, Rajan M Lukose, Tad Hogg","year":"1997","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ieong and Shoham, 2005] S. Ieong and Y. Shoham. Marginal contribution nets: a compact representation scheme for coalitional games. In 6th ACM Conference on Electronic Commerce, pages 193-202, 2005.","title":"Marginal contribution nets: a compact representation scheme for coalitional games","authors":"S Ieong, Y Shoham","year":"2005","venue":"6th ACM Conference on Electronic Commerce","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Knuth, 1998] Donald E. Knuth. The Art of Computer Pro- gramming, Volume 3: Sorting and Searching. Addison Wesley Longman Publishing Co., Inc., 1998.","title":"The Art of Computer Programming, Volume 3: Sorting and Searching","authors":"Donald E Knuth","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kotthoff, 2014] Lars Kotthoff. Algorithm selection for com- binatorial search problems: A survey. AI Magazine, 35(3):48-60, 2014.","title":"Lars Kotthoff. Algorithm selection for combinatorial search problems: A survey","authors":"Kotthoff","year":"2014","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kushagra et al., 2014] Shrinu Kushagra, Alejandro López- Ortiz, J. Ian Munro, and Aurick Qiao. Multi-pivot Quick- sort: Theory and Experiments. In Meeting on Algorithm Engineering & Expermiments, pages 47-60. Society for Industrial and Applied Mathematics, 2014.","title":"Multi-pivot Quicksort: Theory and Experiments","authors":"Kushagra","year":"2014","venue":"Meeting on Algorithm Engineering & Expermiments","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Leyton-Brown et al., 2003] Kevin Leyton-Brown, Eugene Nudelman, Galen Andrew, James McFadden, and Yoav Shoham. A portfolio approach to algorithm selection. In 18th International Joint Conference on Artificial Intelli- gence, pages 1542-1543, 2003.","title":"A portfolio approach to algorithm selection","authors":"Kevin Leyton-Brown, Eugene Nudelman, Galen Andrew, James Mcfadden, Yoav Shoham","year":"2003","venue":"18th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Michalak et al., 2010] T.P. Michalak, D. Marciniak, M. Samotulski, T. Rahwan, P. McBurney, M. Wooldridge, and N.R. Jennings. A logic-based representation for coalitional games with externalities. In 9th International Joint Conference on Autonomous Agents and Multi-Agent Systems, pages 125-132, 2010.","title":"A logic-based representation for coalitional games with externalities","authors":"T P Michalak, D Marciniak, M Samotulski, T Rahwan, P Mcburney, M Wooldridge, N R Jennings","year":"2010","venue":"9th International Joint Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Monderer and Samet, 2002] Dov Monderer and Dov Samet. Variations on the Shapley value. In Handbook of Game Theory with Economic Applications, volume 3, chapter 54, pages 2055-2076. Elsevier, 1st edition, 2002.","title":"Variations on the Shapley value","authors":"Samet Monderer, Dov Monderer, Dov Samet","year":"2002","venue":"Handbook of Game Theory with Economic Applications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[SAT Competitions Website, 2017] SAT Competitions Web- site. http://www.satcompetition.org, 2017.","title":"SAT Competitions Website","authors":null,"year":"2017","venue":null,"doi":null,"url":"http://www.satcompetition.org,2017","internal_url":null},{"position":21,"raw":"[Sedgewick, 1978] Robert Sedgewick. Implementing Quick- sort Programs. Commun. ACM, 21(10):847-857, October 1978.","title":"Implementing Quicksort Programs","authors":"Robert Sedgewick","year":"1978","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shapley, 1953] Lloyd S. Shapley. A value for n-person games. In Contributions to the Theory of Games, volume II, pages 307-317. Princeton University Press, 1953.","title":"A value for n-person games","authors":"Lloyd S Shapley","year":"1953","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Solan et al., 2013] Eilon Solan, Shmuel Zamir, and Michael Maschler. Game Theory. Cambridge University Press, 2013.","title":"Game Theory","authors":"Eilon Solan, Shmuel Zamir, Michael Maschler","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stuckey et al., 2014] Peter J. Stuckey, Thibaut Feydy, An- dreas Schutt, Guido Tack, and Julien Fischer. The MiniZinc Challenge 2008-2013. AI Magazine, 35(2):55-60, 2014.","title":"The MiniZinc Challenge 2008-2013","authors":"J Peter, Thibaut Stuckey, Andreas Feydy, Guido Schutt, Julien Tack, Fischer","year":"2014","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2008] Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Re- search, 32:565-606, 2008.","title":"SATzilla: portfolio-based algorithm selection for SAT","authors":"Lin Xu, Frank Hutter, H Holger, Kevin Hoos, Leyton-Brown","year":"2008","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2012] Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Evaluating component solver contri- butions to portfolio-based algorithm selectors. In 15th In- ternational Conference on Theory and Applications of Sat- isfiability Testing, pages 228-241. Springer-Verlag, 2012.","title":"Evaluating component solver contributions to portfolio-based algorithm selectors","authors":"Lin Xu, Frank Hutter, H Holger, Kevin Hoos, Leyton-Brown","year":"2012","venue":"15th International Conference on Theory and Applications of Satisfiability Testing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yaroslavskiy, 2009] Vladimir Yaroslavskiy. Dual- Pivot Quicksort algorithm. http://codeblab. com/wp-content/uploads/2009/09/ DualPivotQuicksort.pdf, 2009.","title":"Dual-Pivot Quicksort algorithm","authors":"Vladimir Yaroslavskiy","year":"2009","venue":null,"doi":null,"url":"http://codeblab.com/wp-content/uploads/2009/09/DualPivotQuicksort.pdf","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}