{"type":"ScholarlyArticle","title":"Computer Models Solving Intelligence Test Problems: Progress and Implications (Extended Abstract)","authors":[{"name":"José Hernández-Orallo","slug":"jose-hernandez-orallo","given":"José","family":"Hernández-Orallo","affiliation":"Universitat Politècnica de València ,"},{"name":"Fernando Martínez-Plumed","slug":"fernando-martinez-plumed","given":"Fernando","family":"Martínez-Plumed","affiliation":"Universitat Politècnica de València ,"},{"name":"Ute Schmid","slug":"ute-schmid","given":"Ute","family":"Schmid","affiliation":"University of Bamberg , Germany"},{"name":"Michael Siebers","slug":"michael-siebers","given":"Michael","family":"Siebers","affiliation":"University of Bamberg , Germany"},{"name":"David Dowe","slug":"david-dowe","given":"David","family":"Dowe","affiliation":"Monash University , Australia"}],"abstract":"While some computational models of intelligence test problems were proposed throughout the second half of the XXth century, in the first years of the XXIst century we have seen an increasing number of computer systems being able to score well on particular intelligence test tasks. However, despitethis increasing trend there has been no general account of all these works in terms of how theyrelate to each other and what their real achievements are. In this paper, we provide some insighton these issues by giving a comprehensive account of about thirty computer models, from the 1960sto nowadays, and their relationships, focussing on the range of intelligence test tasks they address, thepurpose of the models, how general or specialised these models are, the AI techniques they use in eachcase, their comparison with human performance, and their evaluation of item difficulty.","doi":"10.24963/ijcai.2017/711","url":"https://ijcai.org/proceedings/2017/711","pdf":"https://ijcai.org/proceedings/2017/0711.pdf","markdown":"https://ijcai.org/proceedings/2017/711.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/711","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Journal track","track":"Journal track","number":711,"pages":{"first":5005,"last":5009},"keywords":["Multidisciplinary Topics and Applications: Cognitive Modeling","Multidisciplinary Topics and Applications: Brain Sciences"],"videos":[],"references":[{"position":1,"raw":"[Bayoudh et al., 2012] Meriam Bayoudh, Henri Prade, and Gilles Richard. Evaluation of analogical proportions through kolmogorov complexity. Knowledge-Based Sys- tems, 29(0):20-30, 2012. Artificial Intelligence 2010.","title":"Evaluation of analogical proportions through kolmogorov complexity","authors":"Bayoudh","year":"2010","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bennett, 1969] George K. Bennett. Bennett mechanical comprehension test. Psychological Corporation, 1969.","title":"Bennett mechanical comprehension test","authors":"George K Bennett","year":"1969","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bongard, 1970] Mikhail Moiseevich Bongard. Pattern Recognition. Spartan Books, 1970.","title":"Pattern Recognition","authors":"Bongard","year":"1970","venue":"Mikhail Moiseevich Bongard","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bringsjord and Schimanski, 2003] Selmer Bringsjord and Bettina Schimanski. What is artificial intelligence? Psy- chometric AI as an answer. In International Joint Confer- ence on Artificial Intelligence, pages 887-893, 2003.","title":"What is artificial intelligence? Psychometric AI as an answer","authors":"Schimanski Bringsjord, Selmer Bringsjord, Bettina Schimanski","year":"2003","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2003/127"},{"position":5,"raw":"[Campbell et al., 2002] Murray Campbell, A. Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial Intelligence, 134(1-2):57-83, 2002.","title":"Deep Blue","authors":"Murray Campbell, A Joseph Hoane, Feng-Hsiung Hsu","year":"2002","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carpenter et al., 1990] Patricia A. Carpenter, Marcel A. Just, and Peter Shell. What one intelligence test measures: A theoretical account of the processing in the Raven Pro- gressive Matrices test. Psychological review, 97:404-431, 1990.","title":"What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices test","authors":"Patricia A Carpenter, Marcel A Just, Peter Shell","year":"1990","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Detterman, 2011] Douglas K. Detterman. A challenge to Watson. Intelligence, 39(2-3):77-78, 2011.","title":"A challenge to Watson","authors":"K Douglas, Detterman","year":"2011","venue":"Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dowe and Hernández-Orallo, 2014] David L. Dowe and José Hernández-Orallo. How universal can an intelligence test be? Adaptive Behavior, 22(1):51-69, 2014.","title":"How universal can an intelligence test be?","authors":"David L Dowe, José Hernández-Orallo","year":"2014","venue":"Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Eliasmith et al., 2012] Chris Eliasmith, Terrence C. Stew- art, XXuan Choo, Trevor Bekolay, Travis DeWolf, Yichuan Tang, and Daniel Rasmussen. A large-scale model of the functioning brain. Science, 338(6111):1202- 1205, 2012.","title":"A large-scale model of the functioning brain","authors":"Chris Eliasmith, Terrence C Stewart, Xxuan Choo, Trevor Bekolay, Travis Dewolf, Yichuan Tang, Daniel Rasmussen","year":"2012","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Evans, 1965] Thomas G. Evans. A heuristic program to solve geometric-analogy problems. In Proc. SJCC, vol- ume 25, pages 327-339, 1965. vol. 25. Ferrucci et al., 2010]","title":"A heuristic program to solve geometric-analogy problems","authors":"G Thomas, Evans","year":"1965","venue":"Proc. SJCC","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"David Ferrucci, David Brown, Jen- nifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Ny- berg, John Prager, et al. Building Watson: An overview of the DeepQA project. AI Magazine, 31(3):59-79, 2010.","title":"Building Watson: An overview of the DeepQA project","authors":"David Ferrucci, David Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager","year":"2010","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ferrucci et al., 2013] David Ferrucci, Anthony Levas, Sug- ato Bagchi, David Gondek, and Erik T. Mueller. Watson: Beyond jeopardy! Artificial Intelligence, 199:93-105, 2013.","title":"Watson: Beyond jeopardy!","authors":"David Ferrucci, Anthony Levas, Sugato Bagchi, David Gondek, Erik T Mueller","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Foundalis, 2006] Harry E. Foundalis. Phaeaco: A cogni- tive architecture inspired by Bongard's problems. Doc- toral thesis, Indiana University, Bloomington, 2006.","title":"Phaeaco: A cognitive architecture inspired by Bongard's problems","authors":"Harry E Foundalis","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gulwani et al., 2015] Sumit Gulwani, Jose Hernandez- Orallo, Emanuel Kitzelmann, Stephen H. Muggleton, Ute Schmid, and Benjamin Zorn. Inductive programming meets the real world. Communications of the ACM, 58(11):90-99, 2015.","title":"Inductive programming meets the real world","authors":"Sumit Gulwani, Jose Hernandez-Orallo, Emanuel Kitzelmann, Stephen H Muggleton, Ute Schmid, Benjamin Zorn","year":"2015","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hernández-Orallo et al., 2016] José Hernández-Orallo, Fer- nando Martínez-Plumed, Ute Schmid, Michael Siebers, and David L. Dowe. Computer models solving intelligence test problems: Progress and implications. Artificial Intel- ligence, 230:74 -107, 2016.","title":"Computer models solving intelligence test problems: Progress and implications","authors":"Hernández-Orallo","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hernández-Orallo, 2016] José Hernández-Orallo. Evalua- tion in artificial intelligence: from task-oriented to ability- oriented measurement. Artificial Intelligence Review, pages 1-51, 2016.","title":"José Hernández-Orallo. Evaluation in artificial intelligence: from task-oriented to abilityoriented measurement","authors":"Hernández-Orallo","year":"2016","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hernández-Orallo, 2017] José Hernández-Orallo. The mea- sure of all minds: Evaluating natural and artificial intelli- gence. 2017.","title":"José Hernández-Orallo. The measure of all minds: Evaluating natural and artificial intelligence","authors":"Hernández-Orallo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hofmann et al., 2014] Jacqueline Hofmann, Emanuel Kitzelmann, and Ute Schmid. Applying inductive pro- gram synthesis to induction of number series a case study with IGOR2. In KI 2014: Advances in AI, pages 25-36.","title":"Applying inductive program synthesis to induction of number series a case study with IGOR2","authors":"Jacqueline Hofmann, Emanuel Kitzelmann, Ute Schmid","year":null,"venue":"KI 2014: Advances in AI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hofstadter and Mitchell, 1984] Douglas R. Hofstadter and Melanie Mitchell. The copycat project, 1984.","title":"The copycat project","authors":"Douglas R Hofstadter, Melanie Mitchell","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Klenk et al., 2011] Matthew Klenk, Kenneth Forbus, Em- mett Tomai, and Hyeonkyeong Kim. Using analogical model formulation with sketches to solve Bennett mechan- ical comprehension test problems. Journal of Experimen- tal & Theoretical Artificial Intelligence, 23(3):299-327, 2011.","title":"Using analogical model formulation with sketches to solve Bennett mechanical comprehension test problems","authors":"Matthew Klenk, Kenneth Forbus, Emmett Tomai, Hyeonkyeong Kim","year":"2011","venue":"Journal of Experimental & Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kunda et al., 2013] Maithilee Kunda, Keith McGreggor, and Ashok K. Goel. A computational model for solving problems from the Ravens Progressive Matrices intelli- gence test using iconic visual representations. Cognitive Systems Research, pages 22-23, 47-66, 2013.","title":"A computational model for solving problems from the Ravens Progressive Matrices intelligence test using iconic visual representations","authors":"Kunda","year":"2013","venue":"Cognitive Systems Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lovett and Forbus, 2011] Andrew Lovett and Kenneth For- bus. Cultural commonalities and differences in spatial problem-solving: A computational analysis. Cognition, 121(2):281-287, 2011.","title":"Andrew Lovett and Kenneth Forbus. Cultural commonalities and differences in spatial problem-solving: A computational analysis","authors":"Forbus Lovett","year":"2011","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lovett and Forbus, 2012] Andrew Lovett and Kenneth For- bus. Modeling multiple strategies for solving geometric analogy problems. In Proceedings of the 34th Annual Con- ference of the Cognitive Science Society, pages 701-706, 2012.","title":"Andrew Lovett and Kenneth Forbus. Modeling multiple strategies for solving geometric analogy problems","authors":"Forbus Lovett","year":"2012","venue":"Proceedings of the 34th Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lovett et al., 2010] Andrew Lovett, Kenneth Forbus, and Jeffrey Usher. A structure-mapping model of Ravens Pro- gressive Matrices. In Proceedings of CogSci, volume 10, pages 2761-2766, 2010.","title":"A structure-mapping model of Ravens Progressive Matrices","authors":"Andrew Lovett, Kenneth Forbus, Jeffrey Usher","year":"2010","venue":"Proceedings of CogSci","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Martínez-Plumed et al., 2017] Fernando Martínez-Plumed, César Ferri, José Hernández-Orallo, and María José Ramírez-Quintana. A computational analysis of general intelligence tests for evaluating cognitive development. Cognitive Systems Research, 43:100-118, 2017.","title":"A computational analysis of general intelligence tests for evaluating cognitive development","authors":"Fernando Martínez-Plumed, César Ferri, José Hernández-Orallo, María José Ramírez-Quintana","year":"2017","venue":"Cognitive Systems Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ohlsson et al., 2013] Stellan Ohlsson, Robert H. Sloan, Gyrgy Turán, and Aaron Urasky. Verbal IQ of a four-year old achieved by an AI system. In COMMONSENSE 2013, 11th International Symposium on Logical Formalizations of Commonsense Reasoning, page 6, 2013.","title":"Verbal IQ of a four-year old achieved by an AI system","authors":"Ohlsson","year":"2013","venue":"COMMONSENSE 2013, 11th International Symposium on Logical Formalizations of Commonsense Reasoning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Oppy and Dowe, 2011] Graham Oppy and David L. Dowe. The Turing Test. In Edward N. Zalta, editor, Stanford Encyclopedia of Philosophy, pages Stanford University, http://plato.stanford.edu/entries/turing-test/, 2011.","title":"The Turing Test","authors":"Graham Oppy, David L Dowe","year":null,"venue":null,"doi":null,"url":"http://plato.stanford.edu/entries/turing-test/,2011","internal_url":null},{"position":28,"raw":"[Prade and Richard, 2014] Henri Prade and Gilles Richard. From analogical proportion to logical proportions: A sur- vey. In Computational Approaches to Analogical Reason- ing: Current Trends, volume 548 of Studies in Computa- tional Intelligence, pages 217-244. Springer Berlin Hei- delberg, 2014.","title":"From analogical proportion to logical proportions: A survey","authors":"Richard ; Prade, Henri Prade, Gilles Richard","year":"2014","venue":"Computational Approaches to Analogical Reasoning: Current Trends","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ragni and Klein, 2011] Marco Ragni and Andreas Klein. Predicting numbers: an AI approach to solving number se- ries. In KI 2011: Advances in Artificial Intelligence, pages 255-259. Springer, 2011.","title":"Predicting numbers: an AI approach to solving number series","authors":"Klein Ragni, Marco Ragni, Andreas Klein","year":"2011","venue":"KI 2011: Advances in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ragni and Neubert, 2014] Marco Ragni and Stefanie Neu- bert. Analyzing ravens intelligence test: Cognitive model, demand, and complexity. In Computational Approaches to Analogical Reasoning: Current Trends, volume 548 of Studies in Computational Intelligence, pages 351-370. Springer, 2014.","title":"Analyzing ravens intelligence test: Cognitive model, demand, and complexity","authors":"Neubert ; Ragni, Stefanie Ragni, Neubert","year":"2014","venue":"Computational Approaches to Analogical Reasoning: Current Trends","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Raven et al., 1992] J. C. Raven, John H. Court, and John Raven. Manual for Raven's Progressive Matrices and Vo- cabulary Scale. San Antonio, TX: Psychological Corpora- tion, 1992.","title":"Manual for Raven's Progressive Matrices and Vocabulary Scale","authors":"J C Raven, John H Court, John Raven","year":"1992","venue":"Psychological Corporation","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Samuel, 1959] Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Re- search and Development, 3(3):210-229, 1959.","title":"Some studies in machine learning using the game of checkers","authors":"L Arthur, Samuel","year":"1959","venue":"IBM Journal of Research and Development","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sanghi and Dowe, 2003] Pritika Sanghi and David L. Dowe. A computer program capable of passing IQ tests. In 4th Intl. Conf. on Cognitive Science (ICCS'03), Sydney, pages 570-575, 2003.","title":"A computer program capable of passing IQ tests","authors":"Dowe Sanghi, Pritika Sanghi, David L Dowe","year":"2003","venue":"4th Intl. Conf. on Cognitive Science (ICCS'03)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Schenck et al., 2012] Connor Schenck, Jivko Sinapov, and Alexander Stoytchev. Which object comes next? Grounded order completion by a humanoid robot. Cyber- netics and Information Technologies, 12(3):5-16, 2012.","title":"Which object comes next? Grounded order completion by a humanoid robot","authors":"Schenck","year":"2012","venue":"Cybernetics and Information Technologies","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Schenck, 2013] Connor Schenck. Intelligence tests for robots: Solving perceptual reasoning tasks with a hu- manoid robot. Master's thesis, Iowa State Univ., 2013.","title":"Intelligence tests for robots: Solving perceptual reasoning tasks with a humanoid robot","authors":"Connor Schenck","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Siebers and Schmid, 2012] Michael Siebers and Ute Schmid. Semi-analytic natural number series induction. In KI 2012: Advances in Artificial Intelligence, pages 249-252. Springer, 2012.","title":"Semi-analytic natural number series induction","authors":"Schmid Siebers, Michael Siebers, Ute Schmid","year":"2012","venue":"KI 2012: Advances in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Simon et al., 1991] Tony Simon, David Klahr, and Allen Newell. SCSoar: Pattern induction in series completion problem solving. In European Soar Workshop, page 17, Cambridge, 1991.","title":"SCSoar: Pattern induction in series completion problem solving","authors":"Tony Simon, David Klahr, Allen Newell","year":"1991","venue":"European Soar Workshop","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Simon, 1963] Kenneth Simon, Herbert A.; Kotovsky. Hu- man acquisition of concepts for sequential patterns. Psy- chological Review, 70(6):534, 1963.","title":"Human acquisition of concepts for sequential patterns","authors":"Kenneth Simon, Herbert A Kotovsky","year":"1963","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sinapov and Stoytchev, 2010] Jivko Sinapov and Alexander Stoytchev. The odd one out task: Toward an intelligence test for robots. In Development and Learning (ICDL), 2010 IEEE 9th International Conference on, pages 126- 131. IEEE, 2010.","title":"Jivko Sinapov and Alexander Stoytchev. The odd one out task: Toward an intelligence test for robots","authors":"Stoytchev Sinapov","year":"2010","venue":"IEEE 9th International Conference on","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sternberg (ed.), 2000] R. J. Sternberg (ed.). Handbook of intelligence. Cambridge University Press, 2000.","title":"Handbook of intelligence","authors":null,"year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Strannegård et al., 2013a] Claes Strannegård, Mehrdad Amirghasemi, and Simon Ulfsbücker. An anthropomor- phic method for number sequence problems. Cognitive Systems Research, 2223:27-34, 2013.","title":"An anthropomorphic method for number sequence problems","authors":"Strannegård","year":"2013","venue":"Cognitive Systems Research","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Strannegård et al., 2013b] Claes Strannegård, Simone Cir- illo, and Victor Ström. An anthropomorphic method for progressive matrix problems. Cognitive Systems Research, 2223(0):35-46, 2013.","title":"An anthropomorphic method for progressive matrix problems","authors":"Strannegård","year":"2013","venue":"Cognitive Systems Research","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Tomai et al., 2005] Emmet Tomai, Andrew Lovett, Kenneth Forbus, and Jeffrey Usher. A structure mapping model for solving geometric analogy problems. In Proceedings of the 27th Annual Conference of the Cognitive Science Society, pages 2190-2195, Stressa, Italy, 2005. Cognitive Science Society, Inc.","title":"A structure mapping model for solving geometric analogy problems","authors":"Emmet Tomai, Andrew Lovett, Kenneth Forbus, Jeffrey Usher","year":"2005","venue":"Proceedings of the 27th Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Turing, 1950] Alan M. Turing. Computing machinery and intelligence. Mind, 59(236):433-460, 1950.","title":"Computing machinery and intelligence","authors":"Alan M Turing","year":"1950","venue":"Mind","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Turney, 2011] Peter D. Turney. Analogy perception applied to seven tests of word comprehension. Journal of Exper- imental & Theoretical Artificial Intelligence, 23(3):343- 362, 2011.","title":"Analogy perception applied to seven tests of word comprehension","authors":"D Peter, Turney","year":"2011","venue":"Journal of Experimental & Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[von Ahn et al., 2004] Luis von Ahn, Manuel Blum, and John Langford. Telling humans and computers apart au- tomatically. Communications of the ACM, 47(2):56-60, 2004.","title":"Telling humans and computers apart automatically","authors":"Von Ahn","year":"2004","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}