{"type":"ScholarlyArticle","title":"How Well Do Machines Perform on IQ tests: a Comparison Study on a Large-Scale Dataset","authors":[{"name":"Yusen Liu","slug":"yusen-liu","given":"Yusen","family":"Liu","affiliation":"School of Computer Science and Technology , Tianjin University"},{"name":"Fangyuan He","slug":"fangyuan-he","given":"Fangyuan","family":"He","affiliation":"School of Computer Science and Technology , Tianjin University"},{"name":"Haodi Zhang","slug":"haodi-zhang","given":"Haodi","family":"Zhang","affiliation":"College of Computer Science and Software Engineering , Shenzhen University"},{"name":"Guozheng Rao","slug":"guozheng-rao","given":"Guozheng","family":"Rao","affiliation":"School of Computer Science and Technology , Tianjin University"},{"name":"Zhiyong Feng","slug":"zhiyong-feng","given":"Zhiyong","family":"Feng","affiliation":"School of Computer Science and Technology , Tianjin University"},{"name":"Yi Zhou","slug":"yi-zhou","given":"Yi","family":"Zhou","affiliation":"Shanghai Research Center for Brain Science and Brain-Inspired Intelligence/Zhangjiang Laboratory; School of Natural and Computational Sciences , Massey University"}],"abstract":"AI benchmarking becomes an increasingly important task. As suggested by many researchers, Intelligence Quotient (IQ) tests, which is widely regarded as one of the predominant benchmarks for measuring human intelligence, raises an interesting challenge for AI systems. For better solving IQ tests automatedly by machines, one needs to use, combine and advance many areas in AI including knowledge representation and reasoning, machine learning, natural language processing and image understanding. Also, automated IQ tests provides an ideal testbed for integrating symbolic and sub-symbolic approaches as both are found useful here. Hence, we argue that IQ tests, although not suitable for testing machine intelligence, provides an excellent benchmark for the current development of AI research. Nevertheless, most existing IQ test datasets are not comprehensive enough for this purpose. As a result, the conclusions obtained are not representative. To address this issue, we create IQ10k, a large-scale dataset that contains more than 10,000 IQ test questions. We also conduct a comparison study on IQ10k with a number of state-of-the-art approaches.","doi":"10.24963/ijcai.2019/846","url":"https://ijcai.org/proceedings/2019/846","pdf":"https://ijcai.org/proceedings/2019/0846.pdf","markdown":"https://ijcai.org/proceedings/2019/846.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/846","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Understanding Intelligence and Human-level AI in the New Machine Learning era","track":"Understanding Intelligence and Human-level AI in the New Machine Learning era","number":846,"pages":{"first":6110,"last":6116},"keywords":["Special Track on Understanding Intelligence and Human-level AI in the New Machine Learning era: Learning knowledge representations (Special Track on Human AI and Machine Learning)","Special Track on Understanding Intelligence and Human-level AI in the New Machine Learning era: Integrating Learning and (any form of) Reasoning (Special Track on Human AI and Machine Learning)"],"videos":[],"references":[{"position":1,"raw":"[Bayoudh et al., 2012] Meriam Bayoudh, Henri Prade, and Gilles Richard. Evaluation of Analogical Proportions through Kolmogorov Complexity. Knowledge-Based Sys- tems, 29(3):20-30, 2012.","title":"Evaluation of Analogical Proportions through Kolmogorov Complexity","authors":"Bayoudh","year":"2012","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bringsjord, 2011] Selmer Bringsjord. Psychometric artifi- cial intelligence. Journal of Experimental and Theoretical Artificial Intelligence, 23(3):271-277, 2011.","title":"Psychometric artificial intelligence","authors":"Selmer Bringsjord","year":"2011","venue":"Journal of Experimental and Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buchberger, 1993] Bruno Buchberger. Mathematica: A System for Doing Mathematics by Computer? Interna- tional Symposium on Design and Implementation of Sym- bolic Computation Systems, pages 1-1, 1993.","title":"Mathematica: A System for Doing Mathematics by Computer? International Symposium on Design and Implementation of Symbolic Computation Systems","authors":"Bruno Buchberger","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Detterman, 2011] Douglas K Detterman. A Challenge to Watson. Intelligence, 2(39):77-78, 2011.","title":"A Challenge to Watson","authors":"Douglas K Detterman","year":"2011","venue":"Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dowe and Hernández-Orallo, 2014] David L Dowe and José Hernández-Orallo. How universal can an intelligence test be? Adaptive Behavior -Animals, Animats, Soft- ware Agents, Robots and Adaptive Systems, 22(1):51-69, February 2014.","title":"How universal can an intelligence test be? Adaptive Behavior -Animals","authors":"L David, José Dowe, Hernández-Orallo","year":"2014","venue":"Animats, Software Agents, Robots and Adaptive Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Forbus et al., 2005] Kenneth D Forbus, Andrew Lovett, Emmett Tomai, and Jeffrey Usher. A Structure Mapping Model for Solving Geometric Analogy Problems. Pro- ceedings of the Cognitive Science Society, 27(27), 2005.","title":"A Structure Mapping Model for Solving Geometric Analogy Problems","authors":"Andrew Kenneth D Forbus, Emmett Lovett, Jeffrey Tomai, Usher","year":"2005","venue":"Proceedings of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hernández-Orallo et al., 2016] José Hernández-Orallo, Fer- nando Martínez-Plumed, Ute Schmid, Michael Siebers, and David L Dowe. Computer models solving intelligence test problems: Progress and implications. Artificial Intel- ligence, 230:74-107, 2016.","title":"Computer models solving intelligence test problems: Progress and implications","authors":"Hernández-Orallo","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hernández-Orallo, 2017] José Hernández-Orallo. Evalu- ation in Artificial Intelligence: from Task-oriented to Ability-oriented Measurement. Artificial Intelligence Re- view, 48(3):397-447, 2017.","title":"José Hernández-Orallo. Evaluation in Artificial Intelligence: from Task-oriented to Ability-oriented Measurement","authors":"Hernández-Orallo","year":"2017","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hofmann et al., 2014] Jacqueline Hofmann, Emanuel Kitzelmann, and Ute Schmid. Applying Inductive Pro- gram Synthesis to Induction of Number Series a Case Study with IGOR2. In KI'2014, pages 25-36, 2014.","title":"Applying Inductive Program Synthesis to Induction of Number Series a Case Study with IGOR2","authors":"Jacqueline Hofmann, Emanuel Kitzelmann, Ute Schmid","year":"2014","venue":"KI'2014","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Holzman et al., 1983] Thomas G Holzman, James W Pel- legrino, and Robert Glaser. Cognitive Variables in Se- ries Completion. Journal of Educational Psychology, 75(4):603, 1983.","title":"Cognitive Variables in Series Completion","authors":"James W Thomas G Holzman, Robert Pellegrino, Glaser","year":"1983","venue":"Journal of Educational Psychology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jauhar et al., 2015] Sujay Kumar Jauhar, Chris Dyer, and Eduard H Hovy. Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Mod- els. In HLT-NAACL, pages 683-693, 2015.","title":"Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Models","authors":"Sujay Kumar, Jauhar, Chris Dyer, Eduard H Hovy","year":"2015","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kitano et al., 1997] Hiroaki Kitano, Minoru Asada, Yasuo Kuniyoshi, Itsuki Noda, and Eiichi Osawa. Robocup: The robot world cup initiative. In Proceedings of the first in- ternational conference on Autonomous agents, pages 340- 347. ACM, 1997.","title":"Robocup: The robot world cup initiative","authors":"Hiroaki Kitano, Minoru Asada, Yasuo Kuniyoshi, Itsuki Noda, Eiichi Osawa","year":"1997","venue":"Proceedings of the first international conference on Autonomous agents","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lovett and Forbus, 2017] Andrew Lovett and Kenneth For- bus. Modeling visual problem solving as analogical rea- soning. Psychological review, 124(1):60, 2017.","title":"Andrew Lovett and Kenneth Forbus. Modeling visual problem solving as analogical reasoning","authors":"Forbus Lovett","year":"2017","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lynn and Vanhanen, 2009] Richard Lynn and Tatu Van- hanen. Intelligence and the Wealth and Poverty of Nations, 2009.","title":"Intelligence and the Wealth and Poverty of Nations","authors":"Richard Lynn, Tatu Vanhanen","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Repre- sentations of Words and Phrases and their Compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nakov and Hearst, 2008] Preslav Nakov and Marti A. Hearst. Solving Relational Similarity Problems Using the Web as a Corpus. In Proceedings of the Meeting of the Association for Computational Linguistics, June 15-20, 2008, Columbus, Ohio, Usa, pages 452-460, 2008.","title":"Solving Relational Similarity Problems Using the Web as a Corpus","authors":"Hearst Nakov, Preslav Nakov, Marti A Hearst","year":"2008","venue":"Proceedings of the Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ohlsson et al., 2017] Stellan Ohlsson, Robert H Sloan, György Turán, and Aaron Urasky. Measuring an Artificial Intelligence System's Performance on a Verbal IQ test for Young Children. Journal of Experimental & Theoretical Artificial Intelligence, 29(4):679-693, 2017.","title":"Measuring an Artificial Intelligence System's Performance on a Verbal IQ test for Young Children","authors":"Ohlsson","year":"2017","venue":"Journal of Experimental & Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global Vectors for Word Representation. In EMNLP, volume 14, pages 1532-1543, 2014.","title":"Glove: Global Vectors for Word Representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pietschnig and Voracek, 2015] Jakob Pietschnig and Martin Voracek. One century of global IQ gains: A Formal Meta- analysis of the Flynn Effect (1909-2013). Perspectives on Psychological Science, 10(3):282-306, 2015.","title":"One century of global IQ gains: A Formal Metaanalysis of the Flynn Effect (1909-2013)","authors":"Voracek Pietschnig, Jakob Pietschnig, Martin Voracek","year":"2015","venue":"Perspectives on Psychological Science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Plan, 2016] Strategic Plan. The National Artificial Intelli- gence Research and Development Strategic Plan. 2016. [Ragni and Klein, 2011] Marco Ragni and Andreas Klein. Predicting Numbers: an AI Approach to Solving Number Series. KI 2011: Advances in Artificial Intelligence, pages 255-259, 2011.","title":"Predicting Numbers: an AI Approach to Solving Number Series","authors":"Plan","year":"2011","venue":"Strategic Plan. The National Artificial Intelligence Research and Development Strategic Plan","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ragni and Neubert, 2014] Marco Ragni and Stefanie Neu- bert. Analyzing Raven's Intelligence Test: Cognitive Model, Demand, and Complexity. In Computational Ap- proaches to Analogical Reasoning: Current Trends, pages 351-370. Springer, 2014.","title":"Marco Ragni and Stefanie Neubert","authors":"Neubert Ragni","year":"2014","venue":"Analyzing Raven's Intelligence Test: Cognitive Model, Demand, and Complexity","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rowe et al., 2012] Ellen W Rowe, Cristin Miller, Lauren A Ebenstein, and Dawna F Thompson. Cognitive predictors of reading and math achievement among gifted referrals. School psychology quarterly : the official journal of the Division of School Psychology, American Psychological Association, 27:144-53, 09 2012.","title":"Cognitive predictors of reading and math achievement among gifted referrals","authors":"Ellen W Rowe, Cristin Miller, Lauren A Ebenstein, Dawna F Thompson","year":"2012","venue":"School psychology quarterly : the official journal of the Division of School Psychology, American Psychological Association","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet Large Scale Visual Recogni- tion Challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sanghi and Dowe, 2003] Pritika Sanghi and David L Dowe. A Computer Program Capable of Passing IQ Tests. In 4th Intl. Conf. on Cognitive Science (ICCS'03), Sydney, pages 570-575, 2003.","title":"A Computer Program Capable of Passing IQ Tests","authors":"Dowe Sanghi, Pritika Sanghi, David L Dowe","year":"2003","venue":"4th Intl. Conf. on Cognitive Science (ICCS'03)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Selmer Bringsjord, 2003] Bettina Schimanski Selmer Bringsjord. What is Artificial Intelligence? Psychometric AI as an Answer. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI'03, pages 887-893, 2003.","title":"What is Artificial Intelligence? Psychometric AI as an Answer","authors":"Bettina Schimanski, Selmer Bringsjord","year":"2003","venue":"Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI'03","doi":null,"url":null,"internal_url":"/proceedings/2003/127"},{"position":26,"raw":"[Siebers and Schmid, 2012] Michael Siebers and Ute Schmid. Semi-analytic Natural Number Series Induction. In KI, pages 249-252, 2012.","title":"Semi-analytic Natural Number Series Induction","authors":"Schmid Siebers, Michael Siebers, Ute Schmid","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sloane and others, 2007] Neil JA Sloane et al. The On- line Encyclopedia of Integer Sequences. In Symposium on Towards Mechanized Mathematical Assistants: Inter- national Conference, pages 130-130, 2007.","title":"The Online Encyclopedia of Integer Sequences","authors":"J A Neil, Sloane","year":"2007","venue":"Symposium on Towards Mechanized Mathematical Assistants: International Conference","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Speer et al., 2017] Robert Speer, Joshua Chin, and Cather- ine Havasi. Conceptnet 5.5: An Open Multilingual Graph of General Knowledge. In AAAI, pages 4444-4451, 2017.","title":"Conceptnet 5.5: An Open Multilingual Graph of General Knowledge","authors":"Robert Speer, Joshua Chin, Catherine Havasi","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Strannegård et al., 2013] Claes Strannegård, Abdul Rahim Nizamani, Anders Sjöberg, and Fredrik Engström. Bounded Kolmogorov Complexity Based on Cognitive Models. In International Conference on Artificial General Intelligence, pages 130-139, 2013.","title":"Bounded Kolmogorov Complexity Based on Cognitive Models","authors":"Claes Strannegård, Abdul Rahim Nizamani, Anders Sjöberg, Fredrik Engström","year":"2013","venue":"International Conference on Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Turney, 2012] Peter D Turney. Domain and Function: A Dual-Space Model of Semantic Relations and Composi- tions. Journal of Artificial Intelligence Research, 44:533- 585, 2012.","title":"Domain and Function: A Dual-Space Model of Semantic Relations and Compositions","authors":"D Peter, Turney","year":"2012","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2016] Huazheng Wang, Fei Tian, Bin Gao, Jiang Bian, and Tie-Yan Liu. Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding. EMNLP'16, pages 541-550, 2016.","title":"Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding","authors":"Huazheng Wang, Fei Tian, Bin Gao, Jiang Bian, Tie-Yan Liu","year":"2016","venue":"EMNLP'16","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}