{"type":"ScholarlyArticle","title":"A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning","authors":[{"name":"Mikołaj Małkiński","slug":"miko-aj-ma-kinski","given":"Mikołaj","family":"Małkiński","affiliation":""},{"name":"Jacek Mańdziuk","slug":"jacek-mandziuk","given":"Jacek","family":"Mańdziuk","affiliation":""}],"abstract":"We study generalization and knowledge reuse capabilities of deep neural networks in the domain of abstract visual reasoning (AVR), employing Raven's Progressive Matrices (RPMs), a recognized benchmark task for assessing AVR abilities. Two knowledge transfer scenarios referring to the I-RAVEN dataset are investigated. Firstly, inspired by generalization assessment capabilities of the PGM dataset and popularity of I-RAVEN, we introduce Attributeless-I-RAVEN (A-I-RAVEN), a benchmark with 10 generalization regimes that allow to systematically test generalization of abstract rules applied to held-out attributes at various levels of complexity (primary and extended regimes). In contrast to PGM, A-I-RAVEN features compositionality, a variety of figure configurations, and does not require substantial computational resources. Secondly, we construct I-RAVEN-Mesh, a dataset that enriches RPMs with a novel component structure comprising line-based patterns, facilitating assessment of progressive knowledge acquisition in transfer learning setting. We evaluate 13 strong models from the AVR literature on the introduced datasets, revealing their specific shortcomings in generalization and knowledge transfer.","doi":"10.24963/ijcai.2025/660","url":"https://ijcai.org/proceedings/2025/660","pdf":"https://ijcai.org/proceedings/2025/0660.pdf","markdown":"https://ijcai.org/proceedings/2025/660.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/660","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":660,"pages":{"first":5932,"last":5940},"keywords":["Machine Learning: ML: Benchmarks","Machine Learning: ML: Multi-task and transfer learning","Machine Learning: ML: Evaluation","Machine Learning: ML: Deep learning architectures"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}