{"type":"ScholarlyArticle","title":"MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces","authors":[{"name":"Shaojun E","slug":"shaojun-e","given":"Shaojun","family":"E","affiliation":""},{"name":"Yuchen Yang","slug":"yuchen-yang","given":"Yuchen","family":"Yang","affiliation":""},{"name":"Jiaheng Wu","slug":"jiaheng-wu","given":"Jiaheng","family":"Wu","affiliation":""},{"name":"Yan Zhang","slug":"yan-zhang","given":"Yan","family":"Zhang","affiliation":""},{"name":"Tiejun Zhao","slug":"tiejun-zhao","given":"Tiejun","family":"Zhao","affiliation":""},{"name":"Ziyan Chen","slug":"ziyan-chen","given":"Ziyan","family":"Chen","affiliation":""}],"abstract":"In the latest advancements in multimodal learning, effectively addressing the spatial and semantic losses of visual data after encoding remains a critical challenge. This is because the performance of large multimodal models is positively correlated with the coupling between visual encoders and large language models. Existing approaches often face issues such as vector gaps or semantic disparities, resulting in information loss during the propagation process. To address these issues, we propose MAGE (Multimodal Alignment and Generation Enhancement), a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism. By introducing the Intelligent Alignment Network (IAN), MAGE achieves dimensional and semantic alignment. To reduce the gap between synonymous heterogeneous data, we employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect. Moreover, to enhance MAGE’s “Any-to-Any” capability, we developed a fine-tuning dataset for multimodal tool-calling instructions to expand the model’s output capability boundaries. Finally, our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks, including MME, MMBench, and SEED. Complete code and appendix are available at: https://github.com/GTCOM-NLP/MAGE","doi":"10.24963/ijcai.2025/107","url":"https://ijcai.org/proceedings/2025/107","pdf":"https://ijcai.org/proceedings/2025/0107.pdf","markdown":"https://ijcai.org/proceedings/2025/107.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/107","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":107,"pages":{"first":954,"last":962},"keywords":["Computer Vision: CV: Multimodal learning","Natural Language Processing: NLP: Language generation","Agent-based and Multi-agent Systems: MAS: Multi-agent planning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}