{"type":"ScholarlyArticle","title":"DANCE: Dual-View Distribution Alignment for Dataset Condensation","authors":[{"name":"Hansong Zhang","slug":"hansong-zhang","given":"Hansong","family":"Zhang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing 100092 , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Shikun Li","slug":"shikun-li","given":"Shikun","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing 100092 , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Fanzhao Lin","slug":"fanzhao-lin","given":"Fanzhao","family":"Lin","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing 100092 , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Weiping Wang","slug":"weiping-wang","given":"Weiping","family":"Wang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing 100092 , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Zhenxing Qian","slug":"zhenxing-qian","given":"Zhenxing","family":"Qian","affiliation":"School of Computer Science , Fudan University , Shanghai 200433 , China"},{"name":"Shiming Ge","slug":"shiming-ge","given":"Shiming","family":"Ge","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing 100092 , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing 100049 , China"}],"abstract":"Dataset condensation addresses the problem of data burden by learning a small synthetic training set that preserves essential knowledge from the larger real training set. To date, the state-of-the-art (SOTA) results are often yielded by optimization-oriented methods, but their inefficiency hinders their application to realistic datasets. On the other hand, the Distribution-Matching (DM) methods show remarkable efficiency but sub-optimal results compared to optimization-oriented methods. In this paper, we reveal the limitations of current DM-based methods from the inner-class and inter-class views, i.e., Persistent Training and Distribution Shift. To address these problems, we propose a new DM-based method named Dual-view distribution AligNment for dataset CondEnsation (DANCE), which exploits a few pre-trained models to improve DM from both inner-class and inter-class views. Specifically, from the inner-class view, we construct multiple ``mid encoders'' to perform pseudo long-term distribution alignment, making the condensed set a good proxy of the real one during the whole training process; while from the inter-class view, we use the expert models to perform distribution calibration, ensuring the synthetic data remains in the real class region during condensing. Experiments demonstrate the proposed method achieves a SOTA performance while maintaining comparable efficiency with the original DM across various scenarios. Source codes are available at https://github.com/Hansong-Zhang/DANCE.","doi":"10.24963/ijcai.2024/186","url":"https://ijcai.org/proceedings/2024/186","pdf":"https://ijcai.org/proceedings/2024/0186.pdf","markdown":"https://ijcai.org/proceedings/2024/186.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/186","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":186,"pages":{"first":1679,"last":1687},"keywords":["Machine Learning: ML: Optimization","Computer Vision: CV: Machine learning for vision","Computer Vision: CV: Image and video synthesis and generation"],"videos":[],"references":[{"position":1,"raw":"[Cazenavette et al., 2022] George Cazenavette, Tongzhou Wang, Antonio Torralba, et al. Dataset distillation by matching training trajectories. In CVPR, pages 4750- 4759, 2022.","title":"Dataset distillation by matching training trajectories","authors":"George Cazenavette, Tongzhou Wang, Antonio Torralba","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cazenavette et al., 2023] George Cazenavette, Tongzhou Wang, Antonio Torralba, et al. Generalizing dataset dis- tillation via deep generative prior. In CVPR, pages 3739- 3748, 2023.","title":"Generalizing dataset distillation via deep generative prior","authors":"George Cazenavette, Tongzhou Wang, Antonio Torralba","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Coleman et al., 2020] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In ICLR, 2020.","title":"Selection via proxy: Efficient data selection for deep learning","authors":"Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cui et al., 2022] Justin Cui, Ruochen Wang, Si Si, and Cho- Jui Hsieh. Dc-bench: Dataset condensation benchmark. In NeurIPS, pages 810-822, 2022.","title":"Dc-bench: Dataset condensation benchmark","authors":"Justin Cui, Ruochen Wang, Si Si, Cho-Jui Hsieh","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Du et al., 2023] Jiawei Du, Yidi Jiang, Vincent T. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accu- mulated trajectory error to improve dataset distillation. In CVPR, pages 3749-3758, 2023.","title":"Minimizing the accumulated trajectory error to improve dataset distillation","authors":"Jiawei Du, Yidi Jiang, T F Vincent, Joey Tianyi Tan, Haizhou Zhou, Li","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Farahani and Hekmatfar, 2009] Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.","title":"Facility location: concepts, models, algorithms and case studies","authors":"Reza Zanjirani, Farahani, Masoud Hekmatfar","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geng et al., 2023] Zongxion Geng, Jiahui andg Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimm- ler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. In IJCAI, 2023.","title":"A survey on dataset distillation: Approaches, applications and future directions","authors":"Geng","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/741"},{"position":9,"raw":"[Guo et al., 2024] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In ICLR, 2024.","title":"Towards lossless dataset distillation via difficulty-aligned trajectory matching","authors":"Guo","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2022] Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, and Bing Yin. Condensing graphs via one-step gradient matching. In ACM KDD, pages 720-730, 2022.","title":"Condensing graphs via one-step gradient matching","authors":"Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, Bing Yin","year":"2022","venue":"ACM KDD","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2022] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML, pages 11102-11118, 2022.","title":"Dataset condensation via efficient synthetic-data parameterization","authors":"Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, Hyun Oh Song","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Univer- sity of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Le and Yang, 2015] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.","title":"Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge","authors":"Yang Le","year":"2015","venue":"CS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023] Shikun Li, Tongliang Liu, Jiyong Tan, Dan Zeng, and Shiming Ge. Trustable co-label learning from multiple noisy annotators. IEEE TMM, 25:1045-1057, 2023.","title":"Trustable co-label learning from multiple noisy annotators","authors":"Shikun Li, Tongliang Liu, Jiyong Tan, Dan Zeng, Shiming Ge","year":"2023","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2024] Shikun Li, Xiaobo Xia, Jiankang Deng, Shiming Ge, and Tongliang Liu. Transferring annotator- and instance-dependent transition matrix for learning from crowds. IEEE TPAMI, pages 1-15, 2024.","title":"Transferring annotatorand instance-dependent transition matrix for learning from crowds","authors":"Shikun Li, Xiaobo Xia, Jiankang Deng, Shiming Ge, Tongliang Liu","year":"2024","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Risheng Liu, Jiaxin Gao, Jin Zhang, et al. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE TPAMI, 44(12):10045-10067, 2021.","title":"Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond","authors":"Risheng Liu, Jiaxin Gao, Jin Zhang","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurIPS, pages 1100-1113, 2022.","title":"Dataset distillation via factorization","authors":"Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023a] Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In CVPR, pages 3759-3768, 2023.","title":"Slimmable dataset condensation","authors":"Liu","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2023b] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: Efficient dataset distillation by representative matching. In ICCV, pages 17268-17278, 2023.","title":"DREAM: Efficient dataset distillation by representative matching","authors":"Liu","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loo et al., 2022] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurIPS, pages 13877- 13891, 2022.","title":"Efficient dataset distillation using random feature approximation","authors":"Noel Loo, Ramin Hasani, Alexander Amini, Daniela Rus","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen et al., 2021] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, pages 5186-5198, 2021.","title":"Dataset distillation with infinitely wide convolutional networks","authors":"Timothy Nguyen, Roman Novak, Lechao Xiao, Jaehoon Lee","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sagun et al., 2018] Levent Sagun, Utku Evci, V. Ugur Güney, Yann N. Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural net- works. In ICLR Workshop, 2018.","title":"Empirical analysis of the hessian of over-parametrized neural networks","authors":"Levent Sagun, Utku Evci, V Ugur Güney, Yann N Dauphin, Léon Bottou","year":"2018","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sajedi et al., 2023] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, et al. Datadam: Efficient dataset distillation with attention matching. In ICCV, pages 17097-17107, 2023.","title":"Datadam: Efficient dataset distillation with attention matching","authors":"Ahmad Sajedi, Samir Khaki, Ehsan Amjadian","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sorscher et al., 2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, et al. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurIPS, pages 19523-19536, 2022.","title":"Beyond neural scaling laws: beating power law scaling via data pruning","authors":"Sorscher","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv, 2018.","title":"Dataset distillation","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022] Kai Wang, Bo Zhao, Xiangyu Peng, et al. CAFE: Learning to condense dataset by aligning features. In CVPR, pages 12196-12205, 2022.","title":"CAFE: Learning to condense dataset by aligning features","authors":"Kai Wang, Bo Zhao, Xiangyu Peng","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023] Cheng Wang, Jiacheng Sun, Zhenhua Dong, Ruixuan Li, and Rui Zhang. Gradient matching for categorical data distillation in ctr prediction. In CRS, pages 161-170, 2023.","title":"Gradient matching for categorical data distillation in ctr prediction","authors":"Cheng Wang, Jiacheng Sun, Zhenhua Dong, Ruixuan Li, Rui Zhang","year":"2023","venue":"CRS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Welling, 2009] Max Welling. Herding dynamical weights to learn. In ICML, pages 1121-1128, 2009.","title":"Herding dynamical weights to learn","authors":"Max Welling","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xia et al., 2022] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In ICLR, 2022.","title":"Moderate coreset: A universal method of data selection for real-world data-efficient deep learning","authors":"Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, Tongliang Liu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xia et al., 2024] Xiaobo Xia, Jiale Liu, Shaokun Zhang, Qingyun Wu, Hongxin Wei, and Tongliang Liu. Refined coreset selection: Towards minimal coreset size under model performance constraints. In ICML, 2024.","title":"Refined coreset selection: Towards minimal coreset size under model performance constraints","authors":"Xiaobo Xia, Jiale Liu, Shaokun Zhang, Qingyun Wu, Hongxin Wei, Tongliang Liu","year":"2024","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2023] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In ICLR, 2023.","title":"Dataset pruning: Reducing training data by examining generalization influence","authors":"Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, Ping Li","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2024] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE TPAMI, 46(1):150-170, 2024.","title":"Dataset distillation: A comprehensive review","authors":"Yu","year":"2024","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pages 6022- 6031, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2023] Lei Zhang, Jie Zhang, Bowen Lei, et al. Accelerating dataset distillation via model augmentation. In CVPR, pages 11950-11959, 2023.","title":"Accelerating dataset distillation via model augmentation","authors":"Lei Zhang, Jie Zhang, Bowen Lei","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2024a] Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. M3d: Dataset con- densation by minimizing maximum mean discrepancy. In AAAI, 2024.","title":"M3d: Dataset condensation by minimizing maximum mean discrepancy","authors":"Zhang","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2024b] Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. Coupled confusion cor- rection: Learning from crowds with sparse annotations. In AAAI, 2024.","title":"Coupled confusion correction: Learning from crowds with sparse annotations","authors":"Zhang","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhao and Bilen, 2021a] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, pages 12674-12685, 2021.","title":"Dataset condensation with differentiable siamese augmentation","authors":"Bilen Zhao, Bo Zhao, Hakan Bilen","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhao and Bilen, 2021b] Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.","title":"Dataset condensation with gradient matching","authors":"Bilen Zhao, Bo Zhao, Hakan Bilen","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhao and Bilen, 2023] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, pages 6503-6512, 2023.","title":"Dataset condensation with distribution matching","authors":"Bo Zhao, Hakan Bilen","year":"2023","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhao et al., 2023] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In CVPR, pages 7856-7865, 2023.","title":"Improved distribution matching for dataset condensation","authors":"Zhao","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zheng et al., 2023] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In ICLR, 2023.","title":"Coverage-centric coreset selection for high pruning rates","authors":"Haizhong Zheng, Rui Liu, Fan Lai, Atul Prakash","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhou et al., 2023] Daquan Zhou, Kai Wang, Jianyang Gu, et al. Dataset quantization. In ICCV, pages 17205-17216, 2023.","title":"Dataset quantization","authors":"Daquan Zhou, Kai Wang, Jianyang Gu","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}