{"type":"ScholarlyArticle","title":"Handling Learnwares Developed from Heterogeneous Feature Spaces without Auxiliary Data","authors":[{"name":"Peng Tan","slug":"peng-tan","given":"Peng","family":"Tan","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hao Tan","slug":"zhi-hao-tan","given":"Zhi-Hao","family":"Tan","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Yuan Jiang","slug":"yuan-jiang","given":"Yuan","family":"Jiang","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"}],"abstract":"The learnware paradigm proposed by Zhou [2016] devotes to constructing a market of numerous well-performed models, enabling users to solve problems by reusing existing efforts rather than starting from scratch. A learnware comprises a trained model and the specification which enables the model to be adequately identified according to the user's requirement. Previous studies concentrated on the homogeneous case where models share the same feature space based on Reduced Kernel Mean Embedding (RKME) specification. However, in real-world scenarios, models are typically constructed from different feature spaces. If such a scenario can be handled by the market, all models built for a particular task even with different feature spaces can be identified and reused for a new user task. Generally, this problem would be easier if there were additional auxiliary data connecting different feature spaces, however, obtaining such data in reality is challenging. In this paper, we present a general framework for accommodating heterogeneous learnwares without requiring additional auxiliary data. The key idea is to utilize the submitted RKME specifications to establish the relationship between different feature spaces. Additionally, we give a matrix factorization-based implementation and propose the overall procedure for constructing and exploiting the heterogeneous learnware market. Experiments on real-world tasks validate the efficacy of our method.","doi":"10.24963/ijcai.2023/471","url":"https://ijcai.org/proceedings/2023/471","pdf":"https://ijcai.org/proceedings/2023/0471.pdf","markdown":"https://ijcai.org/proceedings/2023/471.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/471","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":471,"pages":{"first":4235,"last":4243},"keywords":["Machine Learning: ML: Classification","Machine Learning: ML: Multi-task and transfer learning","Machine Learning: ML: Automated machine learning"],"videos":[],"references":[{"position":1,"raw":"[Ben-David et al., 2006] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems 19, 2006.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2006","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blackard and Dean, 1999] Jock A Blackard and Denis J Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, pages 131-151, 1999.","title":"Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables","authors":"Jock A Blackard, Denis J Dean","year":"1999","venue":"Computers and Electronics in Agriculture","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2010] Deng Cai, Xiaofei He, and Jiawei Han. Lo- cally consistent concept factorization for document clus- tering. IEEE Transactions on Knowledge & Data Engi- neering, 23(6):902-913, 2010.","title":"Locally consistent concept factorization for document clustering","authors":"Deng Cai, Xiaofei He, Jiawei Han","year":"2010","venue":"IEEE Transactions on Knowledge & Data Engineering","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2012] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv:1203.3472, 2012.","title":"Super-samples from kernel herding","authors":"Yutian Chen, Max Welling, Alex Smola","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Colonna et al., 2012] Juan Gabriel Colonna, Afonso D. Ribas, Eulanda Miranda dos Santos, and Eduardo Freire Nakamura. Feature subset selection for automatically clas- sifying anuran calls using sensor networks. In Proceedings of the 2012 International Joint Conference on Neural Net- works, pages 1-8, 2012.","title":"Feature subset selection for automatically classifying anuran calls using sensor networks","authors":"Juan Gabriel Colonna, Afonso D Ribas, Eulanda Miranda Dos Santos, Eduardo Freire Nakamura","year":"2012","venue":"Proceedings of the 2012 International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Day and Khoshgoftaar, 2017] Oscar Day and Taghi M Khoshgoftaar. A survey on heterogeneous transfer learn- ing. Journal of Big Data, 4:1-42, 2017.","title":"Oscar Day and Taghi M Khoshgoftaar. A survey on heterogeneous transfer learning","authors":"Khoshgoftaar Day","year":"2017","venue":"Journal of Big Data","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding and Zhou, 2020] Yao-Xiang Ding and Zhi-Hua Zhou. Boosting-based reliable model reuse. In Proceedings of the 12th Asian Conference on Machine Learning, pages 145-160, 2020.","title":"Boosting-based reliable model reuse","authors":"Yao-Xiang Ding, Zhi-Hua Zhou","year":"2020","venue":"Proceedings of the 12th Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2022] Yao-Xiang Ding, Xi-Zhu Wu, Kun Zhou, and Zhi-Hua Zhou. Pre-trained model reusability eval- uation for small-data transfer learning. In Advances in Neural Information Processing Systems 35, pages 37389- 37400, 2022.","title":"Pre-trained model reusability evaluation for small-data transfer learning","authors":"Yao-Xiang Ding, Xi-Zhu Wu, Kun Zhou, Zhi-Hua Zhou","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Duan et al., 2012] Lixin Duan, Dong Xu, and Ivor Tsang. Learning with augmented features for heterogeneous do- main adaptation. arXiv:1206.4660, 2012.","title":"Learning with augmented features for heterogeneous domain adaptation","authors":"Lixin Duan, Dong Xu, Ivor Tsang","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Févotte and Idier, 2011] Cédric Févotte and Jérôme Idier. Algorithms for nonnegative matrix factorization with the β-divergence. Neural Computation, 23(9):2421-2456, 2011.","title":"Algorithms for nonnegative matrix factorization with the β-divergence","authors":"Cédric Févotte, Jérôme Idier","year":"2011","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garris et al., 1997] Michael Garris, J Blue, Gerald Candela, Patrick Grother, Stanley Janet, and Charles Wilson. Nist form-based handprint recognition system. In NIST Intera- gency/Internal Report, 1997. [Harutyunyan et al., 2019] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):1-18, 2019.","title":"Multitask learning and benchmarking with clinical time series data","authors":"Michael Garris, J Blue, Gerald Candela, Patrick Grother, Stanley Janet, Charles Wilson, ; Harutyunyan","year":"1997","venue":"NIST Interagency/Internal Report","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G Mark. MIMIC-III, a freely ac- cessible critical care database. Scientific Data, 3(1):1-9, 2016.","title":"MIMIC-III, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, Li-Wei H Shen, Mengling Lehman, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific Data","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lippmann et al., 2000] Richard Lippmann, Joshua W Haines, David J Fried, Jonathan Korba, and Kumar Das. Analysis and results of the 1999 darpa off-line intrusion detection evaluation. In International Workshop on Recent Advances in Intrusion Detection, pages 162-182, 2000.","title":"Analysis and results of the 1999 darpa off-line intrusion detection evaluation","authors":"Richard Lippmann, Joshua W Haines, David J Fried, Jonathan Korba, Kumar Das","year":"2000","venue":"International Workshop on Recent Advances in Intrusion Detection","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan and Yang, 2009] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowl- edge & Data Engineering, 22(10):1345-1359, 2009.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2009","venue":"IEEE Transactions on Knowledge & Data Engineering","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Purushotham et al., 2018] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics, 83:112-134, 2018.","title":"Benchmarking deep learning models on large healthcare datasets","authors":"Sanjay Purushotham, Chuizheng Meng, Zhengping Che, Yan Liu","year":"2018","venue":"Journal of Biomedical Informatics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schölkopf and Smola, 2002] Bernhard Schölkopf and Alexander Johannes Smola. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press, 2002.","title":"Learning with kernels: Support vector machines, regularization, optimization, and beyond","authors":"Smola Schölkopf, Bernhard Schölkopf, Alexander Johannes, Smola","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Smola et al., 2007] Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In Proceedings of the 18th International Conference on Algorithmic Learning Theory, pages 13- 31, 2007.","title":"A hilbert space embedding for distributions","authors":"Alex Smola, Arthur Gretton, Le Song, Bernhard Schölkopf","year":"2007","venue":"Proceedings of the 18th International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sriperumbudur et al., 2011] Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R. G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(7):2389-2410, 2011.","title":"Universality, characteristic kernels and RKHS embedding of measures","authors":"K Bharath, Kenji Sriperumbudur, Gert R G Fukumizu, Lanckriet","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tan et al., 2022] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Towards enabling learnware to handle het- erogeneous feature spaces. Machine Learning, 2022.","title":"Towards enabling learnware to handle heterogeneous feature spaces","authors":"Peng Tan, Zhi-Hao Tan, Yuan Jiang, Zhi-Hua Zhou","year":"2022","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[van Breukelen et al., 1998] Martijn van Breukelen, Robert PW Duin, David MJ Tax, and JE Den Hartog. Handwritten digit recognition by combined classifiers. Kybernetika, 34(4):381-386, 1998.","title":"Handwritten digit recognition by combined classifiers","authors":"Martijn Van Breukelen, David Mj Robert Pw Duin, Tax, Je Den, Hartog","year":"1998","venue":"Kybernetika","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Mahadevan, 2011] Chang Wang and Sridhar Ma- hadevan. Heterogeneous domain adaptation using mani- fold alignment. In Proceedings of the 22th International Joint Conference on Artificial Intelligence, 2011.","title":"Heterogeneous domain adaptation using manifold alignment","authors":"Chang Wang, Sridhar Mahadevan","year":"2011","venue":"Proceedings of the 22th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/259"},{"position":22,"raw":"[Wang et al., 2016] Hao Wang, Yan Yang, and Tianrui Li. Multi-view clustering via concept factorization with local manifold regularization. In Proceedings of the 16th Inter- national Conference on Data Mining, pages 1245-1250, 2016.","title":"Multi-view clustering via concept factorization with local manifold regularization","authors":"Hao Wang, Yan Yang, Tianrui Li","year":"2016","venue":"Proceedings of the 16th International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF confer- ence on Computer Vision and Pattern Recognition, pages 11293-11302, 2019.","title":"Characterizing and avoiding negative transfer","authors":"Wang","year":"2019","venue":"Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2023] Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowl- edge and Data Engineering, 35(1):699-710, 2023.","title":"Model reuse with reduced kernel mean embedding specification","authors":"Xi-Zhu Wu, Wenkai Xu, Song Liu, Zhi-Hua Zhou","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and Gong, 2004] Wei Xu and Yihong Gong. Document clustering by concept factorization. In Proceedings of the 27th Annual International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pages 202-209, 2004.","title":"Document clustering by concept factorization","authors":"Wei Xu, Yihong Gong","year":"2004","venue":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2015] Liu Yang, Liping Jing, Jian Yu, and Michael K Ng. Learning transferred weights from co- occurrence data for heterogeneous transfer learning. IEEE Transactions on Neural Networks and Learning Systems, 27(11):2187-2200, 2015.","title":"Learning transferred weights from cooccurrence data for heterogeneous transfer learning","authors":"Liu Yang, Liping Jing, Jian Yu, Michael K Ng","year":"2015","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2018] Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Rectify heterogeneous models with semantic mapping. In Proceedings of the 37th Inter- national Conference on Machine Learning, pages 5630- 5639, 2018.","title":"Rectify heterogeneous models with semantic mapping","authors":"Han-Jia Ye, Yuan De-Chuan Zhan, Zhi-Hua Jiang, Zhou","year":"2018","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ye et al., 2020] Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Heterogeneous few-shot model rec- tification with semantic mapping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3878- 3891, 2020.","title":"Heterogeneous few-shot model rectification with semantic mapping","authors":"Han-Jia Ye, Yuan De-Chuan Zhan, Zhi-Hua Jiang, Zhou","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Yu-Jie Zhang, Yu-Hu Yan, Peng Zhao, and Zhi-Hua Zhou. Towards enabling learnware to handle unseen jobs. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 10964-10972, 2021.","title":"Towards enabling learnware to handle unseen jobs","authors":"Yu-Jie Zhang, Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou","year":"2021","venue":"Proceedings of the 35th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2020] Peng Zhao, Le-Wen Cai, and Zhi-Hua Zhou. Handling concept drift via model reuse. Machine Learning, 109(3):533-568, 2020.","title":"Handling concept drift via model reuse","authors":"Peng Zhao, Le-Wen Cai, Zhi-Hua Zhou","year":"2020","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou and Tan, 2022] Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: Small models do big. arXiv:2210.03647, 2022.","title":"Learnware: Small models do big","authors":"Tan ; Zhi-Hua Zhou, Zhi-Hao Zhou, Tan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble methods: founda- tions and algorithms. CRC press, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou, 2016] Zhi-Hua Zhou. Learnware: On the future of machine learning. Frontiers of Computer Science, 10(4):589-590, 2016.","title":"Learnware: On the future of machine learning","authors":"Zhi-Hua Zhou","year":"2016","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}