{"type":"ScholarlyArticle","title":"Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data","authors":[{"name":"Qing Xu","slug":"qing-xu","given":"Qing","family":"Xu","affiliation":"Institute for Infocomm Research , A; Nanyang Technological University , Singapore"},{"name":"Min Wu","slug":"min-wu","given":"Min","family":"Wu","affiliation":"Institute for Infocomm Research , A"},{"name":"Xiaoli Li","slug":"xiaoli-li","given":"Xiaoli","family":"Li","affiliation":"Institute for Infocomm Research , A; Centre for Frontier AI Research , A *STAR , Singapore"},{"name":"Kezhi Mao","slug":"kezhi-mao","given":"Kezhi","family":"Mao","affiliation":"Nanyang Technological University , Singapore; *STAR , Singapore"},{"name":"Zhenghua Chen","slug":"zhenghua-chen","given":"Zhenghua","family":"Chen","affiliation":"Institute for Infocomm Research , A; Centre for Frontier AI Research , A *STAR , Singapore"}],"abstract":"For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called UNiversal and joInt Knowledge Distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher’s and student’s representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks. The source code is available at https://github.com/ijcai2023/UNI KD.","doi":"10.24963/ijcai.2023/496","url":"https://ijcai.org/proceedings/2023/496","pdf":"https://ijcai.org/proceedings/2023/0496.pdf","markdown":"https://ijcai.org/proceedings/2023/496.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/496","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":496,"pages":{"first":4460,"last":4468},"keywords":["Machine Learning: ML: Multi-task and transfer learning","Machine Learning: ML: Time series and data streams","Machine Learning: ML: Unsupervised learning"],"videos":[],"references":[{"position":1,"raw":"[Anguita et al., 2013] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Or- tiz. A public domain dataset for human activity recogni- tion using smartphones. In Proceedings of the 21th in- ternational European symposium on artificial neural net- works, computational intelligence and machine learning, pages 437-442, 2013.","title":"A public domain dataset for human activity recognition using smartphones","authors":"Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, Jorge Luis, Reyes Ortiz","year":"2013","venue":"Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning effi- cient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.","title":"Learning efficient object detection models with knowledge distillation","authors":"Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, Manmohan Chandraker","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian-Sheng Hua. Homm: Higher-order moment matching for unsuper- vised domain adaptation. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 3422- 3429, 2020.","title":"Homm: Higher-order moment matching for unsupervised domain adaptation","authors":"Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, Xian-Sheng Hua","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2020] Inseop Chung, SeongUk Park, Jangho Kim, and Nojun Kwak. Feature-map-level online adver- sarial knowledge distillation. In International Conference on Machine Learning, pages 2006-2015. PMLR, 2020.","title":"Feature-map-level online adversarial knowledge distillation","authors":"Chung","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eldele et al., 2021] Emadeldeen Eldele, Zhenghua Chen, Chengyu Liu, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. An attention-based deep learning ap- proach for sleep stage classification with single-channel eeg. IEEE Transactions on Neural Systems and Rehabili- tation Engineering, 29:809-818, 2021.","title":"An attention-based deep learning approach for sleep stage classification with single-channel eeg","authors":"Eldele","year":"2021","venue":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elsken et al., 2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997- 2017, 2019.","title":"Neural architecture search: A survey","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The jour- nal of machine learning research, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2019] Liang Gao, Haibo Mi, Boqing Zhu, Dawei Feng, Yicong Li, and Yuxing Peng. An adversarial feature distillation method for audio classification. IEEE Access, 7:105319-105330, 2019.","title":"An adversarial feature distillation method for audio classification","authors":"Liang Gao, Haibo Mi, Boqing Zhu, Dawei Feng, Yicong Li, Yuxing Peng","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldberger et al., 2000] Ary L Goldberger, Luis AN Ama- ral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new re- search resource for complex physiologic signals. circu- lation, 101(23):e215-e220, 2000.","title":"Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals","authors":"Ary L Goldberger, A N Luis, Leon Amaral, Jeffrey M Glass, Plamen Hausdorff, Roger G Ch Ivanov, Joseph E Mark, George B Mietus, Chung-Kang Moody, H Eugene Peng, Stanley","year":"2000","venue":"circulation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819, 2021.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Granger et al., 2020] Eric Granger, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin, et al. Joint progressive knowledge distillation and unsupervised domain adapta- tion. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.","title":"Joint progressive knowledge distillation and unsupervised domain adaptation","authors":"Eric Granger, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin","year":"2020","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Work- shop, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"NIPS Deep Learning and Representation Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2021] Zhen Huang, Xu Shen, Jun Xing, Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-Sheng Hua. Revisiting knowl- edge distillation: An inheritance and exploration frame- work. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3579- 3588, 2021.","title":"Revisiting knowledge distillation: An inheritance and exploration framework","authors":"Zhen Huang, Xu Shen, Jun Xing, Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, Xian-Sheng Hua","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lessmeier et al., 2016] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, volume 3, 2016.","title":"Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification","authors":"Lessmeier","year":"2016","venue":"PHM Society European Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2021] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantiza- tion for deep neural network acceleration: A survey. Neu- rocomputing, 461:370-403, 2021.","title":"Pruning and quantization for deep neural network acceleration: A survey","authors":"Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, Xiaotong Zhang","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu and Xue, 2021] Qiao Liu and Hui Xue. Adversarial spectral kernel matching for unsupervised time series do- main adaptation. In IJCAI, pages 2744-2750, 2021.","title":"Adversarial spectral kernel matching for unsupervised time series domain adaptation","authors":"Qiao Liu, Hui Xue","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/378"},{"position":17,"raw":"[Liu et al., 2021] Li Liu, Qingle Huang, Sihao Lin, Hong- wei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for diversity- preserved knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8271-8280, 2021.","title":"Exploring inter-channel correlation for diversitypreserved knowledge distillation","authors":"Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, Xiaodan Liang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial do- main adaptation. Advances in neural information process- ing systems, 31, 2018.","title":"Conditional adversarial domain adaptation","authors":"Mingsheng Long, Zhangjie Cao, Jianmin Wang, Michael I Jordan","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maroto et al., 2022] Javier Maroto, Guillermo Ortiz- Jiménez, and Pascal Frossard. On the benefits of knowledge distillation for adversarial robustness. arXiv preprint arXiv:2203.07159, 2022.","title":"On the benefits of knowledge distillation for adversarial robustness","authors":"Javier Maroto, Guillermo Ortiz-Jiménez, Pascal Frossard","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Passalis and Tefas, 2018] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilis- tic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pages 268-284, 2018.","title":"Learning deep representations with probabilistic knowledge transfer","authors":"Nikolaos Passalis, Anastasios Tefas","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ragab et al., 2023] Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Adatime: A benchmarking suite for domain adaptation on time series data. ACM Trans. Knowl. Discov. Data, mar 2023.","title":"Adatime: A benchmarking suite for domain adaptation on time series data","authors":"Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li","year":"2023","venue":"ACM Trans. Knowl. Discov. Data","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rahman et al., 2020] Mohammad Mahfujur Rahman, Clin- ton Fookes, Mahsa Baktashmotlagh, and Sridha Sridha- ran. On minimum discrepancy estimation for deep domain adaptation. In Domain Adaptation for Visual Understand- ing, pages 81-94. Springer, 2020.","title":"On minimum discrepancy estimation for deep domain adaptation","authors":"Clinton Mohammad Mahfujur Rahman, Mahsa Fookes, Sridha Baktashmotlagh, Sridharan","year":"2020","venue":"Domain Adaptation for Visual Understanding","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ryu et al., 2022] Minho Ryu, Geonseok Lee, and Kichun Lee. Knowledge distillation for bert unsupervised do- main adaptation. Knowledge and Information Systems, 64(11):3113-3128, 2022.","title":"Knowledge distillation for bert unsupervised domain adaptation","authors":"Ryu","year":"2022","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shu et al., 2018] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised do- main adaptation. arXiv preprint arXiv:1802.08735, 2018.","title":"A dirt-t approach to unsupervised domain adaptation","authors":"Rui Shu, Hung H Bui, Hirokazu Narui, Stefano Ermon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Stisen et al., 2015] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjær- gaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmo- bile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems, pages 127-140, 2015.","title":"Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition","authors":"Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, Mads Møller Jensen","year":"2015","venue":"Proceedings of the 13th ACM conference on embedded networked sensor systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun and Saenko, 2016] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adap- tation. In European conference on computer vision, pages 443-450. Springer, 2016.","title":"Deep coral: Correlation alignment for deep domain adaptation","authors":"Saenko ; Sun, Baochen Sun, Kate Saenko","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.","title":"Deep domain confusion: Maximizing for domain invariance","authors":"Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2019] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pattern recognition letters, 119:3-11, 2019.","title":"Deep learning for sensor-based activity recognition: A survey","authors":"Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, Lisha Hu","year":"2019","venue":"Pattern recognition letters","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wilson et al., 2020] Garrett Wilson, Janardhan Rao Doppa, and Diane J Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In Pro- ceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, pages 1768-1778, 2020.","title":"Multi-source deep domain adaptation with weak supervision for time-series sensor data","authors":"Garrett Wilson, Janardhan Rao Doppa, Diane J Cook","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2021] Qing Xu, Zhenghua Chen, Keyu Wu, Chao Wang, Min Wu, and Xiaoli Li. Kdnet-rul: A knowledge distillation framework to compress deep neural networks for machine remaining useful life prediction. IEEE Trans- actions on Industrial Electronics, 69(2):2022-2032, 2021.","title":"Kdnet-rul: A knowledge distillation framework to compress deep neural networks for machine remaining useful life prediction","authors":"Qing Xu, Zhenghua Chen, Keyu Wu, Chao Wang, Min Wu, Xiaoli Li","year":"2021","venue":"IEEE Transactions on Industrial Electronics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2022] Qing Xu, Zhenghua Chen, Mohamed Ragab, Chao Wang, Min Wu, and Xiaoli Li. Contrastive adversarial knowledge distillation for deep model com- pression in time-series regression tasks. Neurocomputing, 485:242-251, 2022.","title":"Contrastive adversarial knowledge distillation for deep model compression in time-series regression tasks","authors":"Qing Xu, Zhenghua Chen, Mohamed Ragab, Chao Wang, Min Wu, Xiaoli Li","year":"2022","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2020] Jianfei Yang, Han Zou, Shuxin Cao, Zhenghua Chen, and Lihua Xie. Mobileda: Toward edge- domain adaptation. IEEE Internet of Things Journal, 7(8):6909-6918, 2020.","title":"Mobileda: Toward edgedomain adaptation","authors":"Jianfei Yang, Han Zou, Shuxin Cao, Zhenghua Chen, Lihua Xie","year":"2020","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2019] Rui Zhao, Ruqiang Yan, Zhenghua Chen, Kezhi Mao, Peng Wang, and Robert X Gao. Deep learning and its applications to machine health monitoring. Me- chanical Systems and Signal Processing, 115:213-237, 2019.","title":"Deep learning and its applications to machine health monitoring","authors":"Rui Zhao, Ruqiang Yan, Zhenghua Chen, Kezhi Mao, Peng Wang, Robert X Gao","year":"2019","venue":"Mechanical Systems and Signal Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}