{"type":"ScholarlyArticle","title":"RDPD: Rich Data Helps Poor Data via Imitation","authors":[{"name":"Shenda Hong","slug":"shenda-hong","given":"Shenda","family":"Hong","affiliation":"School of Electronics Engineering and Computer Science , Peking University , China; Key Laboratory of Machine Perception (Ministry of Education) , Peking University , China; Department of Computational Science and Engineering , Georgia Institute of Technology , USA"},{"name":"Cao Xiao","slug":"cao-xiao","given":"Cao","family":"Xiao","affiliation":"Analytics Center of Excellence , IQVIA , USA"},{"name":"Trong Nghia Hoang","slug":"trong-nghia-hoang","given":"Trong Nghia","family":"Hoang","affiliation":"IBM Research , USA"},{"name":"Tengfei Ma","slug":"tengfei-ma","given":"Tengfei","family":"Ma","affiliation":"IBM Research , USA"},{"name":"Hongyan Li","slug":"hongyan-li","given":"Hongyan","family":"Li","affiliation":"School of Electronics Engineering and Computer Science , Peking University , China; Key Laboratory of Machine Perception (Ministry of Education) , Peking University , China"},{"name":"Jimeng Sun","slug":"jimeng-sun","given":"Jimeng","family":"Sun","affiliation":"Department of Computational Science and Engineering , Georgia Institute of Technology , USA"}],"abstract":"In many situations, we need to build and deploy separate models in related environments with different data qualities. For example, an environment with strong observation equipments (e.g., intensive care units) often provides high-quality multi-modal data, which are acquired from multiple sensory devices and have rich-feature representations. On the other hand, an environment with poor observation equipment (e.g., at home) only provides low-quality, uni-modal data with poor-feature representations. To deploy a competitive model in a poor-data environment without requiring direct access to multi-modal data acquired from a rich-data environment, this paper develops and presents a knowledge distillation (KD) method (RDPD) to enhance a predictive model trained on poor data using knowledge distilled from a high-complexity model trained on rich, private data. We evaluated RDPD on three real-world datasets and shown that its distilled model consistently outperformed all baselines across all datasets, especially achieving the greatest performance improvement over a model trained only on low-quality data by 24.56% on PR-AUC and 12.21% on ROC-AUC, and over that of a state-of-the-art KD model by 5.91% on PR-AUC and 4.44% on ROC-AUC.","doi":"10.24963/ijcai.2019/817","url":"https://ijcai.org/proceedings/2019/817","pdf":"https://ijcai.org/proceedings/2019/0817.pdf","markdown":"https://ijcai.org/proceedings/2019/817.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/817","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":817,"pages":{"first":5895,"last":5901},"keywords":["Special Track on AI for Improving Human-Well Being: Health applications (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654-2662, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bousmalis et al., 2016] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Du- mitru Erhan. Domain separation networks. In NIPS, pages 343-351, USA, 2016.","title":"Domain separation networks","authors":"Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, Dumitru Erhan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Curran Associates Inc. [Bousseljot et al., 1995]","title":null,"authors":"Curran Associates, Inc [ Bousseljot","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"R Bousseljot, D Kreiseler, and A Schnabel. Nutzung der ekg-signaldatenbank cardio- dat der ptb über das internet. Biomedizinische Tech- nik/Biomedical Engineering, 40(s1):317-318, 1995.","title":"Nutzung der ekg-signaldatenbank cardiodat der ptb über das internet","authors":"Bousseljot, Kreiseler, Schnabel","year":"1995","venue":"Biomedizinische Technik/Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2012] Minmin Chen, Zhixiang Xu, Kilian Q. Weinberger, and Fei Sha. Marginalized denoising autoen- coders for domain adaptation. In ICML, pages 1627-1634, USA, 2012. Omnipress.","title":"Marginalized denoising autoencoders for domain adaptation","authors":"Minmin Chen, Zhixiang Xu, Kilian Q Weinberger, Fei Sha","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi et al., 2016] Keunwoo Choi, George Fazekas, Mark Sandler, and Kyunghyun Cho. Convolutional recurrent neural networks for music classification. arXiv preprint arXiv:1609.04243, 2016.","title":"Convolutional recurrent neural networks for music classification","authors":"Choi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Davis and Goadrich, 2006] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240. ACM, 2006.","title":"The relationship between precision-recall and roc curves","authors":"Goadrich Davis, Jesse Davis, Mark Goadrich","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frosst and Hinton, 2017] Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784, 2017.","title":"Distilling a neural network into a soft decision tree","authors":"Hinton Frosst, Nicholas Frosst, Geoffrey Hinton","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17(1):2096-2030, January 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sen- timent classification: A deep learning approach. In ICML, pages 513-520, USA, 2011. Omnipress.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hong et al., 2017] Shenda Hong, Meng Wu, Yuxi Zhou, Qingyun Wang, Junyuan Shang, Hongyan Li, and Junqing Xie. ENCASE: an ENsemble ClASsifiEr for ECG Classi- fication Using Expert Features and Deep Neural Networks. Computing in Cardiology, 44:1, 2017.","title":"ENCASE: an ENsemble ClASsifiEr for ECG Classification Using Expert Features and Deep Neural Networks","authors":"Shenda Hong, Meng Wu, Yuxi Zhou, Qingyun Wang, Junyuan Shang, Hongyan Li, Junqing Xie","year":"2017","venue":"Computing in Cardiology","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang and Wang, 2017] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selec- tivity transfer. arXiv preprint arXiv:1707.01219, 2017.","title":"Like what you like: Knowledge distill via neuron selectivity transfer","authors":"Wang ; Huang, Zehao Huang, Naiyan Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G Mark. Mimic-iii, a freely ac- cessible critical care database. Scientific data, 3(1), 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, Li-Wei H Shen, Mengling Lehman, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Ba Kingma, Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Attention transfer from web images for video recognition. In Proceedings of the 2017 ACM on Multimedia Conference, pages 1-9. ACM, 2017.","title":"Attention transfer from web images for video recognition","authors":"Li","year":"2017","venue":"Proceedings of the 2017 ACM on Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2018] Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. Hierarchical attention transfer network for cross- domain sentiment classification. In AAAI 2018, New Or- leans, Lousiana, USA, February 2-7, 2018, 2018.","title":"Hierarchical attention transfer network for crossdomain sentiment classification","authors":"Zheng Li, Ying Wei, Yu Zhang, Qiang Yang","year":"2018","venue":"AAAI 2018","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lopez-Paz et al., 2015] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying dis- tillation and privileged information. ICLR, 2015.","title":"Unifying distillation and privileged information","authors":"David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ordóñez and Roggen, 2016] Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recog- nition. Sensors, 16(1):115, 2016.","title":"Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition","authors":"Roggen ; Ordóñez, Javier Francisco, Daniel Ordóñez, Roggen","year":"2016","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radosavovic et al., 2017] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. arXiv preprint arXiv:1712.04440, 2017.","title":"Data distillation: Towards omni-supervised learning","authors":"Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reiss and Stricker, 2012] Attila Reiss and Didier Stricker. Creating and benchmarking a new dataset for physical ac- tivity monitoring. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assis- tive Environments, page 40. ACM, 2012.","title":"Creating and benchmarking a new dataset for physical activity monitoring","authors":"Stricker Reiss, Attila Reiss, Didier Stricker","year":"2012","venue":"Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salehinejad et al., 2018] Hojjat Salehinejad, Joseph Barfett, Shahrokh Valaee, and Timothy Dowdell. Training neural networks with very little data -A draft. ICASSP, 2018.","title":"Training neural networks with very little data -A draft","authors":"Salehinejad","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sau and Balasubramanian, 2016] Bharat Bhusan Sau and Vineeth N Balasubramanian. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650, 2016.","title":"Deep model compression: Distilling knowledge from noisy teachers","authors":"Bharat Bhusan, Sau Vineeth N Balasubramanian","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiao et al., 2018] Cao Xiao, Edward Choi, and Jimeng Sun. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics As- sociation, 25(10):1419-1428, jun 2018.","title":"Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review","authors":"Cao Xiao, Edward Choi, Jimeng Sun","year":"2018","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yim et al., 2017] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.","title":"A gift from knowledge distillation: Fast optimization, network minimization and transfer learning","authors":"Yim","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zagoruyko and Komodakis, 2017] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2016] Guangyou Zhou, Zhiwen Xie, Xiangji Huang, and Tingting He. Bi-transferring deep neural net- works for domain adaptation. In ACL, 2016.","title":"Bi-transferring deep neural networks for domain adaptation","authors":"Guangyou Zhou, Zhiwen Xie, Xiangji Huang, Tingting He","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}