{"type":"ScholarlyArticle","title":"Alleviate Dataset Shift Problem in Fine-grained Entity Typing with Virtual Adversarial Training","authors":[{"name":"Haochen Shi","slug":"haochen-shi","given":"Haochen","family":"Shi","affiliation":"Zhejiang University"},{"name":"Siliang Tang","slug":"siliang-tang","given":"Siliang","family":"Tang","affiliation":"Zhejiang University"},{"name":"Xiaotao Gu","slug":"xiaotao-gu","given":"Xiaotao","family":"Gu","affiliation":"University of Illinois at Urbana-Champaign"},{"name":"Bo Chen","slug":"bo-chen","given":"Bo","family":"Chen","affiliation":"Zhejiang University"},{"name":"Zhigang Chen","slug":"zhigang-chen","given":"Zhigang","family":"Chen","affiliation":"State Key Laboratory of Cognitive Intelligence , Hefei , China"},{"name":"Jian Shao","slug":"jian-shao","given":"Jian","family":"Shao","affiliation":"Zhejiang University"},{"name":"Xiang Ren","slug":"xiang-ren","given":"Xiang","family":"Ren","affiliation":"University of Southern California {hcshi ,"}],"abstract":"The recent success of Distant Supervision (DS) brings abundant labeled data for the task of fine-grained entity typing (FET) without human annotation. However, the heuristically generated labels inevitably bring a significant distribution gap, namely dataset shift, between the distantly labeled training set and the manually curated test set. Considerable efforts have been made to alleviate this problem from the label perspective by either intelligently denoising the training labels, or designing noise-aware loss functions. Despite their progress, the dataset shift can hardly be eliminated completely. In this work, complementary to the label perspective, we reconsider this problem from the model perspective: Can we learn a more robust typing model with the existence of dataset shift? To this end, we propose a novel regularization module based on virtual adversarial training (VAT). The proposed approach first uses a self-paced sample selection function to select suitable samples for VAT, then constructs virtual adversarial perturbations based on the selected samples, and finally regularizes the model to be robust to such perturbations. Experiments on two benchmarks demonstrate the effectiveness of the proposed method, with an average 3.8%, 2.5%, and 3.2% improvement in accuracy, Macro F1 and Micro F1 respectively compared to the next best method.","doi":"10.24963/ijcai.2020/539","url":"https://ijcai.org/proceedings/2020/539","pdf":"https://ijcai.org/proceedings/2020/0539.pdf","markdown":"https://ijcai.org/proceedings/2020/539.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/539","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":539,"pages":{"first":3898,"last":3904},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Named Entities","Natural Language Processing: Information Extraction"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25359/file_25359.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2019] Bo Chen, Xiaotao Gu, Yufeng Hu, Sil- iang Tang, Guoping Hu, Yueting Zhuang, and Xiang Ren. Improving distantly-supervised entity typing with compact latent space clustering. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1, pages 2862-2872, 2019.","title":"Improving distantly-supervised entity typing with compact latent space clustering","authors":"Bo Chen, Xiaotao Gu, Yufeng Hu, Siliang Tang, Guoping Hu, Yueting Zhuang, Xiang Ren","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choi et al., 2018] Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics, pages 87-96, 2018.","title":"Ultra-fine entity typing","authors":"Choi","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2015] Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Deep learning for event-driven stock pre- diction. In Twenty-fourth international joint conference on artificial intelligence, 2015.","title":"Deep learning for event-driven stock prediction","authors":"Xiao Ding, Yue Zhang, Ting Liu, Junwen Duan","year":"2015","venue":"Twenty-fourth international joint conference on artificial intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/329"},{"position":5,"raw":"[Gillick et al., 2014] Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. Context- dependent fine-grained entity type tagging. arXiv preprint arXiv:1412.1820, 2014.","title":"Contextdependent fine-grained entity type tagging","authors":"Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, David Huynh","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Keskar et al., 2016] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.","title":"On large-batch training for deep learning: Generalization gap and sharp minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin and Ji, 2019] Ying Lin and Heng Ji. An attentive fine- grained entity typing model with latent type representa- tion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 6198- 6203, 2019.","title":"An attentive finegrained entity typing model with latent type representation","authors":"Ying Lin, Heng Ji","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ling and Weld, 2012] Xiao Ling and Daniel S Weld. Fine- grained entity recognition. In AAAI, volume 12, pages 94- 100, 2012.","title":"Finegrained entity recognition","authors":"Xiao Ling, Daniel S Weld","year":"2012","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Miyato et al., 2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993, 2018.","title":"Virtual adversarial training: a regularization method for supervised and semi-supervised learning","authors":"Miyato","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Moreno-Torres et al., 2012] Jose G Moreno-Torres, Troy Raeder, RocíO Alaiz-RodríGuez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521-530, 2012.","title":"A unifying view on dataset shift in classification","authors":"Troy Jose G Moreno-Torres, Rocío Raeder, Nitesh V Alaiz-Rodríguez, Francisco Chawla, Herrera","year":"2012","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Onoe and Durrett, 2019] Yasumasa Onoe and Greg Durrett. Learning to denoise distantly-labeled data for entity typ- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2407-2417, 2019.","title":"Learning to denoise distantly-labeled data for entity typing","authors":"Durrett Onoe, Yasumasa Onoe, Greg Durrett","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2016a] Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. Afet: Automatic fine- grained entity typing by hierarchical partial-label embed- ding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1369- 1378, 2016.","title":"Afet: Automatic finegrained entity typing by hierarchical partial-label embedding","authors":"Ren","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2016b] Xiang Ren, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, and Jiawei Han. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 1825-1834. ACM, 2016.","title":"Label noise reduction in entity typing by heterogeneous partial-label embedding","authors":"Ren","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xin et al., 2018] Ji Xin, Hao Zhu, Xu Han, Zhiyuan Liu, and Maosong Sun. Put it back: Entity typing with lan- guage model enhancement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 993-998, 2018.","title":"Put it back: Entity typing with language model enhancement","authors":"Ji Xin, Hao Zhu, Xu Han, Zhiyuan Liu, Maosong Sun","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xiong et al., 2019] Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Imposing label-relational inductive bias for ex- tremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1, pages 773-784, 2019.","title":"Imposing label-relational inductive bias for extremely fine-grained entity typing","authors":"Xiong","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu and Barbosa, 2018] Peng Xu and Denilson Bar- bosa. Neural fine-grained entity type classification with hierarchy-aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume 1, pages 16-25, 2018.","title":"Neural fine-grained entity type classification with hierarchy-aware loss","authors":"Peng Xu, Denilson Barbosa","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2019] Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, and Xiang Ren. Learning dynamic context augmentation for global entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 271-281, 2019.","title":"Learning dynamic context augmentation for global entity linking","authors":"Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, Xiang Ren","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yogatama et al., 2015] Dani Yogatama, Daniel Gillick, and Nevena Lazic. Embedding methods for fine grained en- tity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguis- tics and the 7th International Joint Conference on Natural Language Processing, volume 2, pages 291-296, 2015.","title":"Embedding methods for fine grained entity type classification","authors":"Dani Yogatama, Daniel Gillick, Nevena Lazic","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}