{"type":"ScholarlyArticle","title":"Time-evolving Text Classification with Deep Neural Networks","authors":[{"name":"Yu He","slug":"yu-he","given":"Yu","family":"He","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China; State Key Laboratory of Software Development Environment , Beihang University , China"},{"name":"Jianxin Li","slug":"jianxin-li","given":"Jianxin","family":"Li","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China; State Key Laboratory of Software Development Environment , Beihang University , China"},{"name":"Yangqiu Song","slug":"yangqiu-song","given":"Yangqiu","family":"Song","affiliation":"Department of Computer Science and Engineering , HKUST , Hong Kong"},{"name":"Mutian He","slug":"mutian-he","given":"Mutian","family":"He","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China; State Key Laboratory of Software Development Environment , Beihang University , China"},{"name":"Hao Peng","slug":"hao-peng","given":"Hao","family":"Peng","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China; State Key Laboratory of Software Development Environment , Beihang University , China"}],"abstract":"Traditional text classification algorithms are based on the assumption that data are independent and identically distributed. However, in most non-stationary scenarios, data may change smoothly due to long-term evolution and short-term fluctuation, which raises new challenges to traditional methods. In this paper, we present the first attempt to explore evolutionary neural network models for time-evolving text classification. We first introduce a simple way to extend arbitrary neural networks to evolutionary learning by using a temporal smoothness framework, and then propose a diachronic propagation framework to incorporate the historical impact into currently learned features through diachronic connections. Experiments on real-world news data demonstrate that our approaches greatly and consistently outperform traditional neural network models in both accuracy and stability.","doi":"10.24963/ijcai.2018/310","url":"https://ijcai.org/proceedings/2018/310","pdf":"https://ijcai.org/proceedings/2018/0310.pdf","markdown":"https://ijcai.org/proceedings/2018/310.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/310","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":310,"pages":{"first":2241,"last":2247},"keywords":["Machine Learning: Data Mining","Natural Language Processing: Text Classification","Machine Learning: Neural Networks","Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Allan et al., 1998] James Allan, Jaime G Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang. Topic detection and tracking pilot study final report. 1998.","title":"Topic detection and tracking pilot study final report","authors":"James Allan, Jaime G Carbonell, George Doddington, Jonathan Yamron, Yiming Yang","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atefeh and Khreich, 2015] Farzindar Atefeh and Wael Khreich. A survey of techniques for event detection in twitter. Computational Intelligence, 31(1):132-164, 2015.","title":"Farzindar Atefeh and Wael Khreich. A survey of techniques for event detection in twitter","authors":"Khreich Atefeh","year":"2015","venue":"Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chakrabarti et al., 2006] Deepayan Chakrabarti, Ravi Ku- mar, and Andrew Tomkins. Evolutionary clustering. In KDD, pages 554-560, 2006.","title":"Deepayan Chakrabarti, Ravi","authors":"Chakrabarti","year":"2006","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chi et al., 2007] Yun Chi, Xiaodan Song, Dengyong Zhou, Koji Hino, and Belle L Tseng. Evolutionary spectral clus- tering by incorporating temporal smoothness. In KDD, pages 153-162, 2007.","title":"Evolutionary spectral clustering by incorporating temporal smoothness","authors":"Yun Chi, Xiaodan Song, Dengyong Zhou, Koji Hino, Belle L Tseng","year":"2007","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crammer et al., 2006] Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online passive-aggressive algorithms. Journal of Machine Learn- ing Research, 7(Mar):551-585, 2006.","title":"Online passive-aggressive algorithms","authors":"Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, Yoram Singer","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gama et al., 2014] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM Computing Surveys, 46(4):44, 2014.","title":"A survey on concept drift adaptation","authors":"João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, Abdelhamid Bouchachia","year":"2014","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghazikhani et al., 2014] Adel Ghazikhani, Reza Monsefi, and Hadi Sadoghi Yazdi. Online neural network model for non-stationary and imbalanced data stream classification. International Journal of Machine Learning and Cybernet- ics, 5(1):51-62, 2014.","title":"Online neural network model for non-stationary and imbalanced data stream classification","authors":"Adel Ghazikhani, Reza Monsefi, Hadi Sadoghi Yazdi","year":"2014","venue":"International Journal of Machine Learning and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graves and Jaitly, 2014] Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, pages 1764-1772, 2014.","title":"Towards end-to-end speech recognition with recurrent neural networks","authors":"Alex Graves, Navdeep Jaitly","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In ICASSP, pages 6645-6649, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2009] Yangqing Jia, Shuicheng Yan, Changshui Zhang, et al. Semi-supervised classification on evolution- ary data. In IJCAI, pages 1083-1088, 2009.","title":"Semi-supervised classification on evolutionary data","authors":"Jia","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/183"},{"position":13,"raw":"[Kim and Hovy, 2006] Soo-Min Kim and Eduard Hovy. Ex- tracting opinions, opinion holders, and topics expressed in online news media text. In Proceedings of the Workshop on Sentiment and Subjectivity in Text, pages 1-8. Associa- tion for Computational Linguistics, 2006.","title":"Extracting opinions, opinion holders, and topics expressed in online news media text","authors":"Hovy ; Soo-Min Kim, Eduard Kim, Hovy","year":"2006","venue":"Proceedings of the Workshop on Sentiment and Subjectivity in Text","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267-2273, 2015.","title":"Recurrent convolutional neural networks for text classification","authors":"Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5(Apr):361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi- label text classification. In SIGIR, pages 115-124, 2017.","title":"Deep learning for extreme multilabel text classification","authors":"Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, Yiming Yang","year":"2017","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Masnadi-Shirazi and Vasconcelos, 2010] Hamed Masnadi- Shirazi and Nuno Vasconcelos. Risk minimization, proba- bility elicitation, and cost-sensitive svms. In ICML, pages 759-766, 2010.","title":"Risk minimization, probability elicitation, and cost-sensitive svms","authors":"Masnadi-Shirazi, Vasconcelos ; Hamed Masnadi-Shirazi, Nuno Vasconcelos","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ning et al., 2007] Huazhong Ning, Wei Xu, Yun Chi, Yi- hong Gong, and Thomas Huang. Incremental spectral clustering with application to monitoring of evolving blog communities. In SDM, pages 261-272, 2007.","title":"Incremental spectral clustering with application to monitoring of evolving blog communities","authors":"Ning","year":"2007","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sandhaus, 2008] Evan Sandhaus. The new york times anno- tated corpus ldc2008t19. In Linguistic Data Consortium. 2008.","title":"The new york times annotated corpus ldc2008t19","authors":"Evan Sandhaus","year":"2008","venue":"Linguistic Data Consortium","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2008] Lei Tang, Huan Liu, Jianping Zhang, and Zohreh Nazeri. Community evolution in dynamic multi- mode networks. In KDD, pages 677-685, 2008.","title":"Community evolution in dynamic multimode networks","authors":"Lei Tang, Huan Liu, Jianping Zhang, Zohreh Nazeri","year":"2008","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2015] Duyu Tang, Bing Qin, and Ting Liu. Doc- ument modeling with gated recurrent neural network for sentiment classification. In EMNLP, pages 1422-1432, 2015.","title":"Document modeling with gated recurrent neural network for sentiment classification","authors":"Tang","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2012] Lijun Wang, Manjeet Rege, Ming Dong, and Yongsheng Ding. Low-rank kernel matrix factoriza- tion for large-scale evolutionary clustering. IEEE Trans- actions on Knowledge and Data Engineering, 24(6):1036- 1050, 2012.","title":"Low-rank kernel matrix factorization for large-scale evolutionary clustering","authors":"Lijun Wang, Manjeet Rege, Ming Dong, Yongsheng Ding","year":"2012","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2014] Kevin S Xu, Mark Kliger, and Alfred O Hero Iii. Adaptive evolutionary clustering. Data Mining and Knowledge Discovery, 28(2):304-336, 2014.","title":"Adaptive evolutionary clustering","authors":"Kevin S Xu, Mark Kliger, Alfred O Hero Iii","year":"2014","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 1998] Yiming Yang, Tom Pierce, and Jaime Carbonell. A study of retrospective and on-line event de- tection. In SIGIR, pages 28-36, 1998.","title":"A study of retrospective and on-line event detection","authors":"Yiming Yang, Tom Pierce, Jaime Carbonell","year":"1998","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J. Smola, and Eduard H. Hovy. Hi- erarchical attention networks for document classification. In NAACL-HLT, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2009] Jianwen Zhang, Yangqiu Song, Gang Chen, and Changshui Zhang. On-line evolutionary expo- nential family mixture. In IJCAI, pages 1610-1615, 2009.","title":"On-line evolutionary exponential family mixture","authors":"Jianwen Zhang, Yangqiu Song, Gang Chen, Changshui Zhang","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/269"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}