{"type":"ScholarlyArticle","title":"Deep Context: A Neural Language Model for Large-scale Networked Documents","authors":[{"name":"Hao Wu","slug":"hao-wu","given":"Hao","family":"Wu","affiliation":"USC ISI"},{"name":"Kristina Lerman","slug":"kristina-lerman","given":"Kristina","family":"Lerman","affiliation":"USC ISI"}],"abstract":"We propose a scalable neural language model that leverages the links between documents to learn the deep context of documents. Our model, Deep Context Vector, takes advantage of distributed representations to exploit the word order in document sentences, as well as the semantic connections among linked documents in a document network. We evaluate our model on large-scale data collections that include Wikipedia pages, and scientific and legal citations networks. We demonstrate its effectiveness and efficiency on document classification and link prediction tasks.","doi":"10.24963/ijcai.2017/431","url":"https://ijcai.org/proceedings/2017/431","pdf":"https://ijcai.org/proceedings/2017/0431.pdf","markdown":"https://ijcai.org/proceedings/2017/431.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/431","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":431,"pages":{"first":3091,"last":3097},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Semantics","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Airoldi et al., 2009] Edoardo M Airoldi, David M Blei, Stephen E Fienberg, and Eric P Xing. Mixed membership stochastic blockmodels. In NIPS, pages 33-40, 2009.","title":"Mixed membership stochastic blockmodels","authors":"David M Edoardo M Airoldi, Stephen E Blei, Eric P Fienberg, Xing","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. JMLR, 3:1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. JMLR, 3:993-1022, 2003.","title":"Latent dirichlet allocation","authors":"Andrew Y David M Blei, Michael I Ng, Jordan","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang and Blei, 2009] Jonathan Chang and David M Blei. Relational topic models for document networks. In AIS- TATS, pages 81-88, 2009.","title":"Relational topic models for document networks","authors":"Chang, Blei ;, Jonathan Chang, David M Blei","year":"2009","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2015] Shiyu Chang, Wei Han, Jiliang Tang, Guo-Jun Qi, Charu C Aggarwal, and Thomas S Huang. Heterogeneous network embedding via deep architectures. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 119-128. ACM, 2015.","title":"Heterogeneous network embedding via deep architectures","authors":"Shiyu Chang, Wei Han, Jiliang Tang, Guo-Jun Qi, Charu C Aggarwal, Thomas S Huang","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dietz et al., 2007] Laura Dietz, Steffen Bickel, and Tobias Scheffer. Unsupervised prediction of citation influences. In Proceedings of ICML, pages 233-240. ACM, 2007.","title":"Unsupervised prediction of citation influences","authors":"Laura Dietz, Steffen Bickel, Tobias Scheffer","year":"2007","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Djuric et al., 2015] Nemanja Djuric, Hao Wu, Vladan Ra- dosavljevic, Mihajlo Grbovic, and Narayan Bhamidipati. Hierarchical neural language models for joint representa- tion of streaming documents and their content. In Proceed- ings of the 24th International Conference on World Wide Web, pages 248-255, 2015.","title":"Hierarchical neural language models for joint representation of streaming documents and their content","authors":"Nemanja Djuric, Hao Wu, Vladan Radosavljevic, Mihajlo Grbovic, Narayan Bhamidipati","year":"2015","venue":"Proceedings of the 24th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grover and Leskovec, 2016] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of SIGKDD, pages 855-864, 2016.","title":"node2vec: Scalable feature learning for networks","authors":"Leskovec Grover, Aditya Grover, Jure Leskovec","year":"2016","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hoff et al., 2002] Peter D Hoff, Adrian E Raftery, and Mark S Handcock. Latent space approaches to social net- work analysis. Journal of the american Statistical associ- ation, 97(460):1090-1098, 2002.","title":"Latent space approaches to social network analysis","authors":"Adrian E Peter D Hoff, Mark S Raftery, Handcock","year":"2002","venue":"Journal of the american Statistical association","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoffman et al., 2010] Matthew Hoffman, Francis R Bach, and David M Blei. Online learning for latent dirichlet al- location. In NIPS, pages 856-864, 2010.","title":"Online learning for latent dirichlet allocation","authors":"Matthew Hoffman, Francis R Bach, David M Blei","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hofmann, 1999] Thomas Hofmann. Probabilistic latent se- mantic indexing. In Proceedings of SIGIR, pages 50-57. ACM, 1999.","title":"Probabilistic latent semantic indexing","authors":"Thomas Hofmann","year":"1999","venue":"Proceedings of SIGIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kemp et al., 2004] Charles Kemp, Thomas L Griffiths, and Joshua B Tenenbaum. Discovering latent classes in rela- tional data. 2004.","title":"Discovering latent classes in relational data","authors":"Charles Kemp, Thomas L Griffiths, Joshua B Tenenbaum","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kiros et al., 2014] Ryan Kiros, Richard S Zemel, and Rus- lan Salakhutdinov. A multiplicative model for learn- ing distributed text-based attribute representations. arXiv preprint arXiv:1406.2710, 2014.","title":"A multiplicative model for learning distributed text-based attribute representations","authors":"Ryan Kiros, Richard S Zemel, Ruslan Salakhutdinov","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le and Mikolov, 2014] Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014.","title":"Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mccallum et al., 2005] Andrew Mccallum, Andrés Corrada-emmanuel, and Xuerui Wang. Topic and role discovery in social networks. In In IJCAI, 2005.","title":"Topic and role discovery in social networks","authors":"Andrew Mccallum, Andrés Corrada-Emmanuel, Xuerui Wang","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2005/222"},{"position":17,"raw":"[Mei et al., 2008] Qiaozhu Mei, Deng Cai, Duo Zhang, and ChengXiang Zhai. Topic modeling with network regular- ization. In Proceedings of WWW, pages 101-110. ACM, 2008.","title":"Topic modeling with network regularization","authors":"Mei","year":"2008","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih and Hinton, 2007] Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language mod- elling. In Proceedings of ICML, pages 641-648. ACM, 2007.","title":"Three new graphical models for statistical language modelling","authors":"Hinton Mnih, Andriy Mnih, Geoffrey Hinton","year":"2007","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mnih and Kavukcuoglu, 2013] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS, pages 2265-2273, 2013.","title":"Learning word embeddings efficiently with noise-contrastive estimation","authors":"Andriy Mnih, Koray Kavukcuoglu","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nallapati et al., 2008] Ramesh M Nallapati, Amr Ahmed, Eric P Xing, and William W Cohen. Joint latent topic models for text and citations. In Proceedings of SIGKDD, pages 542-550. ACM, 2008.","title":"Joint latent topic models for text and citations","authors":"Ramesh M Nallapati, Amr Ahmed, Eric P Xing, William W Cohen","year":"2008","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social rep- resentations. arXiv preprint arXiv:1403.6652, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christo- pher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In NIPS, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2008] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In Proceedings of SIGKDD, pages 990-998. ACM, 2008.","title":"Arnetminer: extraction and mining of academic social networks","authors":"Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, Zhong Su","year":"2008","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2015a] Jian Tang, Meng Qu, and Qiaozhu Mei. Pte: Predictive text embedding through large-scale hetero- geneous text networks. In Proceedings of SIGKDD, pages 1165-1174. ACM, 2015.","title":"Pte: Predictive text embedding through large-scale heterogeneous text networks","authors":"Tang","year":"2015","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang et al., 2015b] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large- scale information network embedding. In Proceedings of WWW, pages 1067-1077, 2015.","title":"Line: Largescale information network embedding","authors":"Tang","year":"2015","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Daixin Wang, Peng Cui, and Wenwu Zhu. Structural deep network embedding. In Proceed- ings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1225- 1234. ACM, 2016.","title":"Structural deep network embedding","authors":"Wang","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}