{"type":"ScholarlyArticle","title":"A Deep Generative Model for Code Switched Text","authors":[{"name":"Bidisha Samanta","slug":"bidisha-samanta","given":"Bidisha","family":"Samanta","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Sharmila Reddy","slug":"sharmila-reddy","given":"Sharmila","family":"Reddy","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Hussain Jagirdar","slug":"hussain-jagirdar","given":"Hussain","family":"Jagirdar","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Niloy Ganguly","slug":"niloy-ganguly","given":"Niloy","family":"Ganguly","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Soumen Chakrabarti","slug":"soumen-chakrabarti","given":"Soumen","family":"Chakrabarti","affiliation":"Indian Institute of Technology , Bombay"}],"abstract":"Code-switching, the interleaving of two or more languages within a sentence or discourse is pervasive in multilingual societies. Accurate language models for code-switched text are critical for NLP tasks. State-of-the-art data-intensive neural language models are difficult to train well from scarce language-labeled code-switched text. A potential solution is to use deep generative models to synthesize large volumes of realistic code-switched text. Although generative adversarial networks and variational autoencoders can synthesize plausible monolingual text from continuous latent space, they cannot adequately address code-switched text, owing to their informal style and complex interplay between the constituent languages. We introduce VACS, a novel variational autoencoder architecture specifically tailored to code-switching phenomena. VACS encodes to and decodes from a two-level hierarchical representation, which models syntactic contextual signals in the lower level, and language switching signals in the upper layer. Sampling representations from the prior and decoding them produced well-formed, diverse code-switched sentences. Extensive experiments show that using synthetic code-switched text with natural monolingual data results in significant (33.06\\%) drop in perplexity.","doi":"10.24963/ijcai.2019/719","url":"https://ijcai.org/proceedings/2019/719","pdf":"https://ijcai.org/proceedings/2019/0719.pdf","markdown":"https://ijcai.org/proceedings/2019/719.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/719","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":719,"pages":{"first":5175,"last":5181},"keywords":["Natural Language Processing: Tagging, chunking, and parsing","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baheti et al., 2017] Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. Curriculum design for code-switching: Experiments with language identifi- cation and language modeling with deep neural networks. Proceedings of ICON, 2017.","title":"Curriculum design for code-switching: Experiments with language identification and language modeling with deep neural networks","authors":"Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, Kalika Bali","year":"2017","venue":"Proceedings of ICON","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bidisha et al., 2019] Bidisha, Niloy Samanta, Soumen Gan- guly, and Chakrabarti. Improved sentiment detection via label transfer from monolingual to synthetic code- switched text. 2019.","title":"Improved sentiment detection via label transfer from monolingual to synthetic codeswitched text","authors":"Niloy Bidisha, Soumen Samanta, Chakrabarti Ganguly","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015a] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Bowman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bowman et al., 2015b] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, and Dai. Generating sentences from a con- tinuous space. arXiv preprint arXiv:1511.06349, 2015.","title":"Generating sentences from a continuous space","authors":"Bowman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown et al., 1992] Peter F Brown, Vincent J Della Pietra, Robert L Mercer, Stephen A Della Pietra, and Jennifer C Lai. An estimate of an upper bound for the entropy of english. Computational Linguistics, 18(1), 1992.","title":"An estimate of an upper bound for the entropy of english","authors":"Vincent J Della Peter F Brown, Robert L Pietra, Stephen A Mercer, Pietra Della, Jennifer C Lai","year":"1992","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chandu et al., 2018] Khyathi Chandu, Thomas Manzini, Sumeet Singh, and Alan W Black. Language informed modeling of code-switched text. In Proceedings of the Third Workshop on Computational Approaches to Linguis- tic Code-Switching, 2018.","title":"Language informed modeling of code-switched text","authors":"Chandu","year":"2018","venue":"Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chorowski et al., 2015] Jan K Chorowski, Dzmitry Bah- danau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In NIPS, 2015.","title":"Attention-based models for speech recognition","authors":"Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gambäck and Das, 2016] Björn Gambäck and Amitava Das. Comparing the level of code-switching in corpora. In LREC, 2016.","title":"Comparing the level of code-switching in corpora","authors":"Das Gambäck, Björn Gambäck, Amitava Das","year":"2016","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garg et al., 2018] Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. Code-switched language models using dual rnns and same-source pretraining. arXiv preprint arXiv:1809.01962, 2018.","title":"Code-switched language models using dual rnns and same-source pretraining","authors":"Saurabh Garg, Tanmay Parekh, Preethi Jyothi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guzmán et al., 2017] Gualberto A Guzmán, Joseph Ri- card, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. Metrics for modeling code- switching across corpora. In INTERSPEECH, 2017.","title":"Metrics for modeling codeswitching across corpora","authors":"Joseph Gualberto A Guzmán, Jacqueline Ricard, Barbara E Serigos, Bullock, Jacqueline Almeida, Toribio","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kannan and Vinyals, 2017] Anjuli Kannan and Oriol Vinyals. Adversarial evaluation of dialogue models. arXiv preprint arXiv:1701.08198, 2017.","title":"Adversarial evaluation of dialogue models","authors":"Anjuli Kannan, Oriol Vinyals","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2016] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In AAAI, 2016.","title":"Character-aware neural language models","authors":"Yoon Kim, Yacine Jernite, David Sontag, Alexander M Rush","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Muysken et al., 2000] Pieter Muysken, Carmen Pena Díaz, Pieter Cornelis Muysken, et al. Bilingual speech: A ty- pology of code-mixing, volume 11. Cambridge University Press, 2000.","title":"Bilingual speech: A typology of code-mixing","authors":"Pieter Muysken, Carmen Pena Díaz, Pieter Cornelis Muysken","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Myers-Scotton, 1997] Carol Myers-Scotton. Duelling lan- guages: Grammatical structure in codeswitching. Oxford University Press, 1997.","title":"Duelling languages: Grammatical structure in codeswitching","authors":"Carol Myers-Scotton","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Patro et al., 2017] Jasabanta Patro, Bidisha Samanta, Saurabh Singh, Abhipsa Basu, Prithwish Mukherjee, Monojit Choudhury, and Animesh Mukherjee. All that is English may be Hindi: Enhancing language identification through automatic ranking of the likeliness of word borrowing in social media. In EMNLP Conference, 2017.","title":"All that is English may be Hindi: Enhancing language identification through automatic ranking of the likeliness of word borrowing in social media","authors":"Patro","year":"2017","venue":"EMNLP Conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pratapa et al., 2018] Adithya Pratapa, Gayatri Bhat, Mono- jit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In ACL Con- ference, 2018.","title":"Language modeling for code-mixing: The role of linguistic theory based synthetic data","authors":"Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, Kalika Bali","year":"2018","venue":"ACL Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rezende et al., 2014] Danilo Jimenez Rezende, Shakir Mo- hamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.","title":"Stochastic backpropagation and approximate inference in deep generative models","authors":"Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rijhwani et al., 2017] Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. Estimating code-switching on twitter with a novel generalized word-level language detection tech- nique. In ACL Conference, volume 1, 2017.","title":"Estimating code-switching on twitter with a novel generalized word-level language detection technique","authors":"Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, Chandra Shekhar, Maddila","year":"2017","venue":"ACL Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rudra et al., 2016] Koustav Rudra, Shruti Rijhwani, Rafiya Begum, Kalika Bali, Monojit Choudhury, and Niloy Gan- guly. Understanding language preference for expression of opinion and sentiment: What do hindi-english speakers do on twitter? In EMNLP Conference, 2016.","title":"Understanding language preference for expression of opinion and sentiment: What do hindi-english speakers do on twitter?","authors":"Rudra","year":"2016","venue":"EMNLP Conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sønderby et al., 2016] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NIPS, 2016.","title":"Ladder variational autoencoders","authors":"Tapani Casper Kaae Sønderby, Lars Raiko, Søren Maaløe, Ole Kaae Sønderby, Winther","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In IEEE CVPR, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Winata et al., 2018] Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Code-switching lan- guage modeling using syntax-aware multi-task learning. arXiv preprint arXiv:1805.12070, 2018.","title":"Code-switching language modeling using syntax-aware multi-task learning","authors":"Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, Pascale Fung","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. arXiv preprint arXiv:1706.03850, 2017.","title":"Adversarial feature matching for text generation","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}