{"type":"ScholarlyArticle","title":"Controllable Text Generation for Open-Domain Creativity and Fairness","authors":[{"name":"Nanyun (Violet) Peng","slug":"nanyun-violet-peng","given":"Nanyun (Violet)","family":"Peng","affiliation":"Computer Science Department , University of California , Los Angeles"}],"abstract":"Recent advances in large pre-trained language models have demonstrated strong results in generating natural languages and significantly improved performances for many natural language generation (NLG) applications such as machine translation and text summarization. However, when the generation tasks are more open-ended and the content is under-specified, existing techniques struggle to generate long-term coherent and creative content. Moreover, the models exhibit and even amplify social biases that are learned from the training corpora. This happens because the generation models are trained to capture the surface patterns (i.e. sequences of words), instead of capturing underlying semantics and discourse structures, as well as background knowledge including social norms. In this paper, I introduce our recent works on controllable text generation to enhance the creativity and fairness of language generation models. We explore hierarchical generation and constrained decoding, with applications to creative language generation including story, poetry, and figurative languages, and bias mitigation for generation models.","doi":"10.24963/ijcai.2022/818","url":"https://ijcai.org/proceedings/2022/818","pdf":"https://ijcai.org/proceedings/2022/0818.pdf","markdown":"https://ijcai.org/proceedings/2022/818.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/818","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Early Career","track":"Early Career","number":818,"pages":{"first":5821,"last":5825},"keywords":["EC: Natural Language Generation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987225","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985368","length_sec":895,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahmad et al., 2021] Wasi Ahmad, Nanyun Peng, and Kai- Wei Chang. Gate: Graph attention transformer encoder for cross-lingual relation and event extraction. In The Thirty- Fifth AAAI Conference on Artificial Intelligence (AAAI- 21), 2021.","title":"Gate: Graph attention transformer encoder for cross-lingual relation and event extraction","authors":"Wasi Ahmad, Nanyun Peng, Kai-Wei Chang","year":"2021","venue":"The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bender et al., 2021] E. M. Bender, T. Gebru, A. McMillan- Major, and S. Shmitchell. On the dangers of stochas- tic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2021.","title":"On the dangers of stochastic parrots: Can language models be too big?","authors":"E M Bender, T Gebru, A Mcmillan-Major, S Shmitchell","year":"2021","venue":"ACM Conference on Fairness, Accountability, and Transparency (FAccT)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blodgett et al., 2020] S. L. Blodgett, S. Barocas, H. Daume III, and H. Wallach. Language (technology) is power: A critical survey of \"bias\" in NLP. In Association for Com- putational Linguistics (ACL), 2020.","title":"Language (technology) is power: A critical survey of \"bias\" in NLP","authors":"S L Blodgett, S Barocas, H Daume, H Wallach","year":"2020","venue":"Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boudens, 2005] Connie J Boudens. The story of work: A narrative analysis of workplace emotion. Organization Studies, 26(9):1285-1306, 2005.","title":"The story of work: A narrative analysis of workplace emotion","authors":"Connie J Boudens","year":"2005","venue":"Organization Studies","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chakrabarty et al., 2020a] Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. r 3 : Re- verse, retrieve, and rank for sarcasm generation with com- monsense knowledge. In ACL, 2020.","title":"r 3 : Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge","authors":"Chakrabarty","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chakrabarty et al., 2020b] Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. Generating similes effort- lessly like a pro: A style transfer approach for simile gen- eration. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), 2020.","title":"Generating similes effortlessly like a pro: A style transfer approach for simile generation","authors":"Chakrabarty","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chakrabarty et al., 2021] Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. Mermaid: Metaphor generation with symbolism and discriminative decoding. In The 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.","title":"Mermaid: Metaphor generation with symbolism and discriminative decoding","authors":"Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, Nanyun Peng","year":"2021","venue":"The 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fan et al., 2019] Angela Fan, Mike Lewis, and Yann Dauphin. Strategies for structuring story generation. In ACL, 2019.","title":"Strategies for structuring story generation","authors":"Angela Fan, Mike Lewis, Yann Dauphin","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gibbs Jr, 1994] Raymond W Gibbs Jr. The poetics of mind: Figurative thought, language, and understanding. Cam- bridge University Press, 1994.","title":"The poetics of mind: Figurative thought, language, and understanding","authors":"Raymond W Gibbs","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goldfarb-Tarrant et al., 2020] Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. Content planning for neural story generation with aris- totelian rescoring. In the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), 2020.","title":"Content planning for neural story generation with aristotelian rescoring","authors":"Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, Nanyun Peng","year":"2020","venue":"the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2019] Rujun Han, Qiang Ning, and Nanyun Peng. Joint event and temporal relation extraction with shared representations and structured prediction. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 434-444, 2019.","title":"Joint event and temporal relation extraction with shared representations and structured prediction","authors":"Han","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2020] Rujun Han, Yichao Zhou, and Nanyun Peng. Domain knowledge empowered structured neural net for end-to-end event temporal relation extraction. In the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Compu- tational Linguistics, 2020.","title":"Domain knowledge empowered structured neural net for end-to-end event temporal relation extraction","authors":"Han","year":"2020","venue":"the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2021] Rujun Han, Xiang Ren, and Nanyun Peng. Econet: Effective continual pretraining of language mod- els for event temporal reasoning. In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.","title":"Econet: Effective continual pretraining of language models for event temporal reasoning","authors":"Han","year":"2021","venue":"The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Han et al., 2022] Rujun Han, Hong Chen, Yufei Tian, and Nanyun Peng. Go back in time: Generating flashbacks in stories with event plots and temporal prompts. In 2022 An- nual Conference of the North American Chapter of the As- sociation for Computational Linguistics (NAACL), 2022.","title":"Go back in time: Generating flashbacks in stories with event plots and temporal prompts","authors":"Han","year":"2022","venue":"2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2019] He He, Nanyun Peng, and Percy Liang. Pun generation with surprise. In 2019 Annual Confer- ence of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), volume 1, 2019.","title":"Pun generation with surprise","authors":"He","year":"2019","venue":"Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hovy and Spruit, 2016] D. Hovy and S. L. Spruit. The so- cial impact of natural language processing. In Association for Computational Linguistics (ACL), 2016.","title":"The social impact of natural language processing","authors":"D Hovy, S L Spruit","year":"2016","venue":"Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hsu et al., 2022a] I-Hung Hsu, Xiao Guo, Premkumar Natarajan, and Nanyun Peng. Discourse-level relation extraction via graph pooling. In The Thirty-Sixth AAAI Conference On Artificial Intelligence Workshop on Deep Learning on Graphs: Method and Applications (DLG- AAAI), 2022.","title":"Discourse-level relation extraction via graph pooling","authors":"Hsu","year":"2022","venue":"The Thirty-Sixth AAAI Conference On Artificial Intelligence Workshop on Deep Learning on Graphs: Method and Applications (DLG-AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hsu et al., 2022b] I-Hung Hsu, Kuan-Hao Huang, Eliza- beth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. Degree: A data-efficient generation-based event extraction model. In 2022 Annual Conference of the North American Chapter of the Associ- ation for Computational Linguistics (NAACL), 2022.","title":"Degree: A data-efficient generation-based event extraction model","authors":"Hsu","year":"2022","venue":"Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Huang et al., 2019] Xiao Huang, Li Dong, Elizabeth Boschee, and Nanyun Peng. Learning a unified named en- tity tagger from multiple partially annotated corpora for efficient adaptation. In The 2019 SIGNLL Conference on Computational Natural Language Learning (CoNLL), 2019.","title":"Learning a unified named entity tagger from multiple partially annotated corpora for efficient adaptation","authors":"Xiao Huang, Li Dong, Elizabeth Boschee, Nanyun Peng","year":"2019","venue":"The 2019 SIGNLL Conference on Computational Natural Language Learning (CoNLL)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Huang et al., 2021] Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. Document-level entity-based extraction as template generation. In The 2021 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), 2021.","title":"Document-level entity-based extraction as template generation","authors":"Huang","year":"2021","venue":"The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Justo et al., 2014] Raquel Justo, Thomas Corcoran, Stephanie M Lukin, Marilyn Walker, and M Inés Torres. Extracting relevant knowledge for the detection of sar- casm and nastiness in the social web. Knowledge-Based Systems, 69:124-133, 2014.","title":"Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web","authors":"Raquel Justo, Thomas Corcoran, Stephanie M Lukin, Marilyn Walker, M Inés Torres","year":"2014","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu and Peng, 2021] Sidi Lu and Nanyun Peng. On efficient training, controllability and compositional generalization of insertion-based language generators. arXiv preprint arXiv:2102.11008, 2021.","title":"On efficient training, controllability and compositional generalization of insertion-based language generators","authors":"Lu, Peng ; ] Sidi Lu, Nanyun Peng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Martin et al., 2017] Lara Martin, Prithviraj Ammanabrolu, William Hancock, Shruti Singh, Brent Harrison, and Mark Riedl. Event representations for automated story genera- tion with deep neural nets. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, 2017.","title":"Event representations for automated story generation with deep neural nets","authors":"Lara Martin, Prithviraj Ammanabrolu, William Hancock, Shruti Singh, Brent Harrison, Mark Riedl","year":"2017","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Meehan, 1976] James Richard Meehan. The metanovel: writing stories by computer. Technical report, YALE UNIV NEW HAVEN CONN DEPT OF COMPUTER SCIENCE, 1976.","title":"The metanovel: writing stories by computer","authors":"James Richard, Meehan","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mittal et al., 2022] Anirudh Mittal, Yufei Tian, and Nanyun Peng. Ambipun: Generating humorous puns with am- biguous context. In 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), short, 2022.","title":"Ambipun: Generating humorous puns with ambiguous context","authors":"Anirudh Mittal, Yufei Tian, Nanyun Peng","year":"2022","venue":"Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peng et al., 2017] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-Tau Yih. Cross- sentence n-ary relation extraction with graph LSTM. Transaction of the Association for Computational Linguis- tics (TACL), 2017.","title":"Crosssentence n-ary relation extraction with graph LSTM","authors":"Peng","year":"2017","venue":"Transaction of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shah et al., 2020] D. S. Shah, H. A. Schwartz, and D. Hovy. Predictive biases in natural language processing models: A conceptual framework and overview. In Association for Computational Linguistics (ACL), 2020.","title":"Predictive biases in natural language processing models: A conceptual framework and overview","authors":"D S Shah, H A Schwartz, D Hovy","year":"2020","venue":"Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sheng et al., 2019] Emily Sheng, Kai-Wei Chang, Premku- mar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), short, 2019.","title":"The woman worked as a babysitter: On biases in language generation","authors":"Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng","year":"2019","venue":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sheng et al., 2020] Emily Sheng, Kai-Wei Chang, Premku- mar Natarajan, and Nanyun Peng. Towards controllable biases in language generation. In the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)-Findings, long, 2020.","title":"Towards controllable biases in language generation","authors":"Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng","year":"2020","venue":"the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)-Findings","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sheng et al., 2021] Emily Sheng, Kai-Wei Chang, Premku- mar Natarajan, and Nanyun Peng. \"nice try, kiddo\": Ad hominems in dialogue systems. In The 2021 Annual Con- ference of the North American Chapter of the Association for Computational Linguistics (NAACL). Association for Computational Linguistics, 2021.","title":"nice try, kiddo\": Ad hominems in dialogue systems","authors":"Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng","year":"2021","venue":"The 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Stein et al., 2018] Elizabeth Stein, Robert A Pearlman, and Tyler P Tate. Using metaphors and figurative language to improve communication about patient suffering. Journal of Pain and Symptom Management, 56(6):e68-e69, 2018.","title":"Using metaphors and figurative language to improve communication about patient suffering","authors":"Elizabeth Stein, Robert A Pearlman, Tyler P Tate","year":"2018","venue":"Journal of Pain and Symptom Management","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Stowe et al., 2021] Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. Metaphor generation with conceptual mappings. In Pro- ceedings of the Conference of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.","title":"Metaphor generation with conceptual mappings","authors":"Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, Iryna Gurevych","year":"2021","venue":"Proceedings of the Conference of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tian and Peng, 2022] Yufei Tian and Nanyun Peng. Zero- shot sonnet generation with discourse-level coherence and poetic features. In 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022.","title":"Zeroshot sonnet generation with discourse-level coherence and poetic features","authors":"Yufei Tian, Nanyun Peng","year":"2022","venue":"Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yao et al., 2019] Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-And- Write: Towards better automatic storytelling. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 2019.","title":"Plan-And-Write: Towards better automatic storytelling","authors":"Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, Rui Yan","year":"2019","venue":"The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yoshimura et al., 2015] Eriko Yoshimura, Misako Imono, Seiji Tsuchiya, and Hirokazu Watabe. A simile recog- nition system using a commonsense sensory association method. Procedia Computer Science, 60:55-62, 2015.","title":"A simile recognition system using a commonsense sensory association method","authors":"Eriko Yoshimura, Misako Imono, Seiji Tsuchiya, Hirokazu Watabe","year":"2015","venue":"Procedia Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}