{"type":"ScholarlyArticle","title":"Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention","authors":[{"name":"Yichong Xu","slug":"yichong-xu","given":"Yichong","family":"Xu","affiliation":"Microsoft Corporation"},{"name":"Chenguang Zhu","slug":"chenguang-zhu","given":"Chenguang","family":"Zhu","affiliation":"Microsoft Corporation"},{"name":"Shuohang Wang","slug":"shuohang-wang","given":"Shuohang","family":"Wang","affiliation":"Microsoft Corporation"},{"name":"Siqi Sun","slug":"siqi-sun","given":"Siqi","family":"Sun","affiliation":"Microsoft Corporation"},{"name":"Hao Cheng","slug":"hao-cheng","given":"Hao","family":"Cheng","affiliation":"Microsoft Corporation"},{"name":"Xiaodong Liu","slug":"xiaodong-liu","given":"Xiaodong","family":"Liu","affiliation":"Microsoft Corporation"},{"name":"Jianfeng Gao","slug":"jianfeng-gao","given":"Jianfeng","family":"Gao","affiliation":"Microsoft Corporation"},{"name":"Pengcheng He","slug":"pengcheng-he","given":"Pengcheng","family":"He","affiliation":"Microsoft Corporation"},{"name":"Michael Zeng","slug":"michael-zeng","given":"Michael","family":"Zeng","affiliation":"Microsoft Corporation"},{"name":"Xuedong Huang","slug":"xuedong-huang","given":"Xuedong","family":"Huang","affiliation":"Microsoft Corporation"}],"abstract":"Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear.\r\n By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems.\r\n We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. \r\n In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledgeable External Attention for commonsense Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9%.","doi":"10.24963/ijcai.2022/383","url":"https://ijcai.org/proceedings/2022/383","pdf":"https://ijcai.org/proceedings/2022/0383.pdf","markdown":"https://ijcai.org/proceedings/2022/383.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/383","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":383,"pages":{"first":2762,"last":2768},"keywords":["Natural Language Processing: Question Answering","Knowledge Representation and Reasoning: Reasoning about Knowledge and Belief","Knowledge Representation and Reasoning: Common-Sense Reasoning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987088","length_sec":96,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985231","length_sec":720,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bhakthavatsalam et al., 2020] Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660, 2020.","title":"Genericskb: A knowledge base of generic statements","authors":"Sumithra Bhakthavatsalam, Chloe Anastasiades, Peter Clark","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bommasani et al., 2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.","title":"On the opportunities and risks of foundation models","authors":"Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Von Arx, Jeannette Michael S Bernstein, Antoine Bohg, Emma Bosselut, Brunskill","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bosselut et al., 2019] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317, 2019.","title":"Comet: Commonsense transformers for automatic knowledge graph construction","authors":"Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom B Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.","title":"Language models are few-shot learners","authors":"Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2021] Ting-Yun Chang, Yang Liu, Karthik Gopalakr- ishnan, Behnam Hedayatnia, Pei Zhou, and Dilek Hakkani-Tur. Incorporating commonsense knowledge graph in pretrained mod- els for social commonsense tasks, 2021.","title":"Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks","authors":"Ting-Yun, Yang Chang, Karthik Liu, Behnam Gopalakrishnan, Pei Hedayatnia, Dilek Zhou, Hakkani-Tur","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clark et al., 2020] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Confer- ence on Learning Representations, 2020.","title":"Electra: Pre-training text encoders as discriminators rather than generators","authors":"Kevin Clark, Minh-Thang Luong, Quoc V Le, Christopher D Manning","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guu et al., 2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.","title":"REALM: Retrieval-augmented language model pre-training","authors":"Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2020] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentan- gled attention. In International Conference on Learning Repre- sentations, 2020.","title":"Deberta: Decoding-enhanced bert with disentangled attention","authors":"Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2021] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021.","title":"Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing","authors":"Pengcheng He, Jianfeng Gao, Weizhu Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2020] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, 2020.","title":"Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization","authors":"Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lan et al., 2019] Zhenzhong Lan, Mingda Chen, Sebastian Good- man, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2019] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for com- monsense reasoning. arXiv preprint arXiv:1909.02151, 2019.","title":"Kagnet: Knowledge-aware graph networks for commonsense reasoning","authors":"Xinyue Bill Yuchen Lin, Jamin Chen, Xiang Chen, Ren","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387, 2021.","title":"Generated knowledge prompting for commonsense reasoning","authors":"Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Le Ronan, Yejin Bras, Hannaneh Choi, Hajishirzi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hut- ter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mitra et al., 2019] Arindam Mitra, Pratyay Banerjee, Kuntal Ku- mar Pal, Swaroop Mishra, and Chitta Baral. How additional knowledge can improve natural language commonsense question answering? arXiv preprint arXiv:1909.08855, 2019.","title":"How additional knowledge can improve natural language commonsense question answering","authors":"Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal, Swaroop Mishra, Chitta Baral","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Patterson et al., 2021] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021.","title":"Carbon emissions and large neural network training","authors":"David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qin et al., 2019] Lianhui Qin, Michel Galley, Chris Brockett, Xi- aodong Liu, Xiang Gao, William B Dolan, Yejin Choi, and Jian- feng Gao. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5427-5436, 2019.","title":"Conversing by reading: Contentful neural conversation with on-demand machine reading","authors":"Qin","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raffel et al., 2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sap et al., 2019] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Bren- dan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035, 2019.","title":"Atomic: An atlas of machine commonsense for if-then reasoning","authors":"Sap","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schütze et al., 2008] Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval. Cam- bridge University Press Cambridge, 2008.","title":"Introduction to information retrieval","authors":"Schütze","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2021] Chenguang Song, Nianwen Ning, Yunlei Zhang, and Bin Wu. Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection. Neurocomputing, 462:88-100, 2021.","title":"Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection","authors":"Chenguang Song, Nianwen Ning, Yunlei Zhang, Bin Wu","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Speer et al., 2017] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In AAAI conference on artificial intelligence (AAAI), 2017.","title":"Conceptnet 5.5: An open multilingual graph of general knowledge","authors":"Speer","year":"2017","venue":"AAAI conference on artificial intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Talmor et al., 2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question an- swering challenge targeting commonsense knowledge. North American Chapter of the Association for Computational Linguis- tics (NAACL), 2019.","title":"Commonsenseqa: A question answering challenge targeting commonsense knowledge","authors":"Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant","year":"2019","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2019] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi- task benchmark and analysis platform for natural language under- standing. 2019. In the Proceedings of ICLR.","title":"GLUE: A multitask benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R Bowman","year":"2019","venue":"the Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2022] Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data. Association for Computational Linguistics (ACL), 2022.","title":"Training data is more valuable than you think: A simple and effective method by retrieving from training data","authors":"Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, Michael Zeng","year":"2022","venue":"Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2021] Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, and Xuedong Huang. Fusing context into knowledge graph for commonsense question answering. In Asso- ciation for Computational Linguistics (ACL), 2021.","title":"Fusing context into knowledge graph for commonsense question answering","authors":"Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, Xuedong Huang","year":"2021","venue":"Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2022] Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, and Huajun Chen. Kformer: Knowledge injection in transformer feed-forward layers. arXiv preprint arXiv:2201.05742, 2022.","title":"Kformer: Knowledge injection in transformer feed-forward layers","authors":"Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, Huajun Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yasunaga et al., 2021] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answer- ing. arXiv preprint arXiv:2104.06378, 2021.","title":"Qa-gnn: Reasoning with language models and knowledge graphs for question answering","authors":"Yasunaga","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2020] Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhit- ing Hu, Qingyun Wang, Heng Ji, and Meng Jiang. A sur- vey of knowledge-enhanced text generation. arXiv preprint arXiv:2010.04389, 2020.","title":"A survey of knowledge-enhanced text generation","authors":"Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Ji Heng, Meng Jiang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2021] Donghan Yu, Chenguang Zhu, Yuwei Fang, Wen- hao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. Kg-fid: Infusing knowledge graph in fusion- in-decoder for open-domain question answering. arXiv preprint arXiv:2110.04330, 2021.","title":"Kg-fid: Infusing knowledge graph in fusionin-decoder for open-domain question answering","authors":"Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}