{"type":"ScholarlyArticle","title":"COOL, a Context Outlooker, and Its Application to Question Answering and Other Natural Language Processing Tasks","authors":[{"name":"Fangyi Zhu","slug":"fangyi-zhu","given":"Fangyi","family":"Zhu","affiliation":"National University of Singapore"},{"name":"See-Kiong Ng","slug":"see-kiong-ng","given":"See Kiong","family":"Ng","affiliation":"National University of Singapore"},{"name":"Stéphane Bressan","slug":"stephane-bressan","given":"Stéphane","family":"Bressan","affiliation":"National University of Singapore"}],"abstract":"Vision outlooker improves the performance of vision transformers, which implements a self-attention mechanism by adding an outlook attention, a form of local attention.  \r\n\r\nIn natural language processing, as has been the case in computer vision and other domains, transformer-based models constitute the state-of-the-art for most processing tasks. In this domain, too, many authors have argued and demonstrated the importance of local context.\r\n\r\nWe present an outlook attention mechanism, COOL, for natural language processing. COOL, added on top of the self-attention layers of a transformer-based model, encodes local syntactic context considering word proximity and more pair-wise constraints than dynamic convolution used by existing approaches.\r\n\r\nA comparative empirical performance evaluation of an implementation of COOL with different transformer-based models confirms the opportunity for improvement over a baseline using the original models alone for various natural language processing tasks, including question answering. The proposed approach achieves competitive performance with existing state-of-the-art methods on some tasks.","doi":"10.24963/ijcai.2023/590","url":"https://ijcai.org/proceedings/2023/590","pdf":"https://ijcai.org/proceedings/2023/0590.pdf","markdown":"https://ijcai.org/proceedings/2023/590.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/590","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":590,"pages":{"first":5314,"last":5322},"keywords":["Natural Language Processing: NLP: Question answering","Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.","title":"METEOR: An automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bentivogli et al., 2009] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL rec- ognizing textual entailment challenge. In TAC, 2009.","title":"The fifth PASCAL recognizing textual entailment challenge","authors":"Luisa Bentivogli, Peter Clark, Ido Dagan, Danilo Giampiccolo","year":"2009","venue":"TAC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cer et al., 2017] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semantic textual similarity-multilingual and cross-lingual focused evalua- tion. arXiv preprint arXiv:1708.00055, 2017.","title":"Semantic textual similarity-multilingual and cross-lingual focused evaluation","authors":"Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2016] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016.","title":"Long short-term memory-networks for machine reading","authors":"Cheng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"the Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dolan and Brockett, 2005] William B. Dolan and Chris Brockett. Automatically constructing a corpus of senten- tial paraphrases. In the Workshop on Paraphrasing, 2005.","title":"Automatically constructing a corpus of sentential paraphrases","authors":"William B Dolan, Chris Brockett","year":"2005","venue":"the Workshop on Paraphrasing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2018] Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: A no-recurrence sequence-to- sequence model for speech recognition. In the Conference on Acoustics, Speech and Signal Processing, 2018.","title":"Speech-transformer: A no-recurrence sequence-tosequence model for speech recognition","authors":"Dong","year":"2018","venue":"the Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In The Ninth Inter- national Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"The Ninth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In the Conference on Artificial Intelligence and Statistics, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"the Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gulati et al., 2020] Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruom- ing Pang. Conformer: Convolution-augmented trans- former for speech recognition. In Proc. Interspeech, 2020.","title":"Conformer: Convolution-augmented transformer for speech recognition","authors":"Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang","year":"2020","venue":"Proc. Interspeech","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pat- tern Recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- enhanced BERT with disentangled attention. ArXiv, abs/2006.03654, 2021.","title":"DeBERTa: Decodingenhanced BERT with disentangled attention","authors":"Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hofstätter et al., 2020] Sebastian Hofstätter, Hamed Za- mani, Bhaskar Mitra, Nick Craswell, and Allan Hanbury. Local self-attention over long text for efficient document retrieval. In the 43rd International Conference on Re- search and Development in Information Retrieval, 2020.","title":"Local self-attention over long text for efficient document retrieval","authors":"Sebastian Hofstätter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, Allan Hanbury","year":"2020","venue":"the 43rd International Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2020] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference, 2020.","title":"Heterogeneous graph transformer","authors":"Hu","year":"2020","venue":"Proceedings of The Web Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang et al., 2020] Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. ConvBERT: Improving BERT with span-based dynamic convolution. In Advances in Neural Information Process- ing Systems, 2020.","title":"ConvBERT: Improving BERT with span-based dynamic convolution","authors":"Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kalchbrenner et al., 2014] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In the Association for Computational Linguistics, 2014.","title":"A convolutional neural network for modelling sentences","authors":"Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom","year":"2014","venue":"the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karpov et al., 2020] Pavel Karpov, Guillaume Godin, and Igor V Tetko. Transformer-CNN: Swiss knife for qsar modeling and interpretation. Journal of Cheminformatics, 2020.","title":"Transformer-CNN: Swiss knife for qsar modeling and interpretation","authors":"Karpov","year":"2020","venue":"Journal of Cheminformatics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lan et al., 2019] Zhenzhong Lan, Mingda Chen, Sebas- tian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.","title":"Albert: A lite BERT for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021a] Yun Liu, Guolei Sun, Yu Qiu, Le Zhang, Ajad Chhatkuli, and Luc Van Gool. Transformer in convolutional neural networks. arXiv preprint arXiv:2106.03180, 2021.","title":"Transformer in convolutional neural networks","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In the Conference on Learning Representations, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"the Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In the Association for Computational Linguistics, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Michael Sejr et al., 2018] Schlichtkrull Michael Sejr, Kipf Thomas N., Bloem Peter, Berg Rianne Van Den, Titov Ivan, and Welling Max. Modeling relational data with graph convolutional networks. In The Semantic Web -15th International Conference, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Sejr","year":"2018","venue":"The Semantic Web -15th International Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for auto- matic evaluation of machine translation. In the Association for Computational Linguistics, 2002.","title":"BLEU: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems. 2019.","title":"PyTorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qi et al., 2020] Weizhen Qi, Yu Yan, Yeyun Gong, Dayi- heng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. ProphetNet: Predicting future n-gram for sequence-to-Sequence Pre-training. In Findings of the As- sociation for Computational Linguistics, 2020.","title":"ProphetNet: Predicting future n-gram for sequence-to-Sequence Pre-training","authors":"Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou","year":"2020","venue":"Findings of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In the Con- ference on Empirical Methods in Natural Language Pro- cessing, 2016.","title":"SQuAD: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rajpurkar et al., 2018] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In the Association for Computa- tional Linguistics, 2018.","title":"Know what you don't know: Unanswerable questions for SQuAD","authors":"Pranav Rajpurkar, Robin Jia, Percy Liang","year":"2018","venue":"the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sukhbaatar et al., 2019] Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In the Association for Com- putational Linguistics, 2019.","title":"Adaptive attention span in transformers","authors":"Sukhbaatar","year":"2019","venue":"the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sun et al., 2019] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Se- quential recommendation with bidirectional encoder rep- resentations from transformer. In the Conference on Infor- mation and Knowledge Management, 2019.","title":"BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer","authors":"Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang","year":"2019","venue":"the Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tian and Lo, 2015] Yuan Tian and David Lo. A compara- tive study on the effectiveness of part-of-speech tagging techniques on bug reports. In the Conference on Software Analysis, Evolution, and Reengineering, 2015.","title":"A comparative study on the effectiveness of part-of-speech tagging techniques on bug reports","authors":"Yuan Tian, David Lo","year":"2015","venue":"the Conference on Software Analysis, Evolution, and Reengineering","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tjong Kim Sang and De Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In the Seventh Conference on Natural Language Learning at HLT-NAACL, 2003.","title":"Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition","authors":"Tjong Kim, Sang, De Meulder, ; ] Erik, F Tjong, Kim Sang, Fien De, Meulder","year":"2003","venue":"the Seventh Conference on Natural Language Learning at HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the EMNLP Workshop BlackboxNLP, 2018.","title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2018","venue":"the EMNLP Workshop BlackboxNLP","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Williams et al., 2018] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In the Confer- ence of the North American Chapter of the Association for Computational Linguistics, 2018.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel Bowman","year":"2018","venue":"the Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wu et al., 2019] Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In 7th Inter- national Conference on Learning Representations, 2019.","title":"Pay less attention with lightweight and dynamic convolutions","authors":"Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, Michael Auli","year":"2019","venue":"7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xiao et al., 2020] Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE- GEN: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation. arXiv preprint arXiv:2001.11314, 2020.","title":"ERNIE-GEN: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation","authors":"Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hua Hao Tian, Haifeng Wu, Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2020/553"},{"position":41,"raw":"[Xiao et al., 2021] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106.14881, 2021.","title":"Early convolutions help transformers see better","authors":"Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, Ross Girshick","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yamada et al., 2020] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. LUKE: Deep contextualized entity representations with entity- aware self-attention. In the Conference on Empirical Methods in Natural Language Processing, 2020.","title":"LUKE: Deep contextualized entity representations with entityaware self-attention","authors":"Yamada","year":"2020","venue":"the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yang et al., 2018] Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. Modeling localness for self-attention networks. In the Conference on Empirical Methods in Natural Language Processing, 2018.","title":"Modeling localness for self-attention networks","authors":"Yang","year":"2018","venue":"the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yuan et al., 2021] Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. VOLO: Vision outlooker for vi- sual recognition. arXiv preprint arXiv:2106.13112, 2021.","title":"VOLO: Vision outlooker for visual recognition","authors":"Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zellers et al., 2018] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adver- sarial dataset for grounded commonsense inference. In the Conference on Empirical Methods in Natural Language Processing, 2018.","title":"SWAG: A large-scale adversarial dataset for grounded commonsense inference","authors":"Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi","year":"2018","venue":"the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang et al., 2020] Zhebin Zhang, Sai Wu, Gang Chen, and Dawei Jiang. Self-attention and dynamic convolution hy- brid model for neural machine translation. In IEEE Inter- national Conference on Knowledge Graph (ICKG), 2020.","title":"Self-attention and dynamic convolution hybrid model for neural machine translation","authors":"Zhang","year":"2020","venue":"IEEE International Conference on Knowledge Graph (ICKG)","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhuang et al., 2021] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training ap- proach with post-training. In the 20th Chinese National Conference on Computational Linguistics, 2021.","title":"A robustly optimized BERT pre-training approach with post-training","authors":"Liu Zhuang, Lin Wayne, Shi Ya, Zhao Jun","year":"2021","venue":"the 20th Chinese National Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}