{"type":"ScholarlyArticle","title":"FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining","authors":[{"name":"Zhuang Liu","slug":"zhuang-liu","given":"Zhuang","family":"Liu","affiliation":"Dalian University of Technology , Dalian , China"},{"name":"Degen Huang","slug":"degen-huang","given":"Degen","family":"Huang","affiliation":"Dalian University of Technology , Dalian , China"},{"name":"Kaiyu Huang","slug":"kaiyu-huang","given":"Kaiyu","family":"Huang","affiliation":"Dalian University of Technology , Dalian , China"},{"name":"Zhuang Li","slug":"zhuang-li","given":"Zhuang","family":"Li","affiliation":"Union Mobile Financial Technology Co., Ltd. , Beijing , China"},{"name":"Jun Zhao","slug":"jun-zhao","given":"Jun","family":"Zhao","affiliation":"Union Mobile Financial Technology Co., Ltd. , Beijing , China"}],"abstract":"There is growing interest in the tasks of financial text mining. Over the past few years, the progress of Natural Language Processing (NLP) based on deep learning advanced rapidly. Significant progress has been made with deep learning showing promising results on financial text mining models. However, as NLP models require large amounts of labeled training data, applying deep learning to financial text mining is often unsuccessful due to the lack of labeled training data in financial fields. To address this issue, we present FinBERT (BERT for Financial Text Mining) that is a domain specific language model pre-trained on large-scale financial corpora. In FinBERT, different from BERT, we construct six pre-training tasks covering more knowledge, simultaneously trained on general corpora and financial domain corpora, which can enable FinBERT model better to capture language knowledge and semantic information. The results show that our FinBERT outperforms all current state-of-the-art models. Extensive experimental results demonstrate the effectiveness and robustness of FinBERT. The source code and pre-trained models of FinBERT are available online.","doi":"10.24963/ijcai.2020/622","url":"https://ijcai.org/proceedings/2020/622","pdf":"https://ijcai.org/proceedings/2020/0622.pdf","markdown":"https://ijcai.org/proceedings/2020/622.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/622","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":622,"pages":{"first":4513,"last":4519},"keywords":["Foundation for AI in FinTech: Deep learning and representation for FinTech","Foundation for AI in FinTech: General","Foundation for AI in FinTech: Analyzing big financial data","Other areas: Financial decision-support system","AI for marketing: AI for consumer sentiment analysis","AI for lending: General","AI for payment: AI for payment risk modeling","Foundation for AI in FinTech: Data mining and knowledge discovery for FinTech","AI for marketing: General"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24692/file_24692.m3u8","length_sec":273,"provider":"hls"}],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, and Paul Barham. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. volume ab- s/1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Champin et al., 2018] Pierre-Antoine Champin, Fabien L. Gandon, Mounia Lalmas, and Panagiotis G. Ipeirotis, edi- tors. In Proceedings of WWW. ACM, 2018.","title":"editors","authors":"Champin","year":"2018","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACL, pages 4171-4186, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2019] Li Dong, Nan Yang, and Wenhui Wang. Unified language model pre-training for natural language understanding and generation. In Proceedings of NeurIPS, pages 13042-13054, 2019.","title":"Unified language model pre-training for natural language understanding and generation","authors":"Li Dong, Nan Yang, Wenhui Wang","year":"2019","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2019] Jinhua Du, Yan Huang, and Karo Moila- nen. Sentence boundary detection through sequence la- belling and BERT fine-tuning. In FinNLP IJCAI 2019, pages 81-87, Macao, China, August 2019.","title":"Sentence boundary detection through sequence labelling and BERT fine-tuning","authors":"Jinhua Du, Yan Huang, Karo Moilanen","year":"2019","venue":"FinNLP IJCAI 2019","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fatima et al., 2019] Mehwish Fatima, Mark-Christoph Mueller, and Christoph Mark. Machine learning vs. rule-based sentence boundary detection. In FinNLP IJCAI 2019, pages 115-121, Macao, China, August 2019.","title":"Machine learning vs. rule-based sentence boundary detection","authors":"Mehwish Fatima, Mark-Christoph Mueller, Christoph Mark","year":"2019","venue":"FinNLP IJCAI 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[FiQ, 2018] In Macedo Maia, Siegfried Handschuh, An- dre Freitas, Brian Davis, Ross Mcdermott, and Manel Zarrouk, editors, In Proceedings of WWW, 2018.","title":"Proceedings of WWW","authors":"Fiq","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross B. Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyro- la, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1706.02677, 2017.","title":"Accurate, large minibatch SGD: training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross B Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Joshi et al., 2019] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predict- ing spans. CoRR, abs/1907.10529, 2019.","title":"Spanbert: Improving pre-training by representing and predicting spans","authors":"Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, Omer Levy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krishnamoorthy, 2018] Srikumar Krishnamoorthy. Senti- ment analysis of financial news articles using performance indicators. Knowl. Inf. Syst., 56(2):373-394, 2018.","title":"Sentiment analysis of financial news articles using performance indicators","authors":"Srikumar Krishnamoorthy","year":"2018","venue":"Knowl. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. 2020.","title":"ALBERT: A lite BERT for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2019a] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural network- s for natural language understanding. In Proceedings of ACL, pages 4487-4496, 2019.","title":"Multi-task deep neural networks for natural language understanding","authors":"Liu","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019b] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, and Danqi Chen. Roberta: A robustly optimized BERT pretraining approach. CoRR, ab- s/1907.11692, 2019.","title":"Roberta: A robustly optimized BERT pretraining approach","authors":"Liu","year":"2019","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Zhuang Liu, Keli Xiao, Bo Jin, Kaiyu Huang, and Yunxia Zhang. Unified generative adversar- ial networks for multiple-choice oriented machine com- prehension. ACM Transactions on Intelligent Systems and Technology, 11(3):1-20, 2020.","title":"Unified generative adversarial networks for multiple-choice oriented machine com-prehension","authors":"Zhuang Liu, Keli Xiao, Bo Jin, Kaiyu Huang, Yunxia Zhang","year":"2020","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Malo et al., 2014] Pekka Malo, Ankur Sinha, Pekka J. Ko- rhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. JASIST, 65(4):782-796, 2014.","title":"Good debt or bad debt: Detecting semantic orientations in economic texts","authors":"Pekka Malo, Ankur Sinha, Pekka J Korhonen, Jyrki Wallenius, Pyry Takala","year":"2014","venue":"JASIST","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In Proceedings of ICLR, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu","year":"2018","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. Computer Science, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, and Christopher Clark. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark","year":"2018","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Piao et al., 2018] Guangyuan Piao, John G. Breslin, and Ross Mc-Dermott. Financial aspect and sentiment predic- tions with deep neural networks: An ensemble approach. In Proceedings of WWW, pages 1973-1977, 2018.","title":"Financial aspect and sentiment predictions with deep neural networks: An ensemble approach","authors":"Guangyuan Piao, John G Breslin, Ross Mc-Dermott","year":"2018","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Raaci, 2019] Dogu Raaci. Finbert: Financial sentimen- t analysis with pre-trained language models. CoRR, ab- s/1908.10063, 2019.","title":"Finbert: Financial sentiment analysis with pre-trained language models","authors":"Dogu Raaci","year":"2019","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raffel et al., 2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sergeev and Balso, 2018] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. CoRR, abs/1802.05799, 2018.","title":"Horovod: fast and easy distributed deep learning in tensorflow","authors":"Alexander Sergeev, Mike Del, Balso","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2019a] Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, and Danxiang Zhu. ERNIE: enhanced representation through knowledge integration. CoRR, abs/1904.09223, 2019.","title":"ERNIE: enhanced representation through knowledge integration","authors":"Sun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2019b] Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for lan- guage understanding. CoRR, abs/1907.12412, 2019.","title":"ERNIE 2.0: A continual pre-training framework for language understanding","authors":"Sun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian and Peng, 2019] Ke Tian and Zi Jun Peng. Sentence boundary detection in noisy texts from financial docu- ments using deep attention model. In FinNLP IJCAI 2019, pages 88-92, Macao, China, August 2019.","title":"Sentence boundary detection in noisy texts from financial documents using deep attention model","authors":"Peng ; ] Ke Tian, Zi Jun Tian, Peng","year":"2019","venue":"FinNLP IJCAI 2019","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2018] Steve Yang, Jason Rosenfeld, and Jacques Makutonin. Financial aspect-based senti- ment analysis using deep representations. CoRR, abs/1808.07931, 2018.","title":"Financial aspect-based sentiment analysis using deep representations","authors":"Steve Yang, Jason Rosenfeld, Jacques Makutonin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}