{"type":"ScholarlyArticle","title":"S2SNet: A Pretrained Neural Network for Superconductivity Discovery","authors":[{"name":"Ke Liu","slug":"ke-liu","given":"Ke","family":"Liu","affiliation":"Zhejiang University"},{"name":"Kaifan Yang","slug":"kaifan-yang","given":"Kaifan","family":"Yang","affiliation":"Zhejiang University"},{"name":"Jiahong Zhang","slug":"jiahong-zhang","given":"Jiahong","family":"Zhang","affiliation":"Zhejiang University; Yunnan University"},{"name":"Renjun Xu","slug":"renjun-xu","given":"Renjun","family":"Xu","affiliation":"Zhejiang University; ZJU-Hangzhou Global Scientific and Technological Innovation Center"}],"abstract":"Superconductivity allows electrical current to flow without any energy loss, and thus making solids superconducting is a grand goal of physics, material science, and electrical engineering. More than 16 Nobel Laureates have been awarded for their contribution in superconductivity research. Superconductors are valuable  for sustainable development goals (SDGs), such as climate change mitigation, affordable and clean energy, industry, innovation and infrastructure, and so on. However, a unified physics theory explaining all superconductivity mechanism is still unknown. It is believed that superconductivity is microscopically due to not only molecular compositions but also the geometric crystal structure. Hence a new dataset, S2S, containing both crystal structures and superconducting critical temperature, is built upon SuperCon and Material Project. Based on this new dataset, we propose a novel model, S2SNet, which utilizes the attention mechanism for superconductivity prediction. To overcome the shortage of data, S2SNet is pre-trained on the whole Material Project dataset with Masked-Language Modeling (MLM). S2SNet makes a new state-of-the-art, with out-of-sample accuracy of 92% and Area Under Curve (AUC) of 0.92. To the best of our knowledge, S2SNet is the first work to predict superconductivity with only information of crystal structures. This work is beneficial to superconductivity discovery and further SDGs. The code and datasets are available at https://github.com/supercond/S2SNet","doi":"10.24963/ijcai.2022/708","url":"https://ijcai.org/proceedings/2022/708","pdf":"https://ijcai.org/proceedings/2022/0708.pdf","markdown":"https://ijcai.org/proceedings/2022/708.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/708","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI for Good","number":708,"pages":{"first":5101,"last":5107},"keywords":["Multidisciplinary Topics and Applications: Physical Science"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987137","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985280","length_sec":613,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"3rd International Conference on Learning Representations, ICLR 2015","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baldow et al., 2017] Christoph Baldow, Sebastian Salentin, Michael Schroeder, Ingo Roeder, and Ingmar Glauche. Magpie: Simplifying access and execution of computa- tional models in the life sciences. PLoS computational biology, 13(12):e1005898, 2017.","title":"Magpie: Simplifying access and execution of computational models in the life sciences","authors":"Christoph Baldow, Sebastian Salentin, Michael Schroeder, Ingo Roeder, Ingmar Glauche","year":"2017","venue":"PLoS computational biology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bardeen et al., 1957] John Bardeen, Leon N Cooper, and J Robert Schrieffer. Microscopic theory of superconduc- tivity. Physical Review, 106(1):162, 1957.","title":"Microscopic theory of superconductivity","authors":"John Bardeen, Leon N Cooper, Robert Schrieffer","year":"1957","venue":"Physical Review","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bergerhoff et al., 1987] G Bergerhoff, ID Brown, F Allen, et al. Crystallographic databases. International Union of Crystallography, Chester, 360:77-95, 1987.","title":"Crystallographic databases","authors":"Bergerhoff, Id Brown, Allen","year":"1987","venue":"International Union of Crystallography","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as a univer- sal machine learning framework for molecules and crys- tals. Chemistry of Materials, 31(9):3564-3572, 2019.","title":"Graph networks as a universal machine learning framework for molecules and crystals","authors":"Chen","year":"2019","venue":"Chemistry of Materials","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hamidieh, 2018] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a super- conductor. Computational Materials Science, 154:346- 354, 2018.","title":"A data-driven statistical model for predicting the critical temperature of a superconductor","authors":"Kam Hamidieh","year":"2018","venue":"Computational Materials Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2021] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trained mod- els: Past, present and future. arXiv e-prints, pages arXiv- 2106, 2021.","title":"Pre-trained models: Past, present and future","authors":"Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton and Roweis, 2002] Geoffrey Hinton and Sam T Roweis. Stochastic neighbor embedding. In NIPS, vol- ume 15, pages 833-840. Citeseer, 2002.","title":"Stochastic neighbor embedding","authors":"Roweis Hinton, Geoffrey Hinton, Sam T Roweis","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hirschberg and Manning, 2015] Julia Hirschberg and Christopher D Manning. Advances in natural language processing. Science, 349(6245):261-266, 2015.","title":"Advances in natural language processing","authors":"Julia Hirschberg, Christopher D Manning","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jain et al., 2013] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Ger- brand Ceder, and Kristin a. Persson. The Materials Project: A materials genome approach to accelerating materials in- novation. APL Materials, 1(1):011002, 2013.","title":"The Materials Project: A materials genome approach to accelerating materials innovation","authors":"Anubhav Jain, Ping Shyue, Geoffroy Ong, Wei Hautier, William Davidson Chen, Stephen Richards, Shreyas Dacek, Dan Cholia, David Gunter, Gerbrand Skinner, Kristin A Ceder, Persson","year":"2013","venue":"APL Materials","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kirklin et al., 2015] Scott Kirklin, James E. Saal, Bryce Meredig, Alex Thompson, Jeff W. Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The open quantum materials database (oqmd): Assessing the accu- racy of dft formation energies. npj Computational Mate- rials, 1, December 2015. Publisher Copyright: c 2015 Shanghai Institute of Ceramics, Chinese Academy of Sci- ences/Macmillan Publishers Limited.","title":"The open quantum materials database (oqmd): Assessing the accuracy of dft formation energies","authors":"Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, Chris Wolverton","year":"2015","venue":"npj Computational Materials","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kwaśniewska et al., 2017] Alicja Kwaśniewska, Jacek Rumiński, and Paul Rad. Deep features class activation map for thermal face detection and tracking. In 2017 10Th international conference on human system interactions (HSI), pages 41-47. IEEE, 2017.","title":"Deep features class activation map for thermal face detection and tracking","authors":"Alicja Kwaśniewska, Jacek Rumiński, Paul Rad","year":"2017","venue":"2017 10Th international conference on human system interactions (HSI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le-Khac et al., 2020] Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.","title":"Contrastive representation learning: A framework and review","authors":"Graham Phuc H Le-Khac, Alan F Healy, Smeaton","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mercado et al., 2021] Rocío Mercado, Tobias Rastemo, Ed- vard Lindelöf, Günter Klambauer, Ola Engkvist, Hong- ming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2(2):025023, 2021.","title":"Graph networks for molecular design","authors":"Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, Esben Jannik Bjerrum","year":"2021","venue":"Machine Learning: Science and Technology","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miryala, 2021] Muralidhar Miryala. Expert opinion: Rele- vance of high-tc superconductors for sdg goals. In High-Tc Superconducting Technology, pages 1-16.","title":"Expert opinion: Relevance of high-tc superconductors for sdg goals","authors":"Muralidhar Miryala","year":null,"venue":"High-Tc Superconducting Technology","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Park and Wolverton, 2020] Cheol Woo Park and Chris Wolverton. Developing an improved crystal graph con- volutional neural network framework for accelerated ma- terials discovery. Physical Review Materials, 4(6):063801, 2020.","title":"Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery","authors":"Wolverton Park, Cheol Woo Park, Chris Wolverton","year":"2020","venue":"Physical Review Materials","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmidt et al., 2019] Jonathan Schmidt, Mário RG Mar- ques, Silvana Botti, and Miguel AL Marques. Recent ad- vances and applications of machine learning in solid-state materials science. npj Computational Materials, 5(1):1- 36, 2019.","title":"Recent advances and applications of machine learning in solid-state materials science","authors":"Jonathan Schmidt, R G Mário, Silvana Marques, Miguel Al Botti, Marques","year":"2019","venue":"npj Computational Materials","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stanev et al., 2018] Valentin Stanev, Corey Oses, A Gilad Kusne, Efrain Rodriguez, Johnpierre Paglione, Stefano Curtarolo, and Ichiro Takeuchi. Machine learning mod- eling of superconducting critical temperature. npj Compu- tational Materials, 4(1):1-14, 2018.","title":"Machine learning modeling of superconducting critical temperature","authors":"Valentin Stanev, Corey Oses, Efrain Gilad Kusne, Johnpierre Rodriguez, Stefano Paglione, Ichiro Curtarolo, Takeuchi","year":"2018","venue":"npj Computational Materials","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Villars and Phillips, 1988] Pierre Villars and James Charles Phillips. Quantum structural diagrams and high-t c super- conductivity. Physical Review B, 37(4):2345, 1988.","title":"Quantum structural diagrams and high-t c superconductivity","authors":"Phillips Villars, James Charles Phillips","year":"1988","venue":"Physical Review B","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Samy Bengio, and Man- junath Kudlur. Order Matters: Sequence to sequence for sets. arXiv e-prints, page arXiv:1511.06391, November 2015.","title":"Order Matters: Sequence to sequence for sets","authors":"Oriol Vinyals, Samy Bengio, Manjunath Kudlur","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie and Grossman, 2018] Tian Xie and Jeffrey C Gross- man. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material proper- ties. Physical review letters, 120(14):145301, 2018.","title":"Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties","authors":"Tian Xie, Jeffrey C Grossman","year":"2018","venue":"Physical review letters","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2021] Qiong Yang, Hongchao Ji, Hongmei Lu, and Zhimin Zhang. Prediction of liquid chromato- graphic retention time with graph neural networks to as- sist in small molecule identification. Analytical Chemistry, 93(4):2200-2206, 2021.","title":"Prediction of liquid chromatographic retention time with graph neural networks to assist in small molecule identification","authors":"Qiong Yang, Hongchao Ji, Hongmei Lu, Zhimin Zhang","year":"2021","venue":"Analytical Chemistry","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2021] Xiao-Chen Zhang, Cheng-Kun Wu, Zhi-Jiang Yang, Zhen-Xing Wu, Jia-Cai Yi, Chang-Yu Hsieh, Ting-Jun Hou, and Dong-Sheng Cao. Mg-bert: leveraging unsupervised atomic representation learning for molecular property prediction. Briefings in Bioinfor- matics, 2021.","title":"Mg-bert: leveraging unsupervised atomic representation learning for molecular property prediction","authors":"Xiao-Chen Zhang, Cheng-Kun Wu, Zhi-Jiang Yang, Zhen-Xing Wu, Jia-Cai Yi, Chang-Yu Hsieh, Ting-Jun Hou, Dong-Sheng Cao","year":"2021","venue":"Briefings in Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2018] Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, and Shou-Cheng Zhang. Learning atoms for materials discovery. Proceedings of the National Academy of Sciences, 115(28):E6411-E6417, 2018.","title":"Learning atoms for materials discovery","authors":"Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, Shou-Cheng Zhang","year":"2018","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}