{"type":"ScholarlyArticle","title":"Geometric Transformer for End-to-End Molecule Properties Prediction","authors":[{"name":"Yoni Choukroun","slug":"yoni-choukroun","given":"Yoni","family":"Choukroun","affiliation":"School of Computer Science , Tel Aviv University"},{"name":"Lior Wolf","slug":"lior-wolf","given":"Lior","family":"Wolf","affiliation":"School of Computer Science , Tel Aviv University"}],"abstract":"Transformers have become methods of choice in many applications thanks to their ability to represent complex interactions between elements. \r\nHowever, extending the Transformer architecture to non-sequential data such as molecules and enabling its training on small datasets remains a challenge. \r\nIn this work, we introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule. \r\nWe modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism. \r\nWe further suggest an augmentation scheme for molecular data capable of avoiding the overfitting induced by the overparameterized architecture. \r\nThe proposed framework outperforms the state-of-the-art methods while being based on pure machine learning solely, i.e. the method does not incorporate domain knowledge from quantum chemistry and does not use extended geometric inputs besides the pairwise atomic distances.","doi":"10.24963/ijcai.2022/401","url":"https://ijcai.org/proceedings/2022/401","pdf":"https://ijcai.org/proceedings/2022/0401.pdf","markdown":"https://ijcai.org/proceedings/2022/401.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/401","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":401,"pages":{"first":2895,"last":2901},"keywords":["Machine Learning: Attention Models","Multidisciplinary Topics and Applications: Physical Science","Machine Learning: Sequence and Graph Learning","Machine Learning: Representation learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986528","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984671","length_sec":275,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anderson et al., 2019] B Anderson, T Son Hy, and R Kon- dor. Cormorant: Covariant molecular neural networks. arXiv:1906.04015, 2019.","title":"Cormorant: Covariant molecular neural networks","authors":"Son Anderson, Hy, Kondor","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Axilrod and Teller, 1943] BM Axilrod and Ei Teller. Inter- action of the van der waals type between three atoms. The Journal of Chemical Physics, 1943.","title":"Interaction of the van der waals type between three atoms","authors":"Teller ; Bm Axilrod, Ei Axilrod, Teller","year":"1943","venue":"The Journal of Chemical Physics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartok et al., 2010] A Bartok, M Payne, et al. Gaussian ap- proximation potentials: The accuracy of quantum mechan- ics, without the electrons. Physical review letters, 2010.","title":"Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons","authors":"Bartok, Payne","year":"2010","venue":"Physical review letters","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Belkin and Niyogi, 2003] M Belkin and P Niyogi. Lapla- cian eigenmaps for dimensionality reduction and data rep- resentation. Neural computation, 2003.","title":"Laplacian eigenmaps for dimensionality reduction and data representation","authors":"M Belkin, Niyogi","year":"2003","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bello et al., 2019] I Bello, B Zoph, A Vaswani, et al. Atten- tion augmented convolutional networks. In ICCV, 2019.","title":"Attention augmented convolutional networks","authors":"Bello","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2017] Q Chen, X Zhu, Z Ling, et al. Neural natural language inference models enhanced with external knowledge. arXiv:1711.04289, 2017.","title":"Neural natural language inference models enhanced with external knowledge","authors":"X Chen, Zhu, Ling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chmiela et al., 2017] S Chmiela, A Tkatchenko, et al. Ma- chine learning of accurate energy-conserving molecular force fields. Science, 2017.","title":"Machine learning of accurate energy-conserving molecular force fields","authors":"Chmiela, Tkatchenko","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chmiela et al., 2018] S Chmiela, H Sauceda, K-R Müller, et al. Towards exact molecular dynamics simulations with machine-learned force fields. Nature, 2018.","title":"Towards exact molecular dynamics simulations with machine-learned force fields","authors":"H Chmiela, K-R Sauceda, Müller","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Christensen and others, 2020] A Christensen et al. Fchl re- visited: Faster and more accurate quantum machine learn- ing. The Journal of Chemical Physics, 2020.","title":"Fchl revisited: Faster and more accurate quantum machine learning","authors":"Others ; A Christensen, Christensen","year":"2020","venue":"The Journal of Chemical Physics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dwivedi and Bresson, 2020] VP Dwivedi and X Bresson. A generalization of transformer networks to graphs. arXiv:2012.09699, 2020.","title":"A generalization of transformer networks to graphs","authors":"Vp Dwivedi, Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fuchs et al., 2020] F B Fuchs, D E Worrall, V Fischer, and M Welling. Se (3)-transformers: 3d roto-translation equiv- ariant attention networks. arXiv:2006.10503, 2020.","title":"Se (3)-transformers: 3d roto-translation equivariant attention networks","authors":"F B Fuchs, D E Worrall, V Fischer, M Welling","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gilmer et al., 2017] J Gilmer, S Schoenholz, P Riley, O Vinyals, and G Dahl. Neural message passing for quan- tum chemistry. arXiv:1704.01212, 2017.","title":"Neural message passing for quantum chemistry","authors":"S Gilmer, P Schoenholz, O Riley, G Vinyals, Dahl","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goyal and Bengio, 2020] A Goyal and Y Bengio. Induc- tive biases for deep learning of higher-level cognition. arXiv:2011.15091, 2020.","title":"Inductive biases for deep learning of higher-level cognition","authors":"Bengio ; A Goyal, Y Goyal, Bengio","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang and others, 2016] Bing Huang et al. Communica- tion: Understanding molecular representations in machine learning: The role of uniqueness and target similarity. J. of Chemical Physics, 2016.","title":"Communication: Understanding molecular representations in machine learning: The role of uniqueness and target similarity","authors":"Bing Huang, Huang","year":"2016","venue":"J. of Chemical Physics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jensen, 2017] Frank Jensen. Introduction to computational chemistry. John wiley & sons, 2017.","title":"Introduction to computational chemistry","authors":"Frank Jensen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Klicpera et al., 2020a] J Klicpera, S Giri, JT Margraf, et al. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv:2011.14115, 2020.","title":"Fast and uncertainty-aware directional message passing for non-equilibrium molecules","authors":"Klicpera","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Klicpera et al., 2020b] J Klicpera, J Groß, and S Günnemann. Directional message passing for molecular graphs. arXiv:2003.03123, 2020.","title":"Directional message passing for molecular graphs","authors":"Klicpera","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kwak et al., 2021] B Kwak, J Jo, B Lee, and S Yoon. Geometry-aware transformer for molecular property pre- diction. arXiv:2106.15516, 2021.","title":"Geometry-aware transformer for molecular property prediction","authors":"Kwak, B Jo, Lee, Yoon","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2019] J Lee, Y Lee, J Kim, et al. Set trans- former: A framework for attention-based permutation- invariant neural networks. In ICML, 2019.","title":"Set transformer: A framework for attention-based permutationinvariant neural networks","authors":"Lee, Lee, Kim","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2019] C Lu, Q Liu, C Wang, et al. Molecular property prediction: A multilevel quantum interactions modeling perspective. In AAAI, 2019.","title":"Molecular property prediction: A multilevel quantum interactions modeling perspective","authors":"Q Lu, C Liu, Wang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maziarka et al., 2020] L Maziarka, T Danel, S Mucha, et al. Molecule attention transformer. arXiv:2002.08264, 2020.","title":"Molecule attention transformer","authors":"Maziarka, Danel, Mucha","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Maziarka et al., 2021] L Maziarka, D Majchrowski, T Danel, et al. Relative molecule self-attention trans- former. arXiv:2110.05841, 2021.","title":"Relative molecule self-attention transformer","authors":"Maziarka, Majchrowski, Danel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Muto, 1943] Yoshio Muto. Force between nonpolar molecules. J. Phys. Math. Soc. Japan, 1943.","title":"Force between nonpolar molecules","authors":"Yoshio Muto","year":"1943","venue":"J. Phys. Math. Soc","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qiao and others, 2020] Z Qiao et al. Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic- orbital features. J. of Chemical Physics, 2020.","title":"Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomicorbital features","authors":"Qiao","year":"2020","venue":"J. of Chemical Physics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ramakrishnan et al., 2014] R Ramakrishnan, P Dral, M Rupp, et al. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 2014.","title":"Quantum chemistry structures and properties of 134 kilo molecules","authors":"R Ramakrishnan, M Dral, Rupp","year":"2014","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schütt et al., 2018a] K Schütt, H Sauceda, P-J Kindermans, et al. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 2018.","title":"Schnet-a deep learning architecture for molecules and materials","authors":"Schütt","year":"2018","venue":"The Journal of Chemical Physics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schutt et al., 2018b] KT Schutt, P Kessel, et al. Schnetpack: A deep learning toolbox for atomistic systems. Journal of chemical theory and computation, 2018.","title":"Schnetpack: A deep learning toolbox for atomistic systems","authors":"Schutt","year":"2018","venue":"Journal of chemical theory and computation","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Unke and others, 2019] O Unke et al. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. J. chemical theory and computation, 2019.","title":"PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges","authors":"Unke","year":"2019","venue":"J. chemical theory and computation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vaswani et al., 2017] A Vaswani, N Shazeer, N Parmar, et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Vaswani, Shazeer, Parmar","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang and others, 2020] H Wang et al. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020.","title":"Axial-deeplab: Stand-alone axial-attention for panoptic segmentation","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2021] F Wu, Q Zhang, D Radev, et al. 3d- transformer: Molecular representation with transformer in 3d space. arXiv:2110.01191, 2021.","title":"3dtransformer: Molecular representation with transformer in 3d space","authors":"Q Wu, D Zhang, Radev","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and others, 2017] H Zhang et al. mixup: Beyond em- pirical risk minimization. arXiv:1710.09412, 2017.","title":"mixup: Beyond empirical risk minimization","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}