How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?

Abstract

Chemical Language Models (CLMs) pre-trained on large-scale molecular data are widely used for molecular property prediction. However, the common belief that increasing training resources, such as model size, dataset size, and training compute, improves both pre-training loss and downstream task performance has not been systematically validated in the chemical domain. In this work, we evaluate this assumption by pre-training CLMs while scaling training resources and measuring transfer performance across diverse molecular property prediction (MPP) tasks. We find that while pre-training loss consistently decreases with increased training resources, downstream task performance shows limited improvement. Moreover, alternative metrics based on the Hessian or loss landscape also fail to estimate downstream performance in CLMs. We further identify conditions under which downstream performance saturates or degrades despite continued improvements in pre-training metrics, and analyze the underlying task-dependent failure modes through parameter space visualizations. These results expose a gap between pre-training-based evaluation and downstream performance, and emphasize the need for model selection and evaluation strategies that explicitly account for downstream task characteristics. The code is available at https://github.com/sagawatatsuya/MolScaleTransfer.