Proceedings

Pay Attention to the Keys: Visual Piano Transcription Using Transformers

Uros Zivanovic, Ivan Pilkov, Carlos Cancino-Chacón

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI, Arts & Creativity. Pages 10243-10251.

https://doi.org/10.24963/ijcai.2025/1138

Keywords

  • Application domains: Music and sound
  • Methods and resources: AI methods for better understanding human creative processes
  • Methods and resources: Machine learning, deep learning, neural models, reinforcement learning

Abstract

Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone (e.g., from a top-down video of the piano keyboard). In this work we propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs). Our system is trained on the newly introduced R3 dataset, consisting of ca.~31 hours of synchronized video and MIDI recordings of piano performances. We additionally introduce an approach to predict note offsets, which has not been previously explored in this context. We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets.

Uros Zivanovic, Ivan Pilkov, and Carlos Cancino-Chacón. 2025. Pay Attention to the Keys: Visual Piano Transcription Using Transformers. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 10243–10251. https://doi.org/10.24963/ijcai.2025/1138
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards