{"type":"ScholarlyArticle","title":"Pay Attention to the Keys: Visual Piano Transcription Using Transformers","authors":[{"name":"Uros Zivanovic","slug":"uros-zivanovic","given":"Uros","family":"Zivanovic","affiliation":""},{"name":"Ivan Pilkov","slug":"ivan-pilkov","given":"Ivan","family":"Pilkov","affiliation":""},{"name":"Carlos Cancino-Chacón","slug":"carlos-cancino-chacon","given":"Carlos","family":"Cancino-Chacón","affiliation":""}],"abstract":"Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone (e.g., from a top-down video of the piano keyboard). In this work we propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs). Our system is trained on the newly introduced R3 dataset, consisting of ca.~31 hours of synchronized video and MIDI recordings of piano performances. We additionally introduce an approach to predict note offsets, which has not been previously explored in this context. We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets.","doi":"10.24963/ijcai.2025/1138","url":"https://ijcai.org/proceedings/2025/1138","pdf":"https://ijcai.org/proceedings/2025/1138.pdf","markdown":"https://ijcai.org/proceedings/2025/1138.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1138","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":1138,"pages":{"first":10243,"last":10251},"keywords":["Application domains: Music and sound","Methods and resources: AI methods for better understanding human creative processes","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}