{"type":"ScholarlyArticle","title":"The ACCompanion: Combining Reactivity, Robustness, and Musical Expressivity in an Automatic Piano Accompanist","authors":[{"name":"Carlos Cancino-Chacón","slug":"carlos-cancino-chacon","given":"Carlos","family":"Cancino-Chacón","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Silvan Peter","slug":"silvan-peter","given":"Silvan","family":"Peter","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Patricia Hu","slug":"patricia-hu","given":"Patricia","family":"Hu","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Emmanouil Karystinaios","slug":"emmanouil-karystinaios","given":"Emmanouil","family":"Karystinaios","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Florian Henkel","slug":"florian-henkel","given":"Florian","family":"Henkel","affiliation":"SiriusXM + Pandora , USA"},{"name":"Francesco Foscarin","slug":"francesco-foscarin","given":"Francesco","family":"Foscarin","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Gerhard Widmer","slug":"gerhard-widmer","given":"Gerhard","family":"Widmer","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"}],"abstract":"This paper introduces the ACCompanion, an expressive accompaniment system. Similarly to a musician who accompanies a soloist playing a given musical piece, our system can produce a human-like rendition of the accompaniment part that follows the soloist's choices in terms of tempo, dynamics, and articulation. The ACCompanion works in the symbolic domain, i.e., it needs a musical instrument capable of producing and playing MIDI data, with explicitly encoded onset, offset, and pitch for each played note. We describe the components that go into such a system, from real-time score following and prediction to expressive performance generation and online adaptation to the expressive choices of the human player. Based on our experience with repeated live demonstrations in front of various audiences, we offer an analysis of the challenges of combining these components into a system that is highly reactive and precise, while still a reliable musical partner, robust to possible performance errors and responsive to expressive variations.","doi":"10.24963/ijcai.2023/641","url":"https://ijcai.org/proceedings/2023/641","pdf":"https://ijcai.org/proceedings/2023/0641.pdf","markdown":"https://ijcai.org/proceedings/2023/641.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/641","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI and Arts","track":"AI and Arts","number":641,"pages":{"first":5779,"last":5787},"keywords":["Application domains: Music and sound","Methods and resources: Applications and software frameworks","Theory and philosophy of arts and creativity in AI systems: Social (multi-agent) creativity and human-computer co-creation"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Dixon, 2019] Ruchit Agrawal and Simon Dixon. A Hybrid Approach to Audio-To-Score Align- ment. In Proceedings of the International Conference on International Conference on Machine Learning, 2019.","title":"Ruchit Agrawal and Simon Dixon. A Hybrid Approach to Audio-To-Score Alignment","authors":"Dixon Agrawal","year":"2019","venue":"Proceedings of the International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arzt and Lattner, 2018] Andreas Arzt and Stefan Lattner. Audio-To-Score Alignment using Transposition-Invariant Features. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2018.","title":"Audio-To-Score Alignment using Transposition-Invariant Features","authors":"Lattner Arzt, Andreas Arzt, Stefan Lattner","year":"2018","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arzt and Widmer, 2015] Andreas Arzt and Gerhard Wid- mer. Real-Time Music Tracking Using Multiple Perfor- mances as a Reference. In Proceedings of the Interna- tional Society for Music Information Retrieval Conference (ISMIR), 2015.","title":"Real-Time Music Tracking Using Multiple Performances as a Reference","authors":"Widmer Arzt, Andreas Arzt, Gerhard Widmer","year":"2015","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arzt, 2016] Andreas Arzt. Flexible and Robust Music Tracking. PhD thesis, Johannes Kepler University Linz, Austria, 2016.","title":"Flexible and Robust Music Tracking","authors":"Andreas Arzt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bishop, 2023] Laura Bishop. Focus of attention affects togetherness experiences and body interactivity in piano duos. Psychology of Aesthetics, Creativity, and the Arts, 2023.","title":"Focus of attention affects togetherness experiences and body interactivity in piano duos","authors":"Laura Bishop","year":"2023","venue":"Psychology of Aesthetics, Creativity, and the Arts","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cancino-Chacón et al., 2017] Carlos Eduardo Cancino- Chacón, Thassilo Gadermaier, Gerhard Widmer, and Maarten Grachten. An Evaluation of Linear and Non- linear Models of Expressive Dynamics in Classical Piano and Symphonic Music. Machine Learning, 106(6):887-909, 2017.","title":"An Evaluation of Linear and Nonlinear Models of Expressive Dynamics in Classical Piano and Symphonic Music","authors":"Cancino-Chacón","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cancino-Chacón et al., 2018] Carlos E Cancino-Chacón, Maarten Grachten, Werner Goebl, and Gerhard Widmer. Computational Models of Expressive Music Performance: A Comprehensive and Critical Review. Frontiers in Digi- tal Humanities, 5:25, 2018.","title":"Computational Models of Expressive Music Performance: A Comprehensive and Critical Review","authors":"Maarten Carlos E Cancino-Chacón, Werner Grachten, Gerhard Goebl, Widmer","year":"2018","venue":"Frontiers in Digital Humanities","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cancino-Chacón et al., 2022] Carlos Eduardo Cancino- Chacón, Silvan David Peter, Emmanouil Karystinaios, Francesco Foscarin, Maarten Grachten, and Gerhard Widmer. Partitura: A Python Package for Symbolic Music Processing. In Proceedings of the Music Encoding Conference (MEC2022), Halifax, Canada, 2022.","title":"Partitura: A Python Package for Symbolic Music Processing","authors":"Cancino-Chacón","year":"2022","venue":"Proceedings of the Music Encoding Conference (MEC2022)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2014] Chun-Ta Chen, Jyh-Shing Roger Jang, and Wenshan Liou. Improved Score-Performance Align- ment Algorithms on Polyphonic Music. In 2014 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1365-1369. IEEE, 2014.","title":"Improved Score-Performance Alignment Algorithms on Polyphonic Music","authors":"Chun-Ta Chen, Jyh-Shing Roger Jang, Wenshan Liou","year":"2014","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cont, 2008] Arshia Cont. ANTESCOFO: Anticipatory Syn- chronization and Control of Interactive Parameters in Computer Music. In Proceedings of the International Computer Music Conference (ICMC), pages 33-40, 2008.","title":"Arshia Cont. ANTESCOFO: Anticipatory Synchronization and Control of Interactive Parameters in Computer Music","authors":"Cont","year":"2008","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dannenberg, 1984] Roger B Dannenberg. An On-Line Al- gorithm for Real-Time Accompaniment. In Proceedings of the International Computer Music Conference (ICMC), volume 84, pages 193-198, 1984.","title":"An On-Line Algorithm for Real-Time Accompaniment","authors":"Roger B Dannenberg","year":"1984","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dannenberg, 1987] Roger B Dannenberg. Following an Im- provisation in Real-Time. In Proceedings of the Interna- tional Computer Music Conference (ICMC), pages 241- 248, 1987.","title":"Following an Improvisation in Real-Time","authors":"Roger B Dannenberg","year":"1987","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Demos et al., 2016] Alexander P Demos, Tânia Lisboa, and Roger Chaffin. Flexibility of expressive timing in repeated musical performances. Frontiers in Psychology, 7:1490, 2016.","title":"Flexibility of expressive timing in repeated musical performances","authors":"Tânia Alexander P Demos, Roger Lisboa, Chaffin","year":"2016","venue":"Frontiers in Psychology","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dixon et al., 2006] Simon Dixon, Werner Goebl, and Emil- ios Cambouropoulos. Perceptual Smoothness of Tempo in Expressively Performed Music. Music Perception, 23(3):195-214, 2006.","title":"Perceptual Smoothness of Tempo in Expressively Performed Music","authors":"Simon Dixon, Werner Goebl","year":"2006","venue":"Music Perception","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dixon, 2005] Simon Dixon. An On-Line Time Warping Al- gorithm for Tracking Musical Performances. In Proceed- ings of the International Joint Conference on Artificial In- telligence, pages 1727-1728, 2005.","title":"An On-Line Time Warping Algorithm for Tracking Musical Performances","authors":"Simon Dixon","year":"2005","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2005/240"},{"position":16,"raw":"[Duan and Pardo, 2011] Zhiyao Duan and Bryan Pardo. A State Space Model for Online Polyphonic Audio-Score Alignment. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 197-200. IEEE, 2011.","title":"A State Space Model for Online Polyphonic Audio-Score Alignment","authors":"Pardo Duan, Zhiyao Duan, Bryan Pardo","year":"2011","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Flossmann et al., 2010] Sebastian Flossmann, Werner Goebl, Maarten Grachten, Bernhard Niedermayer, and Gerhard Widmer. The Magaloff Project: An Interim Report. Journal of New Music Research, 39(4):363-377, 2010.","title":"The Magaloff Project: An Interim Report","authors":"Sebastian Flossmann, Werner Goebl, Maarten Grachten, Bernhard Niedermayer, Gerhard Widmer","year":"2010","venue":"Journal of New Music Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Friberg et al., 2006] Anders Friberg, Roberto Bresin, and Johan Sundberg. Overview of the KTH Rule System for Musical Performance. Advances in Cognitive Psychology, 2(2):145, 2006.","title":"Overview of the KTH Rule System for Musical Performance","authors":"Friberg","year":"2006","venue":"Advances in Cognitive Psychology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gingras and McAdams, 2011] Bruno Gingras and Stephen McAdams. Improved Score-Performance Matching us- ing Both Structural and Temporal Information from MIDI Recordings. Journal of New Music Research, 40(1):43-57, 2011.","title":"Improved Score-Performance Matching using Both Structural and Temporal Information from MIDI Recordings","authors":"Mcadams Gingras, Bruno Gingras, Stephen Mcadams","year":"2011","venue":"Journal of New Music Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jeong et al., 2019] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, Kyogu Lee, and Juhan Nam. VirtuosoNet: A Hierar- chical RNN-based System for Modeling Expressive Piano Performance. In Proceedings of the International Soci- ety for Music Information Retrieval Conference (ISMIR), pages 908-915, 2019.","title":"VirtuosoNet: A Hierarchical RNN-based System for Modeling Expressive Piano Performance","authors":"Jeong","year":"2019","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Keller et al., 2007] Peter E Keller, Günther Knoblich, and Bruno H Repp. Pianists duet better when they play with themselves: on the possible role of action simula- tion in synchronization. Consciousness and cognition, 16(1):102-111, 2007.","title":"Pianists duet better when they play with themselves: on the possible role of action simulation in synchronization","authors":"Günther Peter E Keller, Bruno H Knoblich, Repp","year":"2007","venue":"Consciousness and cognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Loehr et al., 2011] Janeen D Loehr, Edward W Large, and Caroline Palmer. Temporal Ccoordination and Adaptation to Rate Change in Music Performance. Journal of Experi- mental Psychology: Human Perception and Performance, 37(4):1292, 2011.","title":"Temporal Ccoordination and Adaptation to Rate Change in Music Performance","authors":"Janeen D Loehr, Edward W Large, Caroline Palmer","year":"2011","venue":"Journal of Experimental Psychology: Human Perception and Performance","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Maezawa et al., 2018] Akira Maezawa, Kazuhiko Ya- mamoto, and Takuya Fujishima. Rendering Music Per- formance with Interpretation Variations using Conditional Variational RNN. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2018.","title":"Rendering Music Performance with Interpretation Variations using Conditional Variational RNN","authors":"Maezawa","year":"2018","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mates, 1994] Jiří Mates. A Model of Synchronization of Motor Acts to a Stimulus Sequence: I. Timing and Er- ror Corrections. Biological Cybernetics, 70(5):463-473, 1994.","title":"Jiří Mates. A Model of Synchronization of Motor Acts to a Stimulus Sequence: I. Timing and Error Corrections","authors":"Mates","year":"1994","venue":"Biological Cybernetics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Murphy, 1998] Kevin P Murphy. Switching Kalman Fil- ters. Technical Report 98-10, Compaq Cambridge Re- search Lab, August 1998.","title":"Switching Kalman Filters","authors":"Kevin P Murphy","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nakamura et al., 2014] Eita Nakamura, Nobutaka Ono, Ya- suyuki Saito, and Shigeki Sagayama. Merged-Output Hid- den Markov Model for Score Following of MIDI Perfor- mance with Ornaments, Desynchronized Voices, Repeats and Skips. In International Conference on Mathematics and Computing, 2014.","title":"Merged-Output Hidden Markov Model for Score Following of MIDI Performance with Ornaments, Desynchronized Voices, Repeats and Skips","authors":"Eita Nakamura, Nobutaka Ono, Yasuyuki Saito, Shigeki Sagayama","year":"2014","venue":"International Conference on Mathematics and Computing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nakamura et al., 2015] Tomohiko Nakamura, Eita Naka- mura, and Shigeki Sagayama. Real-Time Audio-To-Score Alignment of Music Performances containing Errors and Arbitrary Repeats and Skips. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(2):329-339, 2015.","title":"Real-Time Audio-To-Score Alignment of Music Performances containing Errors and Arbitrary Repeats and Skips","authors":"Tomohiko Nakamura, Eita Nakamura, Shigeki Sagayama","year":"2015","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nakamura et al., 2017] Eita Nakamura, Kazuyoshi Yoshii, and Haruhiro Katayose. Performance Error Detection and Post-Processing for Fast and Accurate Symbolic Music Alignment. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 347-353, 2017.","title":"Performance Error Detection and Post-Processing for Fast and Accurate Symbolic Music Alignment","authors":"Eita Nakamura, Kazuyoshi Yoshii, Haruhiro Katayose","year":"2017","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"Peter et al., in press] Silvan Peter, Carlos Eduardo Cancino- Chacón, Emmanouil Karystinaios, Francesco Foscarin, Andrew McLeod, and Gerhard Widmer. Automatic Note- Level Score-To-Performance Alignments in the ASAP Dataset. Transactions of the International Society for Mu- sic Information Retrieval (TISMIR), in press.","title":"Automatic Note-Level Score-To-Performance Alignments in the ASAP Dataset","authors":"Peter","year":null,"venue":"Transactions of the International Society for Music Information Retrieval (TISMIR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rabiner and Juang, 1986] Lawrence Rabiner and Biingh- wang Juang. An Introduction to Hidden Markov Models. IEEE Acoustics, Speech, and Signal Processing Magazine, 3(1):4-16, 1986.","title":"Lawrence Rabiner and Biinghwang Juang. An Introduction to Hidden Markov Models","authors":"Juang Rabiner","year":"1986","venue":"IEEE Acoustics, Speech, and Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Raphael and Gu, 2009] Christopher Raphael and Yupeng Gu. Orchestral Accompaniment for a Reproducing Piano. In International Conference on Mathematics and Comput- ing, 2009.","title":"Orchestral Accompaniment for a Reproducing Piano","authors":"Christopher Raphael, Yupeng Gu","year":"2009","venue":"International Conference on Mathematics and Computing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Raphael, 2010] Christopher Raphael. Music Plus One and Machine Learning. In Proceedings of the Interna- tional Conference on International Conference on Ma- chine Learning, pages 21-28, 2010.","title":"Music Plus One and Machine Learning","authors":"Christopher Raphael","year":"2010","venue":"Proceedings of the International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Repp, 2006] Bruno Hermann Repp. Musical Synchroniza- tion. Music, Motor Control, and the Brain, pages 55-76, 2006.","title":"Musical Synchronization","authors":"Bruno Repp, Repp Hermann","year":"2006","venue":"Music, Motor Control, and the Brain","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Van Der Steen and Keller, 2013] Maria Christine Van Der Steen and Peter E Keller. The ADaptation and Antic- ipation model (ADAM) of Sensorimotor Synchronization. Frontiers in human neuroscience, 7:253, 2013.","title":"The ADaptation and Anticipation model (ADAM) of Sensorimotor Synchronization","authors":"Der Van, Steen, Maria Keller, Christine Van, Der Steen, Peter E Keller","year":"2013","venue":"Frontiers in human neuroscience","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vercoe, 1984] Barry Vercoe. The Synthetic Performer in the Context of Live Performance. In Proceedings of the International Computer Music Conference (ICMC), pages 199-200, 1984.","title":"The Synthetic Performer in the Context of Live Performance","authors":"Barry Vercoe","year":"1984","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vorberg and Wing, 1996] Dirk Vorberg and Alan Wing. Modeling Variability and Dependence in Timing. In Hand- book of Perception and Action, volume 2, pages 181-262. Elsevier, 1996.","title":"Modeling Variability and Dependence in Timing","authors":"Wing Vorberg, Dirk Vorberg, Alan Wing","year":"1996","venue":"Handbook of Perception and Action","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Widmer et al., 2009] Gerhard Widmer, Sebastian Floss- mann, and Maarten Grachten. YQX plays Chopin. AI magazine, 30(3):35-35, 2009.","title":"YQX plays Chopin","authors":"Gerhard Widmer, Sebastian Flossmann, Maarten Grachten","year":"2009","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wing and Kristofferson, 1973] Alan M Wing and Alfred B Kristofferson. Response Delays and the Timing of Dis- crete Motor r Responses. Perception & Psychophysics, 14(1):5-12, 1973.","title":"Response Delays and the Timing of Discrete Motor r Responses","authors":"Alan M Wing, Alfred B Kristofferson","year":"1973","venue":"Perception & Psychophysics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xia and Dannenberg, 2015] Guangyu Xia and Roger B Dannenberg. Duet Interaction: Learning Musicianship for Automatic Accompaniment. In Proceedings of the Inter- national Conference on New Interfaces for Musical Ex- pression (NIME), pages 259-264, 2015.","title":"Duet Interaction: Learning Musicianship for Automatic Accompaniment","authors":"Dannenberg ; ] Guangyu Xia, Roger B Xia, Dannenberg","year":"2015","venue":"Proceedings of the International Conference on New Interfaces for Musical Expression (NIME)","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xia et al., 2015] Guangyu Xia, Yun Wang, Roger B Dan- nenberg, and Geoffrey Gordon. Spectral Learning for Ex- pressive Interactive Ensemble Music Performance. In Pro- ceedings of the International Society for Music Informa- tion Retrieval Conference (ISMIR), pages 816-822, 2015.","title":"Spectral Learning for Expressive Interactive Ensemble Music Performance","authors":"Guangyu Xia, Yun Wang, Roger B Dannenberg, Geoffrey Gordon","year":"2015","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2023] Zijun Zhou, Justin Christensen, Jor- den A. Cummings, and Janeen D. Loehr. Not just in sync: Relations between partners' actions influence the sense of joint agency during joint action. Consciousness and Cog- nition, 111:103521, 2023.","title":"Not just in sync: Relations between partners' actions influence the sense of joint agency during joint action","authors":"Zijun Zhou, Justin Christensen, Jorden A Cummings, Janeen D Loehr","year":"2023","venue":"Consciousness and Cognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}