{"type":"ScholarlyArticle","title":"Contrastive Learning Is Not Optimal for Quasiperiodic Time Series","authors":[{"name":"Adrian Atienza","slug":"adrian-atienza","given":"Adrian","family":"Atienza","affiliation":"Department of Health Technology , Technical University of Denmark"},{"name":"Jakob Bardram","slug":"jakob-bardram","given":"Jakob","family":"Bardram","affiliation":"Department of Health Technology , Technical University of Denmark"},{"name":"Sadasivan Puthusserypady","slug":"sadasivan-puthusserypady","given":"Sadasivan","family":"Puthusserypady","affiliation":"Department of Health Technology , Technical University of Denmark"}],"abstract":"Despite recent advancements in Self-Supervised Learning (SSL) for Time Series analysis, a noticeable gap persists between the anticipated achievements and actual performance. While these methods have demonstrated formidable generalization capabilities with minimal labels in various domains, their effectiveness in distinguishing between different classes based on a limited number of annotated records is notably lacking. Our hypothesis attributes this bottleneck to the prevalent use of Contrastive Learning, a shared training objective in previous state-of-the-art (SOTA) methods. \r\nBy mandating distinctiveness between representations for negative pairs drawn from separate records, this approach compels the model to encode unique record-based patterns but simultaneously neglects changes occurring across the entire record. To overcome this challenge, we introduce Distilled Embedding for Almost-Periodic Time Series (DEAPS) in this paper, offering a non-contrastive method tailored for quasiperiodic time series, such as electrocardiogram (ECG) data. By avoiding the use of negative pairs, we not only mitigate the model's blindness to temporal changes but also enable the integration of a \"Gradual Loss (L_gra)\" function. This function guides the model to effectively capture dynamic patterns evolving throughout the record. The outcomes are promising, as DEAPS demonstrates a notable improvement of +10% over existing SOTA methods when just a few annotated records are presented to fit a Machine Learning (ML) model based on the learned representation.","doi":"10.24963/ijcai.2024/405","url":"https://ijcai.org/proceedings/2024/405","pdf":"https://ijcai.org/proceedings/2024/0405.pdf","markdown":"https://ijcai.org/proceedings/2024/405.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/405","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":405,"pages":{"first":3661,"last":3668},"keywords":["Multidisciplinary Topics and Applications: MTA: Health and medicine","Machine Learning: ML: Time series and data streams","Machine Learning: ML: Self-supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Bardes et al., 2022] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regular- ization for self-supervised learning, 2022.","title":"Vicreg: Variance-invariance-covariance regularization for self-supervised learning","authors":"Adrien Bardes, Jean Ponce, Yann Lecun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vi- sion transformers, 2021.","title":"Emerging properties in self-supervised vision transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clifford et al., 2017] Gari D Clifford, Chengyu Liu, Ben- jamin Moody, Li-wei H. Lehman, Ikaro Silva, Qiao Li, A E Johnson, and Roger G. Mark. Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge 2017. In 2017 Computing in Car- diology (CinC), pages 1-4, 2017.","title":"Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge","authors":"Chengyu Gari D Clifford, Benjamin Liu, Moody, H Li-Wei, Ikaro Lehman, Qiao Silva, A E Li, Roger G Johnson, Mark","year":"2017","venue":"2017 Computing in Cardiology (CinC)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Diamant et al., 2022] Nathaniel Diamant, Erik Reinertsen, Steven Song, Aaron D. Aguirre, Collin M. Stultz, and Puneet Batra. Patient contrastive learning: A performant, expressive, and practical approach to electrocardiogram modeling. PLOS Computational Biology, 18(2):1-16, 02 2022.","title":"Patient contrastive learning: A performant, expressive, and practical approach to electrocardiogram modeling","authors":"Nathaniel Diamant, Erik Reinertsen, Steven Song, Aaron D Aguirre, Collin M Stultz, Puneet Batra","year":"2022","venue":"PLOS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fan et al., 2021] Fan, Haoyi, Zhang, Fengbin, Wang, Ruidong, Huang, Xunhua, and Zuoyong Li. Semi- supervised time series classification by temporal relation prediction. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3545-3549, 2021.","title":"Semisupervised time series classification by temporal relation prediction","authors":"Fan, Haoyi, Zhang, Fengbin, Wang, Ruidong, Xunhua Huang, Zuoyong Li","year":"2021","venue":"ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Franceschi et al., 2020] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series, 2020.","title":"Unsupervised scalable representation learning for multivariate time series","authors":"Franceschi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldberger et al., 2000] Ary Goldberger, Luís Amaral, L. Glass, Shlomo Havlin, J. Hausdorg, Plamen Ivanov, R. Mark, J. Mietus, G. Moody, Chung-Kang Peng, H. Stanley, and Physiotoolkit Physiobank. Components of a new research resource for complex physiologic signals. PhysioNet, 101, 01 2000.","title":"Components of a new research resource for complex physiologic signals","authors":"Ary Goldberger, Luís Amaral, L Glass, Shlomo Havlin, J Hausdorg, Plamen Ivanov, R Mark, J Mietus, G Moody, Chung-Kang Peng, H Stanley, Physiotoolkit Physiobank","year":"2000","venue":"PhysioNet","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Flo- rent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhao- han Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning, 2020.","title":"Bootstrap your own latent: A new approach to selfsupervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2017] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. [Kiyasseh et al., 2021]","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Dani Kiyasseh, Tingting Zhu, and David A. Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients, 2021.","title":"Clocs: Contrastive learning of cardiac signals across space, time, and patients","authors":"Dani Kiyasseh, Tingting Zhu, David A Clifton","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moody and Mark, 1983] G.B. Moody and R.G. Mark. A new method for detecting atrial fibrillation using r-r inter- vals. Computers in Cardiology, pages 227-230, 1983.","title":"A new method for detecting atrial fibrillation using r-r intervals","authors":"G B Moody, R G Mark","year":"1983","venue":"Computers in Cardiology","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Platt, 2000] John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif., 10, 06 2000.","title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","authors":"John Platt","year":"2000","venue":"Adv. Large Margin Classif","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Quan et al., 1998] Stuart Quan, Barbara Howard, Conrad Iber, James Kiley, F. Nieto, George O'Connor, David Rapoport, Susan Redline, John Robbins, Jonathan Samet, and ‡Patricia Wahl. The sleep heart health study: Design, rationale, and methods. Sleep, 20:1077-85, 01 1998.","title":"The sleep heart health study: Design, rationale, and methods","authors":"Stuart Quan, Barbara Howard, Conrad Iber, James Kiley, F Nieto, O' George, David Connor, Susan Rapoport, John Redline, Jonathan Robbins, Samet, Wahl","year":"1998","venue":"Sleep","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tipping and Bishop, 1999] Michael E. Tipping and Christo- pher M. Bishop. Mixtures of probabilistic principal com- ponent analyzers. Neural Computation, 11(2):443-482, 1999.","title":"Mixtures of probabilistic principal component analyzers","authors":"Michael E Tipping, Christopher M Bishop","year":"1999","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tonekaboni et al., 2021] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learn- ing for time series with temporal neighborhood coding, 2021.","title":"Unsupervised representation learning for time series with temporal neighborhood coding","authors":"Sana Tonekaboni, Danny Eytan, Anna Goldenberg","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wickstrøm et al., 2022] Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recogni- tion Letters, 155:54-61, mar 2022.","title":"Mixing up contrastive learning: Self-supervised representation learning for time series","authors":"Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, Robert Jenssen","year":"2022","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zbontar et al., 2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction, 2021.","title":"Barlow twins: Selfsupervised learning via redundancy reduction","authors":"Jure Zbontar, Li Jing, Ishan Misra, Yann Lecun, Stéphane Deny","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2018] Guo-Qiang Zhang, Licong Cui, Remo Mueller, Shiqiang Tao, Matthew Kim, Michael Rueschman, Sara Mariani, Daniel Mobley, and Susan Redline. The national sleep research resource: Towards a sleep data commons. Journal of the American Medical Informatics Association, pages 572-572, 08 2018.","title":"The national sleep research resource: Towards a sleep data commons","authors":"Guo-Qiang Zhang, Licong Cui, Remo Mueller, Shiqiang Tao, Matthew Kim, Michael Rueschman, Sara Mariani, Daniel Mobley, Susan Redline","year":"2018","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2022] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised con- trastive pre-training for time series via time-frequency consistency, 2022.","title":"Self-supervised contrastive pre-training for time series via time-frequency consistency","authors":"Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, Marinka Zitnik","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}