{"type":"ScholarlyArticle","title":"Reinforcement Learning Framework for Deep Brain Stimulation Study","authors":[{"name":"Dmitrii Krylov","slug":"dmitrii-krylov","given":"Dmitrii","family":"Krylov","affiliation":"Skolkovo Institute of Science and Technology , Bolshoy blvd. 30/1 , Moscow , 121205 , Russia"},{"name":"Remi Tachet des Combes","slug":"remi-tachet-des-combes","given":"Remi","family":"Tachet des Combes","affiliation":"Microsoft Research Lab , 550-2000 McGill College Ave , Montréal H3A 3H3 , Canada"},{"name":"Romain Laroche","slug":"romain-laroche","given":"Romain","family":"Laroche","affiliation":"Microsoft Research Lab , 550-2000 McGill College Ave , Montréal H3A 3H3 , Canada"},{"name":"Michael Rosenblum","slug":"michael-rosenblum","given":"Michael","family":"Rosenblum","affiliation":"University of Potsdam , Karl-Liebknecht-Str. 24/25 , 14476 Potsdam-Golm , Germany"},{"name":"Dmitry V. Dylov","slug":"dmitry-v-dylov","given":"Dmitry V.","family":"Dylov","affiliation":"Skolkovo Institute of Science and Technology , Bolshoy blvd. 30/1 , Moscow , 121205 , Russia"}],"abstract":"Malfunctioning neurons in the brain sometimes operate synchronously, reportedly causing many neurological diseases, e.g. Parkinson’s. Suppression and control of this collective synchronous activity are therefore of great importance for neuroscience, and can only rely on limited engineering trials due to the need to experiment with live human brains. We present the first Reinforcement Learning (RL) gym framework that emulates this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. We successfully suppress synchrony via RL for three pathological signaling regimes, characterize the framework’s stability to noise, and further remove the unwanted oscillations by engaging multiple PPO agents.","doi":"10.24963/ijcai.2020/394","url":"https://ijcai.org/proceedings/2020/394","pdf":"https://ijcai.org/proceedings/2020/0394.pdf","markdown":"https://ijcai.org/proceedings/2020/394.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/394","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":394,"pages":{"first":2847,"last":2854},"keywords":["Multidisciplinary Topics and Applications: Biology and Medicine","Machine Learning: Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25182/file_25182.m3u8","length_sec":305,"provider":"hls"}],"references":[{"position":1,"raw":"[Benabid et al., 1991] A.L. Benabid, P. Pollak, C. Gervason, D. Hoffmann, D.M. Gao, M. Hommel, J.E. Perret, and J. De Rougemont. Long-term suppression of tremor by chronic stimulation of the ventral intermediate thalamic nucleus. Lancet, 337:403-406, 1991.","title":"Long-term suppression of tremor by chronic stimulation of the ventral intermediate thalamic nucleus","authors":"A L Benabid, P Pollak, C Gervason, D Hoffmann, D M Gao, M Hommel, J E Perret, J De Rougemont","year":"1991","venue":"Lancet","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benabid et al., 2009] A.L. Benabid, S. Chabardes, J. Mitro- fanis, and P. Pollak. Deep brain stimulation of the sub- thalamic nucleus for the treatment of Parkinson's disease. Lancet Neurol., 8(1):67-81, 2009.","title":"Deep brain stimulation of the subthalamic nucleus for the treatment of Parkinson's disease","authors":"A L Benabid, S Chabardes, J Mitrofanis, P Pollak","year":"2009","venue":"Lancet Neurol","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blandini and Armentero, 2012] F. Blandini and M-Th. Ar- mentero. Animal models of Parkinson's disease. The FEBS Journal, 279(7):1156-1166, 2012.","title":"Animal models of Parkinson's disease","authors":"F Blandini, M-Th","year":"2012","venue":"The FEBS Journal","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonhoeffer, 1948] K. F. Bonhoeffer. Activation of passive iron as a model for the extraction of nerve. The Journal of General Physiology, 32(1):69-91, 09 1948.","title":"Activation of passive iron as a model for the extraction of nerve","authors":"K F Bonhoeffer","year":"1948","venue":"The Journal of General Physiology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pet- tersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. [Canavier, 2006]","title":"Openai gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"C. C. Canavier. Phase response curve. Scholarpedia, 1(12):1332, 2006. revision #27615.","title":"Phase response curve","authors":"C C Canavier","year":"2006","venue":"Scholarpedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cestnik and Abel, 2019] R. Cestnik and M. Abel. Inferring the dynamics of oscillatory systems using recurrent neural networks. Chaos, 29:063128, 2019.","title":"Inferring the dynamics of oscillatory systems using recurrent neural networks","authors":"R Cestnik, M Abel","year":"2019","venue":"Chaos","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deniau et al., 2010] J.-M. Deniau, B. Degos, C. Bosch, and N. Maurice. Deep brain stimulation mechanisms: beyond the concept of local functional inhibition. European Jour- nal of Neuroscience, 32(7):1080-1091, 2010.","title":"Deep brain stimulation mechanisms: beyond the concept of local functional inhibition","authors":"J.-M Deniau, B Degos, C Bosch, N Maurice","year":"2010","venue":"European Journal of Neuroscience","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gerstner et al., 2002] W. Gerstner, W. Kistler, and Werner M. Spiking neuron models: Single neurons, populations, plasticity. Spiking Neuron Models: Single Neurons, Popu- lations, Plasticity, 01 2002.","title":"Spiking neuron models: Single neurons, populations, plasticity","authors":"W Gerstner, W Kistler, Werner M","year":"2002","venue":"Spiking Neuron Models: Single Neurons, Populations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gielen and Moss, 2001] S. Gielen and F. Moss. Preface to volume 4 neuro-informatics and neural modelling. In F. Moss and S. Gielen, editors, Neuro-Informatics and Neural Modelling, volume 4 of Handbook of Biological Physics, pages ix -xi. North-Holland, 2001.","title":"Preface to volume 4 neuro-informatics and neural modelling","authors":"S Gielen, F Moss","year":"2001","venue":"Neuro-Informatics and Neural Modelling","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Golomb et al., 2001] D. Golomb, D. Hansel, and G. Mato. Chapter 21. In F. Moss and S. Gielen, editors, Neuro- Informatics and Neural Modelling, volume 4 of Handbook of Biological Physics, pages 887 -968.","title":"Chapter 21","authors":"D Golomb, D Hansel, G Mato","year":null,"venue":"Neuro-Informatics and Neural Modelling","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"N-H, 2001.","title":null,"authors":"N-H","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gradinaru et al., 2009] V. Gradinaru, M. Mogri, K. R. Thompson, J. M. Henderson, and K. Deisseroth. Optical deconstruction of Parkinsonian neural circuitry. Science, 324(5925):354-359, 2009.","title":"Optical deconstruction of Parkinsonian neural circuitry","authors":"V Gradinaru, M Mogri, K R Thompson, J M Henderson, K Deisseroth","year":"2009","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hansel and Sompolinsky, 1992] D. Hansel and H. Som- polinsky. Synchronization and computation in a chaotic neural network. Phys. Rev. Lett., 68:718-721, Feb 1992.","title":"Synchronization and computation in a chaotic neural network","authors":"D Hansel, H Sompolinsky","year":"1992","venue":"Phys. Rev. Lett","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Herzog et al., 2018] S. Herzog, F. Wörgötter, and U. Parlitz. Data-driven modeling and prediction of complex spatio- temporal dynamics in excitable media. Frontiers in Ap- plied Mathematics and Statistics, 4:60, 2018.","title":"Data-driven modeling and prediction of complex spatiotemporal dynamics in excitable media","authors":"S Herzog, F Wörgötter, U Parlitz","year":"2018","venue":"Frontiers in Applied Mathematics and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hill et al., 2018] A. Hill, A. Raffin, M. Ernestus, A. Gleave, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu. Stable baselines. https://github.com/hill-a/ stable-baselines, 2018.","title":"Stable baselines","authors":"A Hill, A Raffin, M Ernestus, A Gleave, R Traore, P Dhariwal, C Hesse, O Klimov, A Nichol, M Plappert, A Radford, J Schulman, S Sidor, Y Wu","year":null,"venue":null,"doi":null,"url":"https://github.com/hill-a/stable-baselines,2018","internal_url":null},{"position":17,"raw":"[Hindmarsh and Rose, 1984] J. L. Hindmarsh and R. M. Rose. A model for neuronal bursting using three coupled first order differential equations. Proc. Roy. Soc. London Ser. B, 221:87, 1984.","title":"A model for neuronal bursting using three coupled first order differential equations","authors":"J L Hindmarsh, R M Rose","year":"1984","venue":"Proc. Roy. Soc. London Ser. B","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Johnson et al., 2008] M. D. Johnson, S. Miocinovic, C. C. McIntyre, and J. L. Vitek. Mechanisms and targets of deep brain stimulation in movement disorders. Neurotherapeu- tics, 5(2):294-308, Apr 2008.","title":"Mechanisms and targets of deep brain stimulation in movement disorders","authors":"M D Johnson, S Miocinovic, C C Mcintyre, J L Vitek","year":"2008","venue":"Neurotherapeutics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krylov et al., 2020] Dmitrii Krylov, Dmitry V. Dylov, and Michael Rosenblum. Reinforcement learning for suppres- sion of collective activity in oscillatory ensembles. Chaos, 30(3), 2020.","title":"Reinforcement learning for suppression of collective activity in oscillatory ensembles","authors":"Dmitrii Krylov, V Dmitry, Michael Dylov, Rosenblum","year":"2020","venue":"Chaos","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kühn and Volkmann, 2017] A.A. Kühn and J. Volkmann. Innovations in deep brain stimulation methodology. Mov. Disorders., 32(1):11, 2017.","title":"Innovations in deep brain stimulation methodology","authors":"A A Kühn, J Volkmann","year":"2017","venue":"Mov. Disorders","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2013] W. Lin, Y. Pu, Y. Guo, and J. Kurths. Oscil- lation suppression and synchronization: Frequencies de- termine the role of control with time delays. EPL (Euro- physics Letters), 102(2):20003, apr 2013.","title":"Oscillation suppression and synchronization: Frequencies determine the role of control with time delays","authors":"W Lin, Y Pu, Y Guo, J Kurths","year":"2013","venue":"Europhysics Letters)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pathak et al., 2018] J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott. Model-free prediction of large spatiotempo- rally chaotic systems from data: A reservoir computing approach. Phys. Rev. Lett., 120:024102, 2018.","title":"Model-free prediction of large spatiotempo-rally chaotic systems from data: A reservoir computing approach","authors":"J Pathak, B Hunt, M Girvan, Z Lu, E Ott","year":"2018","venue":"Phys. Rev. Lett","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pikovsky and Ruffo, 1999] A. Pikovsky and S. Ruffo. Finite-size effects in a population of interacting oscillators. Phys. Rev. E, 59(2):1633-1636, 1999.","title":"Finite-size effects in a population of interacting oscillators","authors":"A Pikovsky, S Ruffo","year":"1999","venue":"Phys. Rev. E","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Popovych et al., 2005] O. Popovych, Ch. Hauptmann, and P. A. Tass. Effective desynchronization by nonlinear de- layed feedback. Phys. Rev. Lett., 94:164102, 2005.","title":"Effective desynchronization by nonlinear delayed feedback","authors":"O Popovych, Ch Hauptmann, P A Tass","year":"2005","venue":"Phys. Rev. Lett","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Quade et al., 2018] M. Quade, M. Abel, J. Nathan Kutz, and S. L. Brunton. Sparse identification of nonlinear dynamics for rapid model recovery. Chaos, 28:063116, 2018.","title":"Sparse identification of nonlinear dynamics for rapid model recovery","authors":"M Quade, M Abel, J Nathan Kutz, S L Brunton","year":"2018","venue":"Chaos","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rosenblum and Pikovsky, 2004a] M. G. Rosenblum and A. S. Pikovsky. Controlling synchrony in ensemble of globally coupled oscillators. PRL, 92:114102, 2004.","title":"Controlling synchrony in ensemble of globally coupled oscillators","authors":"Pikovsky Rosenblum, M G Rosenblum, A S Pikovsky","year":"2004","venue":"PRL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rosenblum and Pikovsky, 2004b] M. G. Rosenblum and A. S. Pikovsky. Delayed feedback control of collective synchrony: An approach to suppression of pathological brain rhythms. Phys. Rev. E., 70:041904, 2004.","title":"Delayed feedback control of collective synchrony: An approach to suppression of pathological brain rhythms","authors":"Pikovsky Rosenblum, M G Rosenblum, A S Pikovsky","year":"2004","venue":"Phys. Rev. E","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Su et al., 2018] D. Su, H. Chen, W. Hu, Y. Liu, Z. Wang, X. Wang, G. Liu, H. Ma, J. Zhou, and T. Feng. Frequency- dependent effects of subthalamic deep brain stimulation on motor symptoms in Parkinson's disease: a meta-analysis of controlled trials. Scientific reports, 8(1):1-9, 2018.","title":"Frequencydependent effects of subthalamic deep brain stimulation on motor symptoms in Parkinson's disease: a meta-analysis of controlled trials","authors":"D Su, H Chen, W Hu, Y Liu, Z Wang, X Wang, G Liu, H Ma, J Zhou, T Feng","year":"2018","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton and Barto, 2018] R. S. Sutton and A. G. Barto. Re- inforcement Learning: An Introduction. MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"R S Sutton, A G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tass, 2001] P. A. Tass. Effective desynchronization by means of double-pulse phase resetting. Europhys Lett., 53(1):15-21, 2001.","title":"Effective desynchronization by means of double-pulse phase resetting","authors":"P A Tass","year":"2001","venue":"Europhys Lett","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tysnes and Storstein, 2017] Ole-Bjørn Tysnes and Anette Storstein. Epidemiology of Parkinson's disease. Journal of Neural Transmission, 124(8):901-905, Aug 2017.","title":"Ole-Bjørn Tysnes and Anette Storstein. Epidemiology of Parkinson's disease","authors":"Storstein Tysnes","year":"2017","venue":"Journal of Neural Transmission","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Weng et al., 2019] T. Weng, H. Yang, C. Gu, J. Zhang, and M. Small. Synchronization of chaotic systems and their machine-learning models. Phys. Rev. E, 99:042203, 2019.","title":"Synchronization of chaotic systems and their machine-learning models","authors":"T Weng, H Yang, C Gu, J Zhang, M Small","year":"2019","venue":"Phys. Rev. E","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yeo and Melnyk, 2019] K. Yeo and I. Melnyk. Deep learn- ing algorithm for data-driven simulation of noisy dynami- cal system. J. of Comp. Physics, 376:1212-1231, 2019.","title":"Deep learning algorithm for data-driven simulation of noisy dynamical system","authors":"K Yeo, I Melnyk","year":"2019","venue":"J. of Comp. Physics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zimmermann and Parlitz, 2018] R. S. Zimmermann and U. Parlitz. Observing spatio-temporal dynamics of excitable media using reservoir computing. Chaos, 28(4):043118, 2018.","title":"Observing spatio-temporal dynamics of excitable media using reservoir computing","authors":"R S Zimmermann, U Parlitz","year":"2018","venue":"Chaos","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}