{"type":"ScholarlyArticle","title":"Q-Learning-Based Model Predictive Variable Impedance Control for Physical Human-Robot Collaboration (Extended Abstract)","authors":[{"name":"Loris Roveda","slug":"loris-roveda","given":"Loris","family":"Roveda","affiliation":"Istituto Dalle Molle di studi sull'Intelligenza Artificiale (IDSIA) , Scuola Universitaria Professionale della Svizzera Italiana (SUPSI) , Università della Svizzera italiana (USI) , via la Santa 1 , 6962 , Lugano , Switzerland"},{"name":"Andrea Testa","slug":"andrea-testa","given":"Andrea","family":"Testa","affiliation":"Politecnico di Milano , Department of Mechanical Engineering , via La Masa 1 , 20156 , Milano , Italy"},{"name":"Asad Ali Shahid","slug":"asad-ali-shahid","given":"Asad Ali","family":"Shahid","affiliation":"Istituto Dalle Molle di studi sull'Intelligenza Artificiale (IDSIA) , Scuola Universitaria Professionale della Svizzera Italiana (SUPSI) , Università della Svizzera italiana (USI) , via la Santa 1 , 6962 , Lugano , Switzerland"},{"name":"Francesco Braghin","slug":"francesco-braghin","given":"Francesco","family":"Braghin","affiliation":"Politecnico di Milano , Department of Mechanical Engineering , via La Masa 1 , 20156 , Milano , Italy"},{"name":"Dario Piga","slug":"dario-piga","given":"Dario","family":"Piga","affiliation":"Istituto Dalle Molle di studi sull'Intelligenza Artificiale (IDSIA) , Scuola Universitaria Professionale della Svizzera Italiana (SUPSI) , Università della Svizzera italiana (USI) , via la Santa 1 , 6962 , Lugano , Switzerland"}],"abstract":"Physical human-robot collaboration is increasingly required in many contexts. To implement an effective collaboration, the robot should be able to recognize the human’s intentions and guarantee safe and adaptive behavior along the intended motion directions. The robot-control strategies with such attributes are particularly demanded in the industrial field. Indeed, with this aim, this work proposes a Q-Learning-based Model Predictive Variable Impedance Control (Q-LMPVIC) to assist the operators in physical human-robot collaboration (pHRC) tasks. A Cartesian impedance control loop is designed to implement decoupled compliant robot dynamics. The impedance control parameters (i.e., setpoint and damping parameters) are then optimized online in order to maximize the performance of the pHRC. For this purpose, an ensemble of neural networks is designed to learn the modeling of the human-robot interaction dynamics while capturing the associated uncertainties. The derived modeling is then exploited by the model predictive controller (MPC), enhanced with stability guarantees by means of Lyapunov constraints. The MPC is solved by making use of a Q-Learning method that, in its online implementation, uses an actor-critic algorithm to approximate the exact solution. Indeed, the Q-learning method provides an accurate and highly efficient solution (in terms of computational time and resources). The proposed approach has been validated through experimental tests, in which a Franka EMIKA panda robot has been used as a test platform.","doi":"10.24963/ijcai.2023/790","url":"https://ijcai.org/proceedings/2023/790","pdf":"https://ijcai.org/proceedings/2023/0790.pdf","markdown":"https://ijcai.org/proceedings/2023/790.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/790","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":790,"pages":{"first":6959,"last":6963},"keywords":["Robotics: ROB: Human robot interaction","Robotics: ROB: Learning in robotics","Robotics: ROB: Behavior and control","Robotics: ROB: Applications"],"videos":[],"references":[{"position":1,"raw":"[Cremer et al., 2019] Sven Cremer, Sumit Kumar Das, In- dika B Wijayasinghe, Dan O Popa, and Frank L Lewis. Model-free online neuroadaptive controller with intent es- timation for physical human-robot interaction. IEEE Transactions on Robotics, 36(1):240-253, 2019.","title":"Model-free online neuroadaptive controller with intent estimation for physical human-robot interaction","authors":"Sven Cremer, Sumit Kumar Das, Dan O Indika B Wijayasinghe, Frank L Popa, Lewis","year":"2019","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dimeas and Aspragathos, 2015] Fotios Dimeas and Nikos Aspragathos. Reinforcement learning of variable admit- tance control for human-robot co-manipulation. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1011-1016. IEEE, 2015.","title":"Fotios Dimeas and Nikos Aspragathos. Reinforcement learning of variable admittance control for human-robot co-manipulation","authors":"Aspragathos Dimeas","year":"2015","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fragapane et al., 2020] Giuseppe Fragapane, Dmitry Ivanov, Mirco Peron, Fabio Sgarbossa, and Jan Ola Strandhagen. Increasing flexibility and productivity in industry 4.0 production networks with autonomous mo- bile robots and smart intralogistics. Annals of operations research, pages 1-19, 2020.","title":"Increasing flexibility and productivity in industry 4.0 production networks with autonomous mobile robots and smart intralogistics","authors":"Giuseppe Fragapane, Dmitry Ivanov, Mirco Peron, Fabio Sgarbossa, Jan Ola Strandhagen","year":"2020","venue":"Annals of operations research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Galin and Meshcheryakov, 2020] Rinat R Galin and Ro- man V Meshcheryakov. Human-robot interaction effi- ciency and human-robot collaboration. In Robotics: In- dustry 4.0 Issues & New Intelligent Control Paradigms, pages 55-63. Springer, 2020.","title":"Human-robot interaction efficiency and human-robot collaboration","authors":"Meshcheryakov ; R Galin, Roman V Galin, Meshcheryakov","year":"2020","venue":"Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Makris, 2021] Sotiris Makris. Cooperating Robots for Flex- ible Manufacturing. Springer, 2021.","title":"Cooperating Robots for Flexible Manufacturing","authors":"Sotiris Makris","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Mauri et al., 2019] Alessandro Mauri, Jacopo Lettori, Gio- vanni Fusi, Davide Fausti, Maurizio Mor, Francesco Braghin, Giovanni Legnani, and Loris Roveda. Mechani- cal and control design of an industrial exoskeleton for ad- vanced human empowering in heavy parts manipulation tasks. Robotics, 8(3):65, 2019.","title":"Mechanical and control design of an industrial exoskeleton for advanced human empowering in heavy parts manipulation tasks","authors":"Alessandro Mauri, Jacopo Lettori, Giovanni Fusi, Davide Fausti, Maurizio Mor, Francesco Braghin, Giovanni Legnani, Loris Roveda","year":"2019","venue":"Robotics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Medina et al., 2019] José R Medina, Hendrik Börner, Satoshi Endo, and Sandra Hirche. Impedance-based gaus- sian processes for modeling human motor behavior in physical and non-physical interaction. IEEE Transactions on Biomedical Engineering, 66(9):2499-2511, 2019.","title":"Impedance-based gaussian processes for modeling human motor behavior in physical and non-physical interaction","authors":"Hendrik José R Medina, Satoshi Börner, Sandra Endo, Hirche","year":"2019","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Peternel et al., 2017] Luka Peternel, Nikos Tsagarakis, and Arash Ajoudani. A human-robot co-manipulation ap- proach based on human sensorimotor information. IEEE Transactions on Neural Systems and Rehabilitation Engi- neering, 25(7):811-822, 2017.","title":"Luka Peternel, Nikos Tsagarakis, and Arash Ajoudani. A human-robot co-manipulation approach based on human sensorimotor information","authors":"Peternel","year":"2017","venue":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Roveda et al., 2018a] Loris Roveda, Nicola Castaman, Ste- fano Ghidoni, Paolo Franceschi, Nicolò Boscolo, Enrico Pagello, and Nicola Pedrocchi. Human-robot cooperative interaction control for the installation of heavy and bulky components. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 339-344. IEEE, 2018.","title":"Human-robot cooperative interaction control for the installation of heavy and bulky components","authors":"Roveda","year":"2018","venue":"IEEE International Conference on Systems, Man, and Cybernetics (SMC)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Roveda et al., 2018b] Loris Roveda, Shaghavezh Haghshenas, Alessio Prini, Tito Dinon, Nicola Pedrocchi, Francesco Braghin, and Lorenzo Molinari Tosatti. Fuzzy impedance control for enhancing capabilities of humans in onerous tasks execution. In 2018 15th international conference on ubiquitous robots (UR), pages 406-411. IEEE, 2018.","title":"Fuzzy impedance control for enhancing capabilities of humans in onerous tasks execution","authors":"Roveda","year":"2018","venue":"15th international conference on ubiquitous robots (UR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Roveda et al., 2019] Loris Roveda, Shaghayegh Haghshenas, Marco Caimmi, Nicola Pedrocchi, and Lorenzo Molinari Tosatti. Assisting operators in heavy industrial tasks: on the design of an optimized cooperative impedance fuzzy-controller with embedded safety rules. Frontiers in Robotics and AI, 6:75, 2019.","title":"Assisting operators in heavy industrial tasks: on the design of an optimized cooperative impedance fuzzy-controller with embedded safety rules","authors":"Loris Roveda, Shaghayegh Haghshenas, Marco Caimmi, Nicola Pedrocchi, Lorenzo Molinari, Tosatti","year":"2019","venue":"Frontiers in Robotics and AI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Roveda et al., 2020] Loris Roveda, Jeyhoon Maskani, Paolo Franceschi, Arash Abdi, Francesco Braghin, Lorenzo Molinari Tosatti, and Nicola Pedrocchi. Model- based reinforcement learning variable impedance control for human-robot collaboration. Journal of Intelligent & Robotic Systems, 100(2):417-433, 2020.","title":"Modelbased reinforcement learning variable impedance control for human-robot collaboration","authors":"Loris Roveda, Jeyhoon Maskani, Paolo Franceschi, Arash Abdi, Francesco Braghin, Lorenzo Molinari Tosatti, Nicola Pedrocchi","year":"2020","venue":"Journal of Intelligent & Robotic Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Roveda et al., 2021a] Loris Roveda, Beatrice Maggioni, Elia Marescotti, Asad Shahid, Andrea Maria Zanchettin, Alberto Bemporad, and Dario Piga. Pairwise preferences- based optimization of a path-based velocity planner in robotic sealing tasks. IEEE Robotics and Automation Let- ters, 2021.","title":"Pairwise preferencesbased optimization of a path-based velocity planner in robotic sealing tasks","authors":"Roveda","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Roveda et al., 2021b] Loris Roveda, Mauro Magni, Mar- tina Cantoni, Dario Piga, and Giuseppe Bucca. Human- robot collaboration in sensorless assembly task learning enhanced by uncertainties adaptation via bayesian opti- mization. Robotics and Autonomous Systems, 136:103711, 2021.","title":"Humanrobot collaboration in sensorless assembly task learning enhanced by uncertainties adaptation via bayesian optimization","authors":"Roveda","year":"2021","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Roveda et al., 2022] Loris Roveda, Andrea Testa, Asad Ali Shahid, Francesco Braghin, and Dario Piga. Q-learning- based model predictive variable impedance control for physical human-robot collaboration. Artificial Intelli- gence, 312:103771, 2022.","title":"Q-learningbased model predictive variable impedance control for physical human-robot collaboration","authors":"Loris Roveda, Andrea Testa, Asad Ali Shahid, Francesco Braghin, Dario Piga","year":"2022","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vicentini et al., 2020] Federico Vicentini, Nicola Pedroc- chi, Manuel Beschi, Matteo Giussani, Niccolò Iannacci, Paolo Magnoni, Stefania Pellegrinelli, Loris Roveda, En- rico Villagrossi, Mehrnoosh Askarpour, et al. Piros: Co- operative, safe and reconfigurable robotic companion for cnc pallets load/unload stations. In Bringing Innovative Robotic Technologies from Research Labs to Industrial End-users, pages 57-96. Springer, 2020.","title":"Piros: Cooperative, safe and reconfigurable robotic companion for cnc pallets load/unload stations","authors":"Federico Vicentini, Nicola Pedrocchi, Manuel Beschi, Matteo Giussani, Niccolò Iannacci, Paolo Magnoni, Stefania Pellegrinelli, Loris Roveda, Enrico Villagrossi, Mehrnoosh Askarpour","year":"2020","venue":"Bringing Innovative Robotic Technologies from Research Labs to Industrial End-users","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Zhang et al., 2020] Hao Zhang, Shaoyuan Li, and Yi Zheng. Q-learning-based model predictive control for nonlin- ear continuous-time systems. Industrial & Engineering Chemistry Research, 59(40):17987-17999, 2020.","title":"Q-learning-based model predictive control for nonlinear continuous-time systems","authors":"Hao Zhang, Shaoyuan Li, Yi Zheng","year":"2020","venue":"Industrial & Engineering Chemistry Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}