{"type":"ScholarlyArticle","title":"Continuous-Time Reward Machines","authors":[{"name":"Amin Falah","slug":"amin-falah","given":"Amin","family":"Falah","affiliation":""},{"name":"Shibashis Guha","slug":"shibashis-guha","given":"Shibashis","family":"Guha","affiliation":""},{"name":"Ashutosh Trivedi","slug":"ashutosh-trivedi","given":"Ashutosh","family":"Trivedi","affiliation":""}],"abstract":"Reinforcement Learning (RL) is a sampling-based method for sequential decision-making, in which a learning agent iteratively converges toward an optimal policy by leveraging feedback from the environment in the form of scalar reward signals.\r\nWhile timing information is often abstracted in discrete-time domains, time-critical learning applications—such as queuing systems, population processes, and manufacturing systems—are naturally modeled as Continuous-Time Markov Decision Processes (CTMDPs).\r\nSince the seminal work of Bradtke and Duff, model-free RL for CTMDPs has become well-understood. However, in many practical applications, practitioners possess high-quality information about system rates derived from traditional queuing theory, which learning agents could potentially exploit to accelerate convergence. Despite this, classical RL algorithms for CTMDPs typically re-learn these parameters through sampling.\r\nIn this work, we propose continuous-time reward machines (CTRMs), a novel framework that embeds reward functions and real-time state-action dynamics into a unified structure. \r\nCTRMs enable RL agents to effectively navigate dense-time environments while leveraging reward shaping and counterfactual experiences for accelerated learning. \r\nOur empirical results demonstrate CTRMs' ability to improve learning efficiency in time-critical environments.","doi":"10.24963/ijcai.2025/563","url":"https://ijcai.org/proceedings/2025/563","pdf":"https://ijcai.org/proceedings/2025/0563.pdf","markdown":"https://ijcai.org/proceedings/2025/563.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/563","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":563,"pages":{"first":5056,"last":5064},"keywords":["Planning and Scheduling: PS: Learning in planning and scheduling","AI Ethics, Trust, Fairness: ETF: Explainability and interpretability","Machine Learning: ML: Reinforcement learning","Planning and Scheduling: PS: Markov decisions processes"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}