{"type":"ScholarlyArticle","title":"Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes","authors":[{"name":"Alessandro Trapasso","slug":"alessandro-trapasso","given":"Alessandro","family":"Trapasso","affiliation":""},{"name":"Luca Iocchi","slug":"luca-iocchi","given":"Luca","family":"Iocchi","affiliation":""},{"name":"Fabio Patrizi","slug":"fabio-patrizi","given":"Fabio","family":"Patrizi","affiliation":""}],"abstract":"Many practical decision-making problems involve tasks whose success depends on the entire system history, rather than on achieving a state with desired properties. Markovian Reinforcement Learning (RL) can be inadequate for such tasks, while modeling them as non-Markovian reward decision processes (NMRDPs) enables agents to handle temporal dependencies.\r\nExisting approaches offer limited formal guarantees on both (near-)optimality and sample efficiency. We address both issues with QR-max, a novel model-based algorithm for NMRDPs with discrete actions that factorizes Markovian transition learning from non-Markovian reward handling via reward machines. To our knowledge, this is the first model-based RL algorithm for discrete-action NMRDPs that leverages this factorization to obtain PAC convergence to ε-optimal policies with polynomial sample complexity.\r\nWe then extend QR-max to continuous state spaces with Bucket-QR-max, a SimHash-based discretizer that preserves the same factorized structure and achieves fast and stable learning without manual gridding or function approximation. We experimentally compare our method with state-of-the-art model-based RL approaches on environments of increasing complexity, showing substantially improved sample efficiency and greater robustness in finding optimal policies.","doi":"10.24963/ijcai.2026/546","url":"https://ijcai.org/proceedings/2026/546","pdf":"https://ijcai.org/proceedings/2026/0546.pdf","markdown":"https://ijcai.org/proceedings/2026/546.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/546","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Machine Learning","track":"Main Track","number":546,"pages":{"first":4903,"last":4911},"keywords":["Machine Learning: Model-based and model learning reinforcement learning","Machine Learning: Reinforcement learning","Machine Learning: Learning theory"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}