---
title: "Online Expectation Maximization for Reinforcement Learning in POMDPs"
authors:
  - "Miao Liu"
  - "Xuejun Liao"
  - "Lawrence Carin"
doi: ""
url: "https://ijcai.org/proceedings/2013/224"
pdf: "https://ijcai.org/proceedings/2013/0224.pdf"
venue: "Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence"
year: 2013
pages: "1501-1506"
track: ""
keywords:
---

# Online Expectation Maximization for Reinforcement Learning in POMDPs

Miao Liu, Xuejun Liao, Lawrence Carin

*Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence*, pages 1501-1506.

## Abstract

We present online nested expectation maximization for model-free reinforcement learning in a POMDP. The algorithm evaluates the policy only in the current learning episode, discarding the episode after the evaluation and memorizing the sufficient statistic, from which the policy is computed in closed-form. As a result, the online algorithm has a time complexity O(n) and a memory complexity O(1), compared to O(n2) and O(n) for the corresponding batch-mode algorithm, where $n$ is the number of learning episodes. The online algorithm, which has a provable convergence, is demonstrated on five benchmark POMDP problems.