{"type":"ScholarlyArticle","title":"Reinforcement Learning in Distributed Domains: Beyond Team Games","authors":[{"name":"David H. Wolpert","slug":"david-h-wolpert","given":"David H.","family":"Wolpert","affiliation":""},{"name":"Joseph Sill","slug":"joseph-sill","given":"Joseph","family":"Sill","affiliation":""},{"name":"Kagan Tumer","slug":"kagan-tumer","given":"Kagan","family":"Tumer","affiliation":""}],"abstract":"Using a distributed algorithm rather than a centralized one can b e extremely b ene cial in large search problems. In addition, the incorp oration of machine learning techniques like Reinforcement Learning (RL) into search algorithms has often b een found to improve their p erformance. In this article we investigate a search algorithm that combines these prop erties by employing RL in a distributed manner, essentially using the team game approach. We then present bi-utility search, which interleaves our distributed algorithm with (centralized) simulated annealing, by using the distributed algorithm to guide the exploration step of the simulated annealing. We investigate using these algorithms in the domain of minimizing the loss of imp ortance-weighted communication data traversing a constellations of communication satellites. To do this we intro duce the idea of running these algorithms \\on top\" of an underlying, learning-free routing algorithm. They do this by having the actions of the distributed learners b e the intro duction of virtual \\ghost\" tra\u000ec into the decision-making of the underlying routing algorithm, tra\u000ec that \\misleads\" the routing algorithm in a way that actually improves p erformance. We nd that using our original distributed RL algorithm to set ghost tra\u000ec improves p erformance, and that bi-utility search | a semi-distributed search algorithm that is widely applicable | substantially outp erforms b oth that distributed RL algorithm and (centralized) simulated annealing in our problem domain.","doi":null,"url":"https://ijcai.org/proceedings/2001/112","pdf":"https://ijcai.org/proceedings/2001/0112.pdf","markdown":"https://ijcai.org/proceedings/2001/112.md","bibtex":"https://ijcai.org/proceedings/2001/bibtex/112","volume":{"slug":"2001","title":"Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2001","url":"https://ijcai.org/proceedings/2001/","year":2001,"location":"Seattle, Washington, USA","dates":"","editors":"","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":112,"pages":{"first":819,"last":823},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}