{"type":"ScholarlyArticle","title":"Temporal Coherence and Prediction Decay in TD Learning","authors":[{"name":"Donald F. Beal","slug":"donald-f-beal","given":"Donald F.","family":"Beal","affiliation":""},{"name":"Martin C. Smith","slug":"martin-c-smith","given":"Martin C.","family":"Smith","affiliation":""}],"abstract":"This paper describes improvements to the temporal difference TD(A.) learning method . The standard form of the TD(A.) method has the problem that two control parameters, learning rate and temporal discount, need to be chosen appropriately. These parameters can have a major effect on performance, particularly the learning rate parameter, which af fects the stability of the process as well as the number of observations required . Our extension to the TD(A.) algorithm automatically sets and subsequently adjusts these parameters. The le arning rate adjustment is based on a new concept we call temporal coherence (TC). The experiments reported here compare the extended TD(A.) algorithm performance with human-chosen parameters and with an earlier method for learning rate adjustment, in a complex game domain . The learning task was that of learning the relative values of pieces, without any initial domain-specific knowledge, and from self-play only. The results show that the improved method leads to better learning (i.e. faster and less subject to the effects of noise), than the selection of human-chosen values for the control parameters, and a comparison method.","doi":null,"url":"https://ijcai.org/proceedings/1999/81","pdf":"https://ijcai.org/proceedings/1999/0081.pdf","markdown":"https://ijcai.org/proceedings/1999/81.md","bibtex":"https://ijcai.org/proceedings/1999/bibtex/81","volume":{"slug":"1999","title":"Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1999","url":"https://ijcai.org/proceedings/1999/","year":1999,"location":"Stockholm, Sweden","dates":"","editors":"Thomas Dean","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Computer Game Playing","track":"Computer Game Playing","number":81,"pages":{"first":564,"last":568},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}