Temporal Coherence and Prediction Decay in TD Learning

Abstract

This paper describes improvements to the temporal difference TD(A.) learning method . The standard form of the TD(A.) method has the problem that two control parameters, learning rate and temporal discount, need to be chosen appropriately. These parameters can have a major effect on performance, particularly the learning rate parameter, which af fects the stability of the process as well as the number of observations required . Our extension to the TD(A.) algorithm automatically sets and subsequently adjusts these parameters. The le arning rate adjustment is based on a new concept we call temporal coherence (TC). The experiments reported here compare the extended TD(A.) algorithm performance with human-chosen parameters and with an earlier method for learning rate adjustment, in a complex game domain . The learning task was that of learning the relative values of pieces, without any initial domain-specific knowledge, and from self-play only. The results show that the improved method leads to better learning (i.e. faster and less subject to the effects of noise), than the selection of human-chosen values for the control parameters, and a comparison method.