{"type":"ScholarlyArticle","title":"Empirical Evidence and Analysis of a Critical Pitfall in Reward Learning from Human Feedback","authors":[{"name":"Taha Shaheen","slug":"taha-shaheen","given":"Taha","family":"Shaheen","affiliation":""},{"name":"Stephen G. West","slug":"stephen-g-west","given":"Stephen G.","family":"West","affiliation":""},{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":""}],"abstract":"Reward learning via human feedback is a crucial capability for beneficial AI. Current methods are built on decision-making theories that assume a matched dynamics model between the learning agent and the feedback provider. However, humans often form imperfect internal dynamics models, and their feedback reflects these misconceptions. While this relationship has long been hypothesised, its manifestation in sequential decision-making remains largely an assumption. Our work provides the first comprehensive empirical investigation of this relationship through a randomized controlled trial (N=211). We followed a two-stage design where we first initialized the participants' understanding of the  dynamics in a grid-world navigation domain and then manipulated it using text-based instructions. Causal mediation analysis revealed that humans' internal models play a mediating role in feedback behaviour. We show that this relationship is invariant across visual contexts and is robust to three common feedback types: pairwise preferences, trajectory corrections, and off-switch interventions. These findings confirm a critical limitation of current reward learning methods and establish the missing psychological foundation for approaches that incorporate dynamics understanding.","doi":"10.24963/ijcai.2026/37","url":"https://ijcai.org/proceedings/2026/37","pdf":"https://ijcai.org/proceedings/2026/0037.pdf","markdown":"https://ijcai.org/proceedings/2026/37.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/37","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":37,"pages":{"first":323,"last":331},"keywords":["Uncertainty in AI: Decision and utility theory","Humans and AI: Cognitive modeling","Agent-based and Multi-agent Systems: Human-agent interaction","Robotics: Human robot interaction","Humans and AI: Human-AI collaboration"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}