{"type":"ScholarlyArticle","title":"Hierarchical Reinforcement Learning on Multi-Channel Hypergraph Neural Network for Course Recommendation","authors":[{"name":"Lu Jiang","slug":"lu-jiang","given":"Lu","family":"Jiang","affiliation":"School of Computer Science and Information Technology , Northeast Normal University , China; Key Laboratory of Applied Statistics of MOE , Northeast Normal University , China"},{"name":"Yanan Xiao","slug":"yanan-xiao","given":"Yanan","family":"Xiao","affiliation":"School of Computer Science and Information Technology , Northeast Normal University , China; Key Laboratory of Applied Statistics of MOE , Northeast Normal University , China"},{"name":"Xinxin Zhao","slug":"xinxin-zhao","given":"Xinxin","family":"Zhao","affiliation":"School of Computer Science and Information Technology , Northeast Normal University , China; Key Laboratory of Applied Statistics of MOE , Northeast Normal University , China"},{"name":"Yuanbo Xu","slug":"yuanbo-xu","given":"Yuanbo","family":"Xu","affiliation":"College of Computer Science and Technology , Jilin University , China; Mobile Intelligent Computing (MIC) Lab , Jilin University , China"},{"name":"Shuli Hu","slug":"shuli-hu","given":"Shuli","family":"Hu","affiliation":"School of Computer Science and Information Technology , Northeast Normal University , China; Key Laboratory of Applied Statistics of MOE , Northeast Normal University , China"},{"name":"Pengyang Wang","slug":"pengyang-wang","given":"Pengyang","family":"Wang","affiliation":"Department of Computer and Information Science , University of Macau , China; The State Key Laboratory of Internet of Things for Smart City , University of Macau , China"},{"name":"Minghao Yin","slug":"minghao-yin","given":"Minghao","family":"Yin","affiliation":"School of Computer Science and Information Technology , Northeast Normal University , China; Key Laboratory of Applied Statistics of MOE , Northeast Normal University , China"}],"abstract":"With the widespread popularity of massive open online courses, personalized course recommendation has become increasingly important due to enhancing users' learning efficiency. While achieving promising performances, current works suffering from the vary across the users and other MOOC entities. To address this problem,  we propose hierarchical reinforcement learning with a multi-channel hypergraphs neural network for course recommendation(called HHCoR). Specifically, we first construct an online course hypergraph as the environment to capture the complex relationships and historical information by considering all entities. Then, we design a multi-channel propagation mechanism to aggregate embeddings in the online course hypergraph and extract user interest through an attention layer. \r\nBesides, we employ two-level decision-making: the low-level focuses on the rating courses, while the high-level integrates these considerations to finalize the decision. Furthermore, in co-optimization, we design a joint reward function to improve the policy of two-layer agents. Finally, we conducted extensive experiments on two real-world datasets and the quantitative results have demonstrated the effectiveness of the proposed method.","doi":"10.24963/ijcai.2024/232","url":"https://ijcai.org/proceedings/2024/232","pdf":"https://ijcai.org/proceedings/2024/0232.pdf","markdown":"https://ijcai.org/proceedings/2024/232.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/232","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Data Mining","track":"Main Track","number":232,"pages":{"first":2099,"last":2107},"keywords":["Data Mining: DM: Applications","Data Mining: DM: Mining heterogenous data","Data Mining: DM: Mining spatial and/or temporal data","Data Mining: DM: Mining graphs"],"videos":[],"references":[{"position":1,"raw":"[Elfwing et al., 2018] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.","title":"Sigmoid-weighted linear units for neural network function approximation in reinforcement learning","authors":"Stefan Elfwing, Eiji Uchibe, Kenji Doya","year":"2018","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fan et al., 2021] Haoyi Fan, Fengbin Zhang, Yuxuan Wei, Zuoyong Li, Changqing Zou, Yue Gao, and Qionghai Dai. Heterogeneous hypergraph variational autoencoder for link prediction. IEEE Trans. Pattern Anal. Mach. In- tell., 44(8):4125-4138, 2021.","title":"Heterogeneous hypergraph variational autoencoder for link prediction","authors":"Fengbin Haoyi Fan, Yuxuan Zhang, Zuoyong Wei, Changqing Li, Yue Zou, Qionghai Gao, Dai","year":"2021","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Feinberg and Shwartz, 2012] Eugene A Feinberg and Adam Shwartz. Handbook of Markov decision processes: meth- ods and applications, volume 40. Springer Science Busi- ness Media, 2012.","title":"Handbook of Markov decision processes: methods and applications","authors":"Shwartz Feinberg, Eugene A Feinberg, Adam Shwartz","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong et al., 2022] Jibing Gong, Yao Wan, Ye Liu, Xuewen Li, Yi Zhao, Cheng Wang, Yuting Lin, Xiaohan Fang, Wenzheng Feng, Jingyi Zhang, et al. Reinforced moocs concept recommendation in heterogeneous information networks. ACM Trans. Web, 2022.","title":"Reinforced moocs concept recommendation in heterogeneous information networks","authors":"Jibing Gong, Ye Yao Wan, Xuewen Liu, Yi Li, Cheng Zhao, Yuting Wang, Xiaohan Lin, Wenzheng Fang, Jingyi Feng, Zhang","year":"2022","venue":"ACM Trans. Web","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collabo- rative filtering. In WWW, pages 173-182, 2017.","title":"Neural collaborative filtering","authors":"He","year":"2017","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2018] Xiangnan He, Zhankui He, Jingkuan Song, Zhenguang Liu, Yu-Gang Jiang, and Tat-Seng Chua. Nais: Neural attentive item similarity model for recommenda- tion. IEEE Trans. Knowl. Data Eng., 30(12):2354-2366, 2018.","title":"Nais: Neural attentive item similarity model for recommendation","authors":"He","year":"2018","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2020] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR, pages 639-648, 2020.","title":"Lightgcn: Simplifying and powering graph convolution network for recommendation","authors":"He","year":"2020","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hou et al., 2018] Yifan Hou, Pan Zhou, Jie Xu, and Dapeng Oliver Wu. Course recommendation of mooc with big data support: A contextual online learning approach. In INFOCOM WKSHPS, pages 106-111. IEEE, 2018.","title":"Course recommendation of mooc with big data support: A contextual online learning approach","authors":"Yifan Hou, Pan Zhou, Jie Xu, Dapeng Oliver Wu","year":"2018","venue":"INFOCOM WKSHPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2023] Lu Jiang, Kunpeng Liu, Yibin Wang, Dongjie Wang, Pengyang Wang, Yanjie Fu, and Minghao Yin. Reinforced explainable knowledge concept recom- mendation in moocs. ACM Trans. Intell. Syst. Technol., 14(3):1-20, 2023.","title":"Reinforced explainable knowledge concept recommendation in moocs","authors":"Lu Jiang, Kunpeng Liu, Yibin Wang, Dongjie Wang, Pengyang Wang, Yanjie Fu, Minghao Yin","year":"2023","venue":"ACM Trans. Intell. Syst. Technol","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kabbur et al., 2013] Santosh Kabbur, Xia Ning, and George Karypis. Fism: Factored item similarity models for top-n recommender systems. In KDD, pages 659-667, 2013.","title":"Fism: Factored item similarity models for top-n recommender systems","authors":"Santosh Kabbur, Xia Ning, George Karypis","year":"2013","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kakade, 2001] Sham M Kakade. A natural policy gradi- ent. Advances in neural information processing systems, 14, 2001.","title":"A natural policy gradient","authors":"M Sham, Kakade","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lapan, 2018] Maxim Lapan. Deep Reinforcement Learn- ing Hands-On: Apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, Al- phaGo Zero and more. Packt Publishing Ltd, 2018.","title":"Maxim Lapan","authors":"Lapan","year":"2018","venue":"Deep Reinforcement Learning Hands-On: Apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, Al-phaGo Zero and more","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap et al., 2016] T Lillicrap, J Hunt, Alexander Pritzel, N Hess, Tom Erez, D Silver, Y Tassa, and D Wiestra. Con- tinuous control with deep reinforcement learning. In ICRL, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Lillicrap, Alexander Hunt, N Pritzel, Tom Hess, D Erez, Y Silver, Tassa, Wiestra","year":"2016","venue":"ICRL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2022] Yuanguo Lin, Fan Lin, Lvqing Yang, Wen- hua Zeng, Yong Liu, and Pengcheng Wu. Context-aware reinforcement learning for course recommendation. Ap- plied Soft Computing, 125:109189, 2022.","title":"Context-aware reinforcement learning for course recommendation","authors":"Yuanguo Lin, Fan Lin, Lvqing Yang, Wenhua Zeng, Yong Liu, Pengcheng Wu","year":"2022","venue":"Applied Soft Computing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shao et al., 2021] Erzhuo Shao, Shiyuan Guo, and Zachary A Pardos. Degree planning with plan-bert: Multi-semester recommendation using future courses of interest. In AAAI, volume 35, pages 14920-14929, 2021.","title":"Degree planning with plan-bert: Multi-semester recommendation using future courses of interest","authors":"Shao","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Valdivia et al., 2021] Paola Valdivia, Paolo Buono, Cather- ine Plaisant, Nicole Dufournaud, and Jean-Daniel Fekete. Analyzing dynamic hypergraphs with parallel aggregated ordered hypergraph visualization. IEEE Trans. Vis. Com- put. Graph., 27(1):1-13, 2021.","title":"Analyzing dynamic hypergraphs with parallel aggregated ordered hypergraph visualization","authors":"Paola Valdivia, Paolo Buono, Catherine Plaisant, Nicole Dufournaud, Jean-Daniel Fekete","year":"2021","venue":"IEEE Trans. Vis. Comput. Graph","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2021] Jingjing Wang, Haoran Xie, Fu Lee Wang, Lap-Kei Lee, and Oliver Tat Sheung Au. Top-n per- sonalized recommendation with graph neural networks in moocs. Computers and Education: Artificial Intelligence, 2:100010, 2021.","title":"Top-n personalized recommendation with graph neural networks in moocs","authors":"Jingjing Wang, Haoran Xie, Lee Fu, Lap-Kei Wang, Oliver Lee, Sheung Tat, Au","year":"2021","venue":"Computers and Education: Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2022] Xinhua","title":null,"authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Wang, Wenyun Ma, Lei Guo, Haoran Jiang, Fangai Liu, and Changdi Xu. Hgnn: Hyperedge-based graph neural network for mooc course recommendation. Inf. Process. Manag., 59(3):102938, 2022.","title":"Hgnn: Hyperedge-based graph neural network for mooc course recommendation","authors":"Wenyun Wang, Lei Ma, Haoran Guo, Fangai Jiang, Changdi Liu, Xu","year":"2022","venue":"Inf. Process. Manag","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2019] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In AAAI, volume 33, pages 346-353, 2019.","title":"Session-based recommendation with graph neural networks","authors":"Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, Tieniu Tan","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xia et al., 2021a] Xin Xia, Hongzhi Yin, Junliang Yu, Yingxia Shao, and Lizhen Cui. Self-supervised graph co- training for session-based recommendation. In CIKM '21, page 2180-2190, New York, NY, USA, 2021. ACM.","title":"Self-supervised graph cotraining for session-based recommendation","authors":"Xia","year":"2021","venue":"CIKM '21","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xia et al., 2021b] Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Self-supervised hypergraph convolutional networks for session-based recommendation. In AAAI, volume 35, pages 4503-4511, 2021.","title":"Self-supervised hypergraph convolutional networks for session-based recommendation","authors":"Xia","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia et al., 2022] Lianghao Xia, Chao Huang, and Chuxu Zhang. Self-supervised hypergraph transformer for rec- ommender systems. In KDD, pages 2100-2109, 2022.","title":"Self-supervised hypergraph transformer for recommender systems","authors":"Xia","year":"2022","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2021] Ruobing Xie, Shaoliang Zhang, Rui Wang, Feng Xia, and Leyu Lin. Hierarchical reinforcement learn- ing for integrated recommendation. In AAAI, volume 35, pages 4521-4528, 2021.","title":"Hierarchical reinforcement learning for integrated recommendation","authors":"Xie","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2022] Yuanbo Xu, En Wang, Yongjian Yang, and Yi Chang. A unified collaborative representation learn- ing for neural-network based recommender systems. IEEE Trans. Knowl. Data Eng., 34(11):5126-5139, 2022.","title":"A unified collaborative representation learning for neural-network based recommender systems","authors":"Yuanbo Xu, En Wang, Yongjian Yang, Yi Chang","year":"2022","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2024] Yuanbo Xu, En Wang, Yongjian Yang, and Hui Xiong. GS-RS: A generative approach for alleviat- ing cold start and filter bubbles in recommender systems. IEEE Trans. Knowl. Data Eng., 36(2):668-681, 2024.","title":"GS-RS: A generative approach for alleviating cold start and filter bubbles in recommender systems","authors":"Yuanbo Xu, En Wang, Yongjian Yang, Hui Xiong","year":"2024","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang and Cai, 2022] Shuang Yang and Xuesong Cai. Bilat- eral knowledge graph enhanced online course recommen- dation. Information Systems, 107:102000, 2022.","title":"Bilateral knowledge graph enhanced online course recommendation","authors":"Yang, Cai, Shuang Yang, Xuesong Cai","year":"2022","venue":"Information Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang and Jiang, 2019] Xixi Yang and Wenjun Jiang. Dy- namic online course recommendation based on course net- work and user network. In iSCI, pages 180-196. Springer, 2019.","title":"Dynamic online course recommendation based on course network and user network","authors":"Yang, Jiang, Xixi Yang, Wenjun Jiang","year":"2019","venue":"iSCI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2020] Jifan Yu, Gan Luo, Tong Xiao, Qingyang Zhong, Yuquan Wang, Wenzheng Feng, Junyi Luo, Chenyu Wang, Lei Hou, Juanzi Li, et al. Mooccube: a large-scale data repository for nlp applications in moocs. In ACL, pages 3135-3142, 2020.","title":"Mooccube: a large-scale data repository for nlp applications in moocs","authors":"Jifan Yu, Gan Luo, Tong Xiao, Qingyang Zhong, Yuquan Wang, Wenzheng Feng, Junyi Luo, Chenyu Wang, Lei Hou, Juanzi Li","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2019] Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, and Jimeng Sun. Hierarchi- cal reinforcement learning for course recommendation in moocs. In AAAI, volume 33, pages 435-442, 2019.","title":"Hierarchical reinforcement learning for course recommendation in moocs","authors":"Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, Jimeng Sun","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022] Xiaokun Zhang, Bo Xu, Liang Yang, Chenliang Li, Fenglong Ma, Haifeng Liu, and Hongfei Lin. Price does matter! modeling price and interest prefer- ences in session-based recommendation. In SIGIR, pages 1684-1693, 2022.","title":"Price does matter! modeling price and interest preferences in session-based recommendation","authors":"Xiaokun Zhang, Bo Xu, Liang Yang, Chenliang Li, Fenglong Ma, Haifeng Liu, Hongfei Lin","year":"2022","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2024] Zhaofan Zhang, Yanan Xiao, Lu Jiang, Dingqi Yang, Minghao Yin, and Pengyang Wang. Spatial- temporal interplay in human mobility: A hierarchical re- inforcement learning approach with hypergraph represen- tation. pages 9396-9404. AAAI Press, 2024.","title":"Spatialtemporal interplay in human mobility: A hierarchical reinforcement learning approach with hypergraph representation","authors":"Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2020] Dongyang Zhao, Liang Zhang, Bo Zhang, Lizhou Zheng, Yongjun Bao, and Weipeng Yan. Mahrl: Multi-goals abstraction based deep hierar- chical reinforcement learning for recommendations. In SIGIR, pages 871-880, 2020.","title":"Mahrl: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations","authors":"Zhao","year":"2020","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2020] Yifan Zhu, Hao Lu, Ping Qiu, Kaize Shi, James Chambua, and Zhendong Niu. Heterogeneous teaching evaluation network based offline course recom- mendation with graph learning and tensor factorization. Neurocomputing, 415:84-95, 2020.","title":"Heterogeneous teaching evaluation network based offline course recommendation with graph learning and tensor factorization","authors":"Yifan Zhu, Hao Lu, Ping Qiu, Kaize Shi, James Chambua, Zhendong Niu","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2023a] Yifan Zhu, Fangpeng Cong, Dan Zhang, Wenwen Gong, Qika Lin, Wenzheng Feng, Yuxiao Dong, and Jie Tang. WinGNN: dynamic graph neural networks with random gradient aggregation window. In The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023. ACM, 2023.","title":"WinGNN: dynamic graph neural networks with random gradient aggregation window","authors":"Zhu","year":"2023","venue":"The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2023b] Yifan Zhu, Qika Lin, Hao Lu, Kaize Shi, Donglei Liu, James Chambua, Shanshan Wan, and Zhen- dong Niu. Recommending learning objects through atten- tive heterogeneous graph convolution and operation-aware neural network. IEEE Transactions on Knowledge and Data Engineering, 35(4):4178-4189, 2023.","title":"Recommending learning objects through attentive heterogeneous graph convolution and operation-aware neural network","authors":"Zhu","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}