{"type":"ScholarlyArticle","title":"Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods","authors":[{"name":"Andrej Tschalzev","slug":"andrej-tschalzev","given":"Andrej","family":"Tschalzev","affiliation":"University of Mannheim"},{"name":"Paul Nitschke","slug":"paul-nitschke","given":"Paul","family":"Nitschke","affiliation":"Harvard University"},{"name":"Lukas Kirchdorfer","slug":"lukas-kirchdorfer","given":"Lukas","family":"Kirchdorfer","affiliation":"University of Mannheim; SAP Signavio"},{"name":"Stefan Lüdtke","slug":"stefan-ludtke","given":"Stefan","family":"Lüdtke","affiliation":"University of Rostock"},{"name":"Christian Bartelt","slug":"christian-bartelt","given":"Christian","family":"Bartelt","affiliation":"University of Mannheim"},{"name":"Heiner Stuckenschmidt","slug":"heiner-stuckenschmidt","given":"Heiner","family":"Stuckenschmidt","affiliation":"University of Mannheim"}],"abstract":"Neural networks often assume independence among input data samples, disregarding correlations arising from inherent clustering patterns in real-world datasets (e.g., due to different sites or repeated measurements). Recently, mixed effects neural networks (MENNs) which separate cluster-specific 'random effects' from cluster-invariant 'fixed effects' have been proposed to improve generalization and interpretability for clustered data. However, existing methods only allow for approximate quantification of cluster effects and are limited to regression and binary targets with only one clustering feature. We present MC-GMENN, a novel approach employing Monte Carlo techniques to train Generalized Mixed Effects Neural Networks. We empirically demonstrate that MC-GMENN outperforms existing mixed effects deep learning models in terms of generalization performance, time complexity, and quantification of inter-cluster variance. Additionally, MC-GMENN is applicable to a wide range of datasets, including multi-class classification tasks with multiple high-cardinality categorical features. For these datasets, we show that MC-GMENN outperforms conventional encoding and embedding methods, simultaneously offering a principled methodology for interpreting the effects of clustering patterns.","doi":"10.24963/ijcai.2024/555","url":"https://ijcai.org/proceedings/2024/555","pdf":"https://ijcai.org/proceedings/2024/0555.pdf","markdown":"https://ijcai.org/proceedings/2024/555.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/555","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":555,"pages":{"first":5018,"last":5026},"keywords":["Machine Learning: ML: Probabilistic machine learning","Machine Learning: ML: Classification","Machine Learning: ML: Deep learning architectures","Machine Learning: ML: Explainable/Interpretable machine learning"],"videos":[],"references":[{"position":1,"raw":"[Agresti, 2012] Alan Agresti. Categorical data analysis, vol- ume 792.","title":"Categorical data analysis","authors":"Alan Agresti","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Archila, 2016] Felipe Humberto Acosta Archila. Markov chain Monte Carlo for linear mixed models. PhD thesis, university of minnesota, 2016.","title":"Markov chain Monte Carlo for linear mixed models","authors":"Felipe Humberto, Acosta Archila","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Avanzi et al., 2023] Benjamin Avanzi, Greg Taylor, Melan- tha Wang, and Bernard Wong. Machine learning with high-cardinality categorical features in actuarial applica- tions. arXiv preprint arXiv:2301.12710, 2023.","title":"Machine learning with high-cardinality categorical features in actuarial applications","authors":"Benjamin Avanzi, Greg Taylor, Melantha Wang, Bernard Wong","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blei et al., 2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American statistical Association, 112(518):859-877, 2017.","title":"Variational inference: A review for statisticians","authors":"Alp David M Blei, Jon D Kucukelbir, Mcauliffe","year":"2017","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bolker et al., 2009] Benjamin M Bolker, Mollie E Brooks, Connie J Clark, Shane W Geange, John R Poulsen, M Henry H Stevens, and Jada-Simone S White. Gener- alized linear mixed models: a practical guide for ecology and evolution. Trends in ecology & evolution, 24(3):127- 135, 2009.","title":"Generalized linear mixed models: a practical guide for ecology and evolution","authors":"Mollie E Benjamin M Bolker, Connie J Brooks, Shane W Clark, Geange, M John R Poulsen, H Henry, Jada-Simone S Stevens, White","year":"2009","venue":"Trends in ecology & evolution","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Borisov et al., 2021] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01889, 2021.","title":"Deep neural networks and tabular data: A survey","authors":"Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, Gjergji Kasneci","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cafri et al., 2019] Guy Cafri, Wei Wang, Priscilla H Chan, and Peter C Austin. A review and empirical comparison of causal inference methods for clustered observational data with application to the evaluation of the effectiveness of medical devices. Statistical Methods in Medical Research, 28(10-11):3142-3162, 2019.","title":"A review and empirical comparison of causal inference methods for clustered observational data with application to the evaluation of the effectiveness of medical devices","authors":"Guy Cafri, Wei Wang, Priscilla H Chan, Peter C Austin","year":"2019","venue":"Statistical Methods in Medical Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Erickson et al., 2020] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020.","title":"Autogluon-tabular: Robust and accurate automl for structured data","authors":"Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, Alexander Smola","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fei et al., 2021] Mengqi Fei, Huizhong Tan, Xixian Peng, Qiuzhen Wang, and Lei Wang. Promoting or attenuat- ing? an eye-tracking study on the role of social cues in e-commerce livestreaming. Decision Support Systems, 142:113466, 2021.","title":"Promoting or attenuating? an eye-tracking study on the role of social cues in e-commerce livestreaming","authors":"Fei","year":"2021","venue":"Decision Support Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grinsztajn et al., 2022] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still out- perform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35:507-520, 2022.","title":"Why do tree-based models still outperform deep learning on typical tabular data","authors":"Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo and Berkhahn, 2016] Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv preprint arXiv:1604.06737, 2016.","title":"Entity embeddings of categorical variables","authors":"Cheng Guo, Felix Berkhahn","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hajjem et al., 2017] Ahlem Hajjem, Denis Larocque, and Franc ¸ois Bellavance. Generalized mixed effects regres- sion trees. Statistics & Probability Letters, 126:114-118, 2017.","title":"Generalized mixed effects regression trees","authors":"Hajjem","year":"2017","venue":"Statistics & Probability Letters","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hancock and Khoshgoftaar, 2020] John T Hancock and Taghi M Khoshgoftaar. Survey on categorical data for neu- ral networks. Journal of Big Data, 7(1):1-41, 2020.","title":"Survey on categorical data for neural networks","authors":"T John, Taghi M Hancock, Khoshgoftaar","year":"2020","venue":"Journal of Big Data","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Harrison et al., 2018] Xavier A Harrison, Lynda Donald- son, Maria Eugenia Correa-Cano, Julian Evans, David N Fisher, Cecily ED Goodwin, Beth S Robinson, David J Hodgson, and Richard Inger. A brief introduction to mixed effects modelling and multi-model inference in ecology. PeerJ, 6:e4794, 2018.","title":"A brief introduction to mixed effects modelling and multi-model inference in ecology","authors":"Lynda Xavier A Harrison, Maria Donaldson, Eugenia Correa-Cano, Julian Evans, David N Fisher, Cecily Ed Goodwin, Beth S Robinson, David J Hodgson, Richard Inger","year":"2018","venue":"PeerJ","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hoffman and Gelman, 2011] Matthew D. Hoffman and An- drew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo, 2011.","title":"The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo","authors":"Matthew D Hoffman, Andrew Gelman","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2020] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabu- lar data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.","title":"Tabtransformer: Tabular data modeling using contextual embeddings","authors":"Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jospin et al., 2022] Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Ben- namoun. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29-48, 2022.","title":"Hands-on bayesian neural networks-a tutorial for deep learning users","authors":"Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, Mohammed Bennamoun","year":"2022","venue":"IEEE Computational Intelligence Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kilian et al., 2023] Pascal Kilian, Sangbeak Ye, and Au- gustin Kelava. Mixed effects in machine learning-a flex- ible mixedml framework to add random effects to super- vised machine learning regression. Transactions on Ma- chine Learning Research, 2023.","title":"Mixed effects in machine learning-a flexible mixedml framework to add random effects to supervised machine learning regression","authors":"Pascal Kilian, Sangbeak Ye, Augustin Kelava","year":"2023","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mandel et al., 2021] Francesca Mandel, Riddhi Pratim Ghosh, and Ian Barnett. Neural networks for clustered and longitudinal data using mixed effects models. Biometrics, 2021.","title":"Neural networks for clustered and longitudinal data using mixed effects models","authors":"Francesca Mandel, Riddhi Pratim Ghosh, Ian Barnett","year":"2021","venue":"Biometrics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[McCulloch, 1997] Charles E McCulloch. Maximum like- lihood algorithms for generalized linear mixed mod- els. Journal of the American statistical Association, 92(437):162-170, 1997.","title":"Maximum likelihood algorithms for generalized linear mixed models","authors":"Charles E Mcculloch","year":"1997","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McCulloch, 2003] Charles E McCulloch. Generalized lin- ear mixed models. Ims, 2003.","title":"Generalized linear mixed models","authors":"Charles E Mcculloch","year":"2003","venue":"Ims","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Micci-Barreca, 2001] Daniele Micci-Barreca. A prepro- cessing scheme for high-cardinality categorical attributes in classification and prediction problems. ACM SIGKDD Explorations Newsletter, 3(1):27-32, 2001.","title":"A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems","authors":"Daniele Micci-Barreca","year":"2001","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Monnahan and Kristensen, 2018] Cole C Monnahan and Kasper Kristensen. No-u-turn sampling for fast bayesian inference in admb and tmb: Introducing the adnuts and tmbstan r packages. PloS one, 13(5):e0197954, 2018.","title":"No-u-turn sampling for fast bayesian inference in admb and tmb: Introducing the adnuts and tmbstan r packages","authors":"C Cole, Kasper Monnahan, Kristensen","year":"2018","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ngufor et al., 2019] Che Ngufor, Holly Van Houten, Brian S Caffo, Nilay D Shah, and Rozalina G McCoy. Mixed effect machine learning: a framework for predict- ing longitudinal change in hemoglobin a1c. Journal of biomedical informatics, 89:56-67, 2019.","title":"Mixed effect machine learning: a framework for predicting longitudinal change in hemoglobin a1c","authors":"Che Ngufor, Holly Van Houten, Brian S Caffo, Nilay D Shah, Rozalina G Mccoy","year":"2019","venue":"Journal of biomedical informatics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nguyen et al., 2023] Kevin P Nguyen, Alex H Treacher, and Albert A Montillo. Adversarially-regularized mixed effects deep learning (armed) models improve inter- pretability, performance, and generalization on clustered (non-iid) data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.","title":"Adversarially-regularized mixed effects deep learning (armed) models improve interpretability, performance, and generalization on clustered (non-iid) data","authors":"Kevin P Nguyen, Alex H Treacher, Albert A Montillo","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pargent et al., 2022] Florian Pargent, Florian Pfisterer, Janek Thomas, and Bernd Bischl. Regularized target en- coding outperforms traditional methods in supervised ma- chine learning with high cardinality features. Computa- tional Statistics, pages 1-22, 2022.","title":"Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features","authors":"Florian Pargent, Florian Pfisterer, Janek Thomas, Bernd Bischl","year":"2022","venue":"Computational Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pinheiro and Chao, 2006] José C Pinheiro and Edward C Chao. Efficient laplacian and adaptive gaussian quadrature algorithms for multilevel generalized linear mixed mod- els. Journal of Computational and Graphical Statistics, 15(1):58-81, 2006.","title":"Efficient laplacian and adaptive gaussian quadrature algorithms for multilevel generalized linear mixed models","authors":"Chao ; ] Pinheiro, C José, Edward C Pinheiro, Chao","year":"2006","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Prokhorenkova et al., 2018] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categor- ical features. Advances in neural information processing systems, 31, 2018.","title":"Catboost: unbiased boosting with categorical features","authors":"Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sela and Simonoff, 2012] Rebecca J Sela and Jeffrey S Si- monoff. Re-em trees: a data mining approach for longi- tudinal and clustered data. Machine learning, 86(2):169- 207, 2012.","title":"Re-em trees: a data mining approach for longitudinal and clustered data","authors":"Rebecca J, Sela, Jeffrey S Simonoff","year":"2012","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shi et al., 2022] Jun Shi, Chengming Jiang, Aman Gupta, Mingzhou Zhou, Yunbo Ouyang, Qiang Charles Xiao, Qingquan Song, Yi Wu, Haichao Wei, and Huiji Gao. Gen- eralized deep mixed models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3869-3877, 2022.","title":"Generalized deep mixed models","authors":"Jun Shi, Chengming Jiang, Aman Gupta, Mingzhou Zhou, Yunbo Ouyang, Qiang Charles Xiao, Qingquan Song, Yi Wu, Haichao Wei, Huiji Gao","year":"2022","venue":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sigrist, 2022] Fabio Sigrist. Gaussian process boosting. Journal of Machine Learning Research, 23(232):1-46, 2022.","title":"Gaussian process boosting","authors":"Fabio Sigrist","year":"2022","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sigrist, 2023] Fabio Sigrist. A comparison of machine learning methods for data with high-cardinality categori- cal variables. arXiv preprint arXiv:2307.02071, 2023.","title":"A comparison of machine learning methods for data with high-cardinality categorical variables","authors":"Fabio Sigrist","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Simchoni and Rosset, 2021] Giora Simchoni and Saharon Rosset. Using random effects to account for high- cardinality categorical features and repeated measures in deep neural networks. Advances in Neural Information Processing Systems, 34:25111-25122, 2021.","title":"Using random effects to account for highcardinality categorical features and repeated measures in deep neural networks","authors":"Rosset Simchoni, Giora Simchoni, Saharon Rosset","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Simchoni and Rosset, 2023] Giora Simchoni and Saharon Rosset. Integrating random effects in deep neural networks. Journal of Machine Learning Research, 24(156):1-57, 2023.","title":"Integrating random effects in deep neural networks","authors":"Rosset Simchoni, Giora Simchoni, Saharon Rosset","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tandon et al., 2006] Reeti Tandon, Sudeshna Adak, and Jeffrey A Kaye. Neural networks for longitudinal studies in alzheimer's disease. Artificial intelligence in medicine, 36(3):245-255, 2006.","title":"Neural networks for longitudinal studies in alzheimer's disease","authors":"Tandon","year":"2006","venue":"Artificial intelligence in medicine","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tran et al., 2020] M-N Tran, Nghia Nguyen, David Nott, and Robert Kohn. Bayesian deep net glm and glmm. Jour- nal of Computational and Graphical Statistics, 29(1):97- 113, 2020.","title":"Bayesian deep net glm and glmm","authors":"M-N Tran, Nghia Nguyen, David Nott, Robert Kohn","year":"2020","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wörtwein et al., 2023] Torsten Wörtwein, Nicholas B Allen, Lisa B Sheeber, Randy P Auerbach, Jeffrey F Cohn, and Louis-Philippe Morency. Neural mixed effects for nonlinear personalized predictions. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 445-454, 2023.","title":"Neural mixed effects for nonlinear personalized predictions","authors":"Torsten Wörtwein, Lisa B Nicholas B Allen, Randy P Sheeber, Jeffrey F Auerbach, Louis-Philippe Cohn, Morency","year":"2023","venue":"Proceedings of the 25th International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xiong et al., 2019] Yunyang Xiong, Hyunwoo J Kim, and Vikas Singh. Mixed effects neural networks (menets) with applications to gaze estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7743-7752, 2019.","title":"Mixed effects neural networks (menets) with applications to gaze estimation","authors":"Yunyang Xiong, J Hyunwoo, Vikas Kim, Singh","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}