{"type":"ScholarlyArticle","title":"Generalization-Aware Structured Regression towards Balancing Bias and Variance","authors":[{"name":"Martin Pavlovski","slug":"martin-pavlovski","given":"Martin","family":"Pavlovski","affiliation":"Center for Data Analytics and Biomedical Informatics , Temple University , Philadelphia , PA , USA; Macedonian Academy of Sciences and Arts , Skopje , Republic of Macedonia"},{"name":"Fang Zhou","slug":"fang-zhou","given":"Fang","family":"Zhou","affiliation":"Center for Data Analytics and Biomedical Informatics , Temple University , Philadelphia , PA , USA"},{"name":"Nino Arsov","slug":"nino-arsov","given":"Nino","family":"Arsov","affiliation":"Macedonian Academy of Sciences and Arts , Skopje , Republic of Macedonia"},{"name":"Ljupco Kocarev","slug":"ljupco-kocarev","given":"Ljupco","family":"Kocarev","affiliation":"Macedonian Academy of Sciences and Arts , Skopje , Republic of Macedonia"},{"name":"Zoran Obradovic","slug":"zoran-obradovic","given":"Zoran","family":"Obradovic","affiliation":"Center for Data Analytics and Biomedical Informatics , Temple University , Philadelphia , PA , USA"}],"abstract":"Attaining the proper balance between underfitting and overfitting is one of the central challenges in machine learning. It has been approached mostly by deriving bounds on generalization risks of learning algorithms. Such bounds are, however, rarely controllable. In this study, a novel bias-variance balancing objective function is introduced in order to improve generalization performance. By utilizing distance correlation, this objective function is able to indirectly control a stability-based upper bound on a model's expected true risk. In addition, the Generalization-Aware Collaborative Ensemble Regressor (GLACER) is developed, a model that bags a crowd of structured regression models, while allowing them to collaborate in a fashion that minimizes the proposed objective function. The experimental results on both synthetic and real-world data indicate that such an objective enhances the overall model's predictive performance. When compared against a broad range of both traditional and structured regression models GLACER was ~10-56% and ~49-99% more accurate for the task of predicting housing prices and hospital readmissions, respectively.","doi":"10.24963/ijcai.2018/363","url":"https://ijcai.org/proceedings/2018/363","pdf":"https://ijcai.org/proceedings/2018/0363.pdf","markdown":"https://ijcai.org/proceedings/2018/363.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/363","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":363,"pages":{"first":2616,"last":2622},"keywords":["Machine Learning: Machine Learning","Machine Learning: Structured Prediction","Machine Learning: Ensemble Methods"],"videos":[],"references":[{"position":1,"raw":"[Alabdulmohsin, 2015] Ibrahim M Alabdulmohsin. Algo- rithmic stability and uniform generalization. In Advances in Neural Information Processing Systems, pages 19-27, 2015.","title":"Algorithmic stability and uniform generalization","authors":"Ibrahim M Alabdulmohsin","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alabdulmohsin, 2017] Ibrahim Alabdulmohsin. An information-theoretic route from generalization in ex- pectation to generalization in probability. In Artificial Intelligence and Statistics, pages 92-100, 2017.","title":"An information-theoretic route from generalization in expectation to generalization in probability","authors":"Ibrahim Alabdulmohsin","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andonova et al., 2002] Savina Andonova, Andre Elisseeff, Theodoros Evgeniou, and Massimiliano Pontil. A simple algorithm for learning stable machines. In Proceedings of the 15th European Conference on Artificial Intelligence, pages 513-517. IOS Press, 2002.","title":"A simple algorithm for learning stable machines","authors":"Savina Andonova, Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil","year":"2002","venue":"Proceedings of the 15th European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arsov et al., 2017] Nino Arsov, Martin Pavlovski, Lasko Basnarkov, and Ljupco Kocarev. Generating highly accu- rate prediction hypotheses through collaborative ensemble learning. Scientific Reports, 7:44649, 2017.","title":"Generating highly accurate prediction hypotheses through collaborative ensemble learning","authors":"Arsov","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Audibert and Bousquet, 2007] Jean-Yves Audibert and Olivier Bousquet. Combining pac-bayesian and generic chaining bounds. Journal of Machine Learning Research, 8(Apr):863-889, 2007.","title":"Combining pac-bayesian and generic chaining bounds","authors":"Bousquet ; Jean-Yves Audibert, Olivier Audibert, Bousquet","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bartlett and Mendelson, 2002] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learn- ing Research, 3(Nov):463-482, 2002.","title":"Rademacher and gaussian complexities: Risk bounds and structural results","authors":"L Peter, Shahar Bartlett, Mendelson","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Blumer et al., 1989] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), 36(4):929-965, 1989.","title":"Learnability and the vapnik-chervonenkis dimension","authors":"Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, Manfred K Warmuth","year":"1989","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2(Mar):499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Büchlmann and Yu, 2002] Peter Büchlmann and Bin Yu. Analyzing bagging. Annals of Statistics, pages 927-961, 2002.","title":"Analyzing bagging","authors":"Yu ; Büchlmann, Peter Büchlmann, Bin Yu","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Elisseeff et al., 2003] André Elisseeff, Massimiliano Pontil, et al. Leave-one-out error and stability of learning algo- rithms with applications. NATO science series sub series iii computer and systems sciences, 190:111-130, 2003.","title":"Leave-one-out error and stability of learning algorithms with applications","authors":"André Elisseeff, Massimiliano Pontil","year":"2003","venue":"NATO science series sub series iii computer and systems sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Elisseeff et al., 2005] Andre Elisseeff, Theodoros Evge- niou, and Massimiliano Pontil. Stability of randomized learning algorithms. Journal of Machine Learning Re- search, 6(Jan):55-79, 2005.","title":"Stability of randomized learning algorithms","authors":"Elisseeff","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedman, 2001] Jerome H Friedman. Greedy function ap- proximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome H Friedman","year":"2001","venue":"Annals of statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hallac et al., 2015] David Hallac, Jure Leskovec, and Stephen Boyd. Network lasso: Clustering and optimiza- tion in large graphs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discov- ery and data mining, pages 387-396. ACM, 2015.","title":"Network lasso: Clustering and optimization in large graphs","authors":"David Hallac, Jure Leskovec, Stephen Boyd","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kutin and Niyogi, 2002] Samuel Kutin and Partha Niyogi. Almost-everywhere algorithmic stability and generaliza- tion error. In Proceedings of the Eighteenth conference on Uncertainty in artificial intelligence, pages 275-282.","title":"Almost-everywhere algorithmic stability and generalization error","authors":"Niyogi Kutin, Samuel Kutin, Partha Niyogi","year":"2002","venue":"Proceedings of the Eighteenth conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McAllester, 2003] David A McAllester. Pac-bayesian stochastic model selection. Machine Learning, 51(1):5- 21, 2003.","title":"Pac-bayesian stochastic model selection","authors":"A David, Mcallester","year":"2003","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pavlovski et al., 2017] Martin Pavlovski, Fang Zhou, Ivan Stojkovic, Ljupco Kocarev, and Zoran Obradovic. Adap- tive skip-train structured regression for temporal networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2017.","title":"Adaptive skip-train structured regression for temporal networks","authors":"Martin Pavlovski, Fang Zhou, Ivan Stojkovic, Ljupco Kocarev, Zoran Obradovic","year":"2017","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Poggio et al., 2004] Tomaso Poggio, Ryan Rifkin, Sayan Mukherjee, and Partha Niyogi. General conditions for pre- dictivity in learning theory. Nature, 428(6981):419, 2004.","title":"General conditions for predictivity in learning theory","authors":"Tomaso Poggio, Ryan Rifkin, Sayan Mukherjee, Partha Niyogi","year":"2004","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qin et al., 2009] Tao Qin, Tie-Yan Liu, Xu-Dong Zhang, De-Sheng Wang, and Hang Li. Global ranking using con- tinuous conditional random fields. In Advances in neural information processing systems, pages 1281-1288, 2009.","title":"Global ranking using continuous conditional random fields","authors":"Tao Qin, Tie-Yan Liu, Xu-Dong Zhang, De-Sheng Wang, Hang Li","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radosavljevic et al., 2010] Vladan Radosavljevic, Slobo- dan Vucetic, and Zoran Obradovic. Continuous condi- tional random fields for regression in remote sensing. In ECAI, pages 809-814, 2010.","title":"Continuous conditional random fields for regression in remote sensing","authors":"Vladan Radosavljevic, Slobodan Vucetic, Zoran Obradovic","year":"2010","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Székely et al., 2007] Gábor J Székely, Maria L Rizzo, Nail K Bakirov, et al. Measuring and testing depen- dence by correlation of distances. The annals of statistics, 35(6):2769-2794, 2007.","title":"Measuring and testing dependence by correlation of distances","authors":"J Gábor, Maria L Székely, Rizzo, Nail K Bakirov","year":"2007","venue":"The annals of statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Székely et al., 2009] Gábor J Székely, Maria L Rizzo, et al. Brownian distance covariance. The annals of applied statistics, 3(4):1236-1265, 2009.","title":"Brownian distance covariance","authors":"J Gábor, Maria L Székely, Rizzo","year":"2009","venue":"The annals of applied statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Talagrand, 1996] Michel Talagrand. Majorizing measures: the generic chaining. The Annals of Probability, pages 1049-1103, 1996.","title":"Majorizing measures: the generic chaining","authors":"Michel Talagrand","year":"1996","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vapnik, 1999] Vladimir N Vapnik. An overview of statisti- cal learning theory. IEEE transactions on neural networks, 10(5):988-999, 1999.","title":"An overview of statistical learning theory","authors":"N Vladimir, Vapnik","year":"1999","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu and Mannor, 2012] Huan Xu and Shie Mannor. Robust- ness and generalization. Machine learning, 86(3):391- 423, 2012.","title":"Robustness and generalization","authors":"Huan Xu, Shie Mannor","year":"2012","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}