{"type":"ScholarlyArticle","title":"HDI-Forest: Highest Density Interval Regression Forest","authors":[{"name":"Lin Zhu","slug":"lin-zhu","given":"Lin","family":"Zhu","affiliation":"Ctrip Travel Network Technology Co. , Limited."},{"name":"Jiaxing Lu","slug":"jiaxing-lu","given":"Jiaxing","family":"Lu","affiliation":"Ctrip Travel Network Technology Co. , Limited."},{"name":"Yihong Chen","slug":"yihong-chen","given":"Yihong","family":"Chen","affiliation":"Ctrip Travel Network Technology Co. , Limited."}],"abstract":"By seeking the narrowest prediction intervals (PIs) that satisfy the specified coverage probability requirements, the recently proposed quality-based PI learning principle can extract high-quality PIs that better summarize the predictive certainty in regression tasks, and has been widely applied to solve many practical problems. Currently, the state-of-the-art quality-based PI estimation methods are based on deep neural networks or linear models. In this paper, we propose Highest Density Interval Regression Forest (HDI-Forest), a novel quality-based PI estimation method that is instead based on Random Forest. HDI-Forest does not require additional model training, and directly reuses the trees learned in a standard Random Forest model. By utilizing the special properties of Random Forest, HDI-Forest could efficiently and more directly optimize the PI quality metrics. Extensive experiments on benchmark datasets show that HDI-Forest significantly outperforms previous approaches, reducing the average PI width by over 20% while achieving the same or better coverage probability.","doi":"10.24963/ijcai.2019/621","url":"https://ijcai.org/proceedings/2019/621","pdf":"https://ijcai.org/proceedings/2019/0621.pdf","markdown":"https://ijcai.org/proceedings/2019/621.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/621","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":621,"pages":{"first":4468,"last":4474},"keywords":["Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Barbieri et al., 2016] Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. Improving post-click user engage- ment on native ads via survival analysis. In WWW, pages 761-770, 2016.","title":"Improving post-click user engagement on native ads via survival analysis","authors":"Nicola Barbieri, Fabrizio Silvestri, Mounia Lalmas","year":"2016","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Box and Tiao, 1973] George EP Box and George C Tiao. Bayesian inference in statistical analysis. Wiley, 1973.","title":"Bayesian inference in statistical analysis","authors":"E P George, George C Box, Tiao","year":"1973","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ji Feng and Zhi-Hua Zhou. Autoen- coder by forest. In AAAI, 2018. Feng et al., 2018] Ji Feng, Yang Yu, and Zhi-Hua Zhou. Multi-layered gradient boosting decision trees. In NIPS, pages 3555-3565, 2018.","title":"Multi-layered gradient boosting decision trees","authors":"Ji Feng, Yang Yu, Zhi-Hua Zhou","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Friedman, 2001] Jerome H Friedman. Greedy function ap- proximation: a gradient boosting machine. Annals of S- tatistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome H Friedman","year":"2001","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050- 1059, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Galván et al., 2017] Inés M Galván, José M Valls, Alejan- dro Cervantes, and Ricardo Aler. Multi-objective evolu- tionary optimization of prediction intervals for solar en- ergy forecasting with neural networks. Information Sci- ences, 418:363-382, 2017.","title":"Multi-objective evolutionary optimization of prediction intervals for solar energy forecasting with neural networks","authors":"Inés M Galván, M José, Alejandro Valls, Ricardo Cervantes, Aler","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2015] Yiqing Huang, Fangzhou Zhu, Mingx- uan Yuan, Ke Deng, Yanhua Li, Bing Ni, Wenyuan Dai, Qiang Yang, and Jia Zeng. Telco churn prediction with big data. In SIGMOD, pages 607-618, 2015.","title":"Telco churn prediction with big data","authors":"Yiqing Huang, Fangzhou Zhu, Mingxuan Yuan, Ke Deng, Yanhua Li, Bing Ni, Wenyuan Dai, Qiang Yang, Jia Zeng","year":"2015","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khosravi and Nahavandi, 2014] Abbas Khosravi and Saei- d Nahavandi. An optimized mean variance estimation method for uncertainty quantification of wind power fore- casts. International Journal of Electrical Power & Energy Systems, 61:446-454, 2014.","title":"Abbas Khosravi and Saeid Nahavandi. An optimized mean variance estimation method for uncertainty quantification of wind power forecasts","authors":"Nahavandi Khosravi","year":"2014","venue":"International Journal of Electrical Power & Energy Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khosravi et al., 2011a] Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Comprehensive review of neural network-based prediction intervals and new advances. IEEE Transactions on neural networks, 22(9):1341-1356, 2011.","title":"Comprehensive review of neural network-based prediction intervals and new advances","authors":"Khosravi","year":"2011","venue":"IEEE Transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Khosravi et al., 2011b] Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Lower upper bound estimation method for construction of neural network- based prediction intervals. IEEE Trans. Neural Networks, 22(3):337-346, 2011.","title":"Lower upper bound estimation method for construction of neural networkbased prediction intervals","authors":"Khosravi","year":"2011","venue":"IEEE Trans. Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klambauer et al., 2017] Günter Klambauer, Thomas Un- terthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. In NIPS, pages 971-980, 2017.","title":"Selfnormalizing neural networks","authors":"Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koenker and Hallock, 2001] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic per- spectives, 15(4):143-156, 2001.","title":"Quantile regression","authors":"Hallock Koenker, Roger Koenker, Kevin F Hallock","year":"2001","venue":"Journal of economic perspectives","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lakshminarayanan et al., 2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensem- bles. In NIPS, pages 6402-6413, 2017.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Martin, 2017] Alexander Hanbo Li and Andrew Martin. Forest-type regression with general losses and ro- bust forest. In ICML, pages 2091-2100, 2017.","title":"Forest-type regression with general losses and robust forest","authors":"Martin ; Li, Alexander Hanbo, Li, Andrew Martin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lian et al., 2016] Cheng Lian, Zhigang Zeng, Wei Yao, Huiming Tang, and Chun Lung Philip Chen. Landslide displacement prediction with uncertainty based on neural networks with random hidden weights. IEEE Trans. Neu- ral Networks and Learning systems, 27(12):2683-2695, 2016.","title":"Landslide displacement prediction with uncertainty based on neural networks with random hidden weights","authors":"Cheng Lian, Zhigang Zeng, Wei Yao, Huiming Tang, Chun Lung, Philip Chen","year":"2016","venue":"IEEE Trans. Neural Networks and Learning systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin and Jeon, 2006] Yi Lin and Yongho Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association, 101(474):578-590, 2006.","title":"Random forests and adaptive nearest neighbors","authors":"Lin, Jeon, Yi Lin, Yongho Jeon","year":"2006","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meinshausen, 2006] Nicolai Meinshausen. Quantile regres- sion forests. JMLR, 7(Jun):983-999, 2006.","title":"Quantile regression forests","authors":"Nicolai Meinshausen","year":"2006","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pearce et al., 2018] Tim Pearce, Alexandra Brintrup, Mo- hamed Zaki, and Andy Neely. High-quality prediction in- tervals for deep learning: A distribution-free, ensembled approach. In ICML, pages 4075-4084, 2018.","title":"High-quality prediction intervals for deep learning: A distribution-free, ensembled approach","authors":"Tim Pearce, Alexandra Brintrup, Mohamed Zaki, Andy Neely","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. JMLR, 12(Oct):2825-2830, 2011.","title":"Scikit-learn: Machine learning in python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rivals and Personnaz, 2000] Isabelle Rivals and Léon Per- sonnaz. Construction of confidence intervals for neural networks based on least squares estimation. Neural Net- works, 13(4-5):463-484, 2000.","title":"Isabelle Rivals and Léon Personnaz. Construction of confidence intervals for neural networks based on least squares estimation","authors":"Personnaz Rivals","year":"2000","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rosenfeld et al., 2018] Nir Rosenfeld, Yishay Mansour, and Elad Yom-Tov. Discriminative learning of prediction inter- vals. In AISTATS, pages 347-355, 2018.","title":"Discriminative learning of prediction intervals","authors":"Nir Rosenfeld, Yishay Mansour, Elad Yom-Tov","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sathe and Aggarwal, 2017] Saket Sathe and Charu C Ag- garwal. Similarity forests. In SIGKDD, pages 395-403, 2017.","title":"Similarity forests","authors":"Saket Sathe, Charu C Aggarwal","year":"2017","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sharpe, 1970] Ken Sharpe. Robustness of normal tolerance intervals. Biometrika, pages 71-78, 1970.","title":"Robustness of normal tolerance intervals","authors":"Ken Sharpe","year":"1970","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shrivastava et al., 2015] Nitin Anand Shrivastava, Abbas Khosravi, and Bijaya Ketan Panigrahi. Prediction inter- val estimation of electricity prices using pso-tuned support vector machines. IEEE Transactions on Industrial Infor- matics, 11(2):322-331, 2015.","title":"Prediction interval estimation of electricity prices using pso-tuned support vector machines","authors":"Nitin Anand Shrivastava, Abbas Khosravi, Bijaya Ketan, Panigrahi","year":"2015","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Steinberger and Leeb, 2016] Lukas Steinberger and Hannes Leeb. Leave-one-out prediction intervals in linear regres- sion models with many variables. arXiv preprint arX- iv:1602.05801, 2016.","title":"Leave-one-out prediction intervals in linear regression models with many variables","authors":"Leeb Steinberger, Lukas Steinberger, Hannes Leeb","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stine, 1985] Robert A Stine. Bootstrap prediction intervals for regression. Journal of the American Statistical Associ- ation, 80(392):1026-1031, 1985.","title":"Bootstrap prediction intervals for regression","authors":"A Robert, Stine","year":"1985","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}