{"type":"ScholarlyArticle","title":"Human-in-the-loop Active Covariance Learning for Improving Prediction in Small Data Sets","authors":[{"name":"Homayun Afrabandpey","slug":"homayun-afrabandpey","given":"Homayun","family":"Afrabandpey","affiliation":"Helsinki Institute for Information Technology HIIT; Department of Computer Science , Aalto University"},{"name":"Tomi Peltola","slug":"tomi-peltola","given":"Tomi","family":"Peltola","affiliation":"Helsinki Institute for Information Technology HIIT; Department of Computer Science , Aalto University"},{"name":"Samuel Kaski","slug":"samuel-kaski","given":"Samuel","family":"Kaski","affiliation":"Helsinki Institute for Information Technology HIIT; Department of Computer Science , Aalto University"}],"abstract":"Learning predictive models from small high-dimensional data sets is a key problem in high-dimensional statistics. Expert knowledge elicitation can help, and a strong line of work focuses on directly eliciting informative prior distributions for parameters. This either requires considerable statistical expertise or is laborious, as the emphasis has been on accuracy and not on efficiency of the process. Another line of work queries about importance of features one at a time, assuming them to be independent and hence missing covariance information. In contrast, we propose eliciting expert knowledge about pairwise feature similarities, to borrow statistical strength in the predictions, and using sequential decision making techniques to minimize the effort of the expert. Empirical results demonstrate improvement in predictive performance on both simulated and real data, in high-dimensional linear regression tasks, where we learn the covariance structure with a Gaussian process, based on sequential elicitation.","doi":"10.24963/ijcai.2019/271","url":"https://ijcai.org/proceedings/2019/271","pdf":"https://ijcai.org/proceedings/2019/0271.pdf","markdown":"https://ijcai.org/proceedings/2019/271.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/271","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":271,"pages":{"first":1959,"last":1966},"keywords":["Machine Learning: Active Learning","Machine Learning: Probabilistic Machine Learning","Humans and AI: Human-Computer Interaction"],"videos":[],"references":[{"position":1,"raw":"[Afrabandpey et al., 2017] Homayun Afrabandpey, Tomi Peltola, and Samuel Kaski. Interactive prior elicitation of feature similarities for small sample size prediction. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, pages 265-269. ACM, 2017.","title":"Interactive prior elicitation of feature similarities for small sample size prediction","authors":"Afrabandpey","year":"2017","venue":"Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blitzer et al., 2007] John Blitzer, Mark Dredze, and Fer- nando Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Associ- ation of Computational Linguistics, pages 440-447, 2007.","title":"Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification","authors":"John Blitzer, Mark Dredze, Fernando Pereira","year":"2007","venue":"Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carpenter et al., 2016] Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Be- tancourt, Michael A Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming lan- guage. Journal of Statistical Software, 20:1-37, 2016.","title":"Stan: A probabilistic programming language","authors":"Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Michael A Brubaker, Jiqiang Guo, Peter Li, Allen Riddell","year":"2016","venue":"Journal of Statistical Software","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daee et al., 2017] Pedram Daee, Tomi Peltola, Marta Soare, and Samuel Kaski. Knowledge elicitation via sequen- tial probabilistic inference for high-dimensional predic- tion. Machine Learning, 106(9-10):1599-1620, 2017.","title":"Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction","authors":"Pedram Daee, Tomi Peltola, Marta Soare, Samuel Kaski","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan and Li, 2006] Jianqing Fan and Runze Li. Statistical challenges with high dimensionality: feature selection in knowledge discovery. In Proceedings oh the Interna- tional Congress of Mathematicians: Madrid, August 22- 30, 2006: invited lectures, pages 595-622, 2006.","title":"Statistical challenges with high dimensionality: feature selection in knowledge discovery","authors":"Fan, Li ;, Jianqing Fan, Runze Li","year":"2006","venue":"Proceedings oh the International Congress of Mathematicians","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Friedman et al., 2010] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Regularization paths for general- ized linear models via coordinate descent. Journal of Sta- tistical Software, 33(1):1-22, 2010.","title":"Regularization paths for generalized linear models via coordinate descent","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2010","venue":"Journal of Statistical Software","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garthwaite et al., 2008] Paul H Garthwaite, James B Chilcott, David J Jenkinson, and Paul Tappenden. Use of expert knowledge in evaluating costs and benefits of alternative service provisions: A case study. Interna- tional Journal of Technology Assessment in Health Care, 24(3):350-357, 2008.","title":"Use of expert knowledge in evaluating costs and benefits of alternative service provisions: A case study","authors":"James B Paul H Garthwaite, David J Chilcott, Paul Jenkinson, Tappenden","year":"2008","venue":"International Journal of Technology Assessment in Health Care","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haakma et al., 2011] Wieke Haakma, Laura Bojke, Lotte Steuten, and Maarten IJzerman. Pmd4 expert elicita- tion to populate early health economic models of medi- cal diagnostic devices in development. Value in Health, 14(7):A244-A245, 2011.","title":"Pmd4 expert elicitation to populate early health economic models of medical diagnostic devices in development","authors":"Wieke Haakma, Laura Bojke, Lotte Steuten, Maarten Ijzerman","year":"2011","venue":"Value in Health","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krupka and Tishby, 2007] Eyal Krupka and Naftali Tishby. Incorporating prior knowledge on features into learning. In Artificial Intelligence and Statistics, pages 227-234, 2007.","title":"Incorporating prior knowledge on features into learning","authors":"Tishby Krupka, Eyal Krupka, Naftali Tishby","year":"2007","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2010] Qing Li, Nan Lin, et al. The Bayesian elastic net. Bayesian Analysis, 5(1):151-170, 2010.","title":"The Bayesian elastic net","authors":"Qing Li, Nan Lin","year":"2010","venue":"Bayesian Analysis","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Micallef et al., 2017] Luana Micallef, Iiris Sundin, Pekka Marttinen, Muhammad Ammad-ud din, Tomi Peltola, Marta Soare, Giulio Jacucci, and Samuel Kaski. Inter- active elicitation of knowledge on feature relevance im- proves predictions in small data sets. In Proceedings of the 22nd International Conference on Intelligent User In- terfaces, IUI '17, pages 547-552. ACM, 2017.","title":"Interactive elicitation of knowledge on feature relevance improves predictions in small data sets","authors":"Luana Micallef, Iiris Sundin, Pekka Marttinen, Muhammad Ammad-Ud Din, Tomi Peltola, Marta Soare, Giulio Jacucci, Samuel Kaski","year":"2017","venue":"Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI '17","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In Advances in Neural Information Processing Sys- tems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[O'Hagan, 1998] Anthony O'Hagan. Eliciting expert beliefs in substantial practical applications. Journal of the Royal Statistical Society: Series D (The Statistician), 47(1):21- 35, 1998.","title":"Eliciting expert beliefs in substantial practical applications","authors":"O' Anthony, Hagan","year":"1998","venue":"Journal of the Royal Statistical Society: Series D (The Statistician)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rasmussen and Williams, 2006] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.","title":"Gaussian Processes for Machine Learning","authors":"Carl Edward Rasmussen, Christopher K I Williams","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Salton, 1971] G. Salton. The SMART Retrieval System- Experiments in Automatic Document Processing.","title":"The SMART Retrieval System-Experiments in Automatic Document Processing","authors":"G Salton","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Seeger, 2008] Matthias W Seeger. Bayesian inference and optimal design for the sparse linear model. Journal of Ma- chine Learning Research, 9(Apr):759-813, 2008.","title":"Bayesian inference and optimal design for the sparse linear model","authors":"Matthias W Seeger","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sundin et al., 2018] I Sundin, T Peltola, L Micallef, H Afra- bandpey, M Soare, M Majumder Mamun, P Daee, C He, B Serim, A Havulinna, et al. Improving genomics-based predictions for precision medicine through active elicita- tion of expert knowledge. Bioinformatics, 34(13):i395- i403, 2018.","title":"Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge","authors":"Sundin","year":"2018","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xiao et al., 2018] Cao Xiao, Yan Jin, Ji Liu, Bo Zeng, and Shuai Huang. Optimal expert knowledge elicitation for Bayesian network structure identification. IEEE Transac- tions on Automation Science and Engineering, 2018.","title":"Optimal expert knowledge elicitation for Bayesian network structure identification","authors":"Cao Xiao, Yan Jin, Ji Liu, Bo Zeng, Shuai Huang","year":"2018","venue":"IEEE Transactions on Automation Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong et al., 2015] Sicheng Xiong, Yuanli Pei, Romer Ros- ales, and Xiaoli Z Fern. Active learning from relative com- parisons. IEEE Transactions on Knowledge and Data En- gineering, 27(12):3166-3175, 2015.","title":"Active learning from relative comparisons","authors":"Sicheng Xiong, Yuanli Pei, Romer Rosales, Xiaoli Z Fern","year":"2015","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2007] Liu Yang, Rong Jin, and Rahul Suk- thankar. Bayesian active distance metric learning. In Pro- ceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, pages 442-449. AUAI Press, 2007.","title":"Bayesian active distance metric learning","authors":"Liu Yang, Rong Jin, Rahul Sukthankar","year":"2007","venue":"Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}