{"type":"ScholarlyArticle","title":"Binary Classification from Positive Data with Skewed Confidence","authors":[{"name":"Kazuhiko Shinoda","slug":"kazuhiko-shinoda","given":"Kazuhiko","family":"Shinoda","affiliation":"Frontier Research Center , Toyota Motor Corporation"},{"name":"Hirotaka Kaji","slug":"hirotaka-kaji","given":"Hirotaka","family":"Kaji","affiliation":"Frontier Research Center , Toyota Motor Corporation"},{"name":"Masashi Sugiyama","slug":"masashi-sugiyama","given":"Masashi","family":"Sugiyama","affiliation":"Center for Advanced Intelligence Project , RIKEN; Graduate School of Frontier Sciences , The University of Tokyo {kazuhiko"}],"abstract":"Positive-confidence (Pconf) classification [Ishida et al., 2018] is a promising weakly-supervised learning method which trains a binary classifier only from positive data equipped with confidence. However, in practice, the confidence may be skewed by bias arising in an annotation process. The Pconf classifier cannot be properly learned with skewed confidence, and consequently, the classification performance might be deteriorated. In this paper, we introduce the parameterized model of the skewed confidence, and propose the method for selecting the hyperparameter which cancels out the negative impact of the skewed confidence under the assumption that we have the misclassification rate of positive samples as a prior knowledge. We demonstrate the effectiveness of the proposed method through a synthetic experiment with simple linear models and benchmark problems with neural network models. We also apply our method to drivers’ drowsiness prediction to show that it works well with a real-world problem where confidence is obtained based on manual annotation.","doi":"10.24963/ijcai.2020/460","url":"https://ijcai.org/proceedings/2020/460","pdf":"https://ijcai.org/proceedings/2020/0460.pdf","markdown":"https://ijcai.org/proceedings/2020/460.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/460","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning Applications","track":"Main track","number":460,"pages":{"first":3328,"last":3334},"keywords":["Multidisciplinary Topics and Applications: Transportation","Machine Learning Applications: Applications of Unsupervised Learning","Humans and AI: Personalization and User Modeling"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24957/file_24957.m3u8","length_sec":274,"provider":"hls"}],"references":[{"position":1,"raw":"[Akiba et al., 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[AlHanai and Ghassemi, 2017] Tuka Waddah AlHanai and Mohammad Mahdi Ghassemi. Predicting latent narrative mood using audio and physiologic data. In AAAI, 2017.","title":"Predicting latent narrative mood using audio and physiologic data","authors":"Ghassemi ; ] Alhanai, Waddah Tuka, Mohammad Alhanai, Ghassemi Mahdi","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bao and Intille, 2004] Ling Bao and Stephen Intille. Activ- ity recognition from user-annotated acceleration data. In International Conference on Pervasive Computing, 2004.","title":"Activity recognition from user-annotated acceleration data","authors":"Ling Bao, Stephen Intille","year":"2004","venue":"International Conference on Pervasive Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bao et al., 2018] Han Bao, Gang Niu, and Masashi Sugiyama. Classification from pairwise similarity and un- labeled data. In ICML, 2018.","title":"Classification from pairwise similarity and unlabeled data","authors":"Han Bao, Gang Niu, Masashi Sugiyama","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chapelle and Zien, 2005] Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density sepa- ration. In AISTATS, 2005.","title":"Semi-supervised classification by low density separation","authors":"Zien Chapelle, Olivier Chapelle, Alexander Zien","year":"2005","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle et al., 2010] Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 2010.","title":"Semi-Supervised Learning","authors":"Chapelle","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cour et al., 2011] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501-1536, 2011.","title":"Learning from partial labels","authors":"Timothee Cour, Ben Sapp, Ben Taskar","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dietterich et al., 1997] Thomas G. Dietterich, Richard H. Lathrop, and Tomas Lozano-Perez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1-2):31-71, 1997.","title":"Solving the multiple instance problem with axis-parallel rectangles","authors":"G Thomas, Richard H Dietterich, Tomas Lathrop, Lozano-Perez","year":"1997","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Marthinus C. du Plessis and Masashi Sugiyama. Clustering unclustered data: Un- supervised binary labeling of two datasets having different class balances. In TAAI, 2013.","title":"Clustering unclustered data: Unsupervised binary labeling of two datasets having different class balances","authors":"C Marthinus, Masashi Du Plessis, Sugiyama","year":"2013","venue":"TAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[du Plessis et al., 2014] Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In NeurIPS. 2014.","title":"Analysis of learning from positive and unlabeled data","authors":"C Marthinus, Gang Du Plessis, Masashi Niu, Sugiyama","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Elkan and Noto, 2008] Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In KDD, 2008.","title":"Learning classifiers from only positive and unlabeled data","authors":"Noto Elkan, Charles Elkan, Keith Noto","year":"2008","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hachisuka et al., 2011] Satori Hachisuka, Kenji Ishida, Takeshi Enya, and Masayoshi Kamijo. Facial expres- sion measurement for detecting driver drowsiness. In Don Harris, editor, Engineering Psychology and Cognitive Er- gonomics, pages 135-144. Springer, 2011.","title":"Facial expression measurement for detecting driver drowsiness","authors":"Satori Hachisuka, Kenji Ishida, Takeshi Enya, Masayoshi Kamijo","year":"2011","venue":"Engineering Psychology and Cognitive Ergonomics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hallvig et al., 2013] David Hallvig, Anna Anund, Carina Fors, Göran Kecklund, Johan G. Karlsson, Mattias Wahde, and Torbjörn Åkerstedt. Sleepy driving on the real road and in the simulator-A comparison. Accident Analysis & Prevention, 50:44-50, 2013.","title":"Sleepy driving on the real road and in the simulator-A comparison","authors":"David Hallvig, Anna Anund, Carina Fors, Göran Kecklund, Johan G Karlsson, Mattias Wahde, Torbjörn Åkerstedt","year":"2013","venue":"Accident Analysis & Prevention","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Horne and Reyner, 1999] Jim Horne and Louise Reyner. Vehicle accidents related to sleep: A review. Occupational and environmental medicine, 56:289-94, 1999.","title":"Vehicle accidents related to sleep: A review","authors":"Reyner Horne, Jim Horne, Louise Reyner","year":"1999","venue":"Occupational and environmental medicine","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ishida et al., 2017] Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In NeurIPS, 2017.","title":"Learning from complementary labels","authors":"Takashi Ishida, Gang Niu, Weihua Hu, Masashi Sugiyama","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ishida et al., 2018] Takashi Ishida, Gang Niu, and Masashi Sugiyama. Binary classification from positive-confidence data. In NeurIPS. 2018.","title":"Binary classification from positive-confidence data","authors":"Takashi Ishida, Gang Niu, Masashi Sugiyama","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kaji et al., 2018] Hirotaka Kaji, Hayato Yamaguchi, and Masashi Sugiyama. Multi task learning with positive and unlabeled data and its application to mental state predic- tion. In ICASSP, 2018.","title":"Multi task learning with positive and unlabeled data and its application to mental state prediction","authors":"Hirotaka Kaji, Hayato Yamaguchi, Masashi Sugiyama","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kaji et al., 2019] Hirotaka Kaji, Hisashi Iizuka, and Masashi Sugiyama. ECG-based concentration recogni- tion with multi-task regression. IEEE Transactions on Biomedical Engineering, 66(1):101-110, 2019.","title":"ECG-based concentration recognition with multi-task regression","authors":"Hirotaka Kaji, Hisashi Iizuka, Masashi Sugiyama","year":"2019","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kato et al., 2019] Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. In ICLR, 2019.","title":"Learning from positive and unlabeled data with a selection bias","authors":"Masahiro Kato, Takeshi Teshima, Junya Honda","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kitajima et al., 1997] Hiroki Kitajima, Nakaho Numata, Keiichi Yamamoto, and Yoshihiro Goi. Prediction of auto- mobile driver sleepiness (1st report, rating of sleepiness based on facial expression and examination of effective predictor indexes of sleepiness). Transactions of the Japan Society of Mechanical Engineers. C, 63(613):3059-3066, 1997.","title":"Prediction of automobile driver sleepiness (1st report, rating of sleepiness based on facial expression and examination of effective predictor indexes of sleepiness)","authors":"Hiroki Kitajima, Nakaho Numata, Keiichi Yamamoto, Yoshihiro Goi","year":"1997","venue":"Transactions of the Japan Society of Mechanical Engineers. C","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lane et al., 2014] Nicholas D. Lane, Mu Lin, Mashfiqui Mohammod, Xiaochao Yang, Hong Lu, Giuseppe Car- done, Shahid Ali, Afsaneh Doryab, Ethan Berke, An- drew T. Campbell, and Tanzeem Choudhury. Bewell: Sensing sleep, physical activities and social interactions to promote wellbeing. Mobile Networks and Applications, 19(3):345-359, 2014.","title":"Bewell: Sensing sleep, physical activities and social interactions to promote wellbeing","authors":"Nicholas D Lane, Mu Lin, Mashfiqui Mohammod, Xiaochao Yang, Hong Lu, Giuseppe Cardone, Shahid Ali, Afsaneh Doryab, Ethan Berke, Andrew T Campbell, Tanzeem Choudhury","year":"2014","venue":"Mobile Networks and Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu and Tao, 2016] Tongliang Liu and Dacheng Tao. Clas- sification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine In- telligence, 38(3):447-461, 2016.","title":"Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting","authors":"Tao Liu","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2018] Bin Liu, Ying Li, Zhaonan Sun, Soumya Ghosh, and Kenney Ng. Early prediction of diabetes com- plications from electronic health records: A multi-task sur- vival analysis approach. In AAAI, 2018.","title":"Early prediction of diabetes complications from electronic health records: A multi-task survival analysis approach","authors":"Bin Liu, Ying Li, Zhaonan Sun, Soumya Ghosh, Kenney Ng","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lu et al., 2019] Nan Lu, Gang Niu, Aditya K. Menon, and Masashi Sugiyama. On the minimal supervision for train- ing any binary classifier from only unlabeled data. In ICLR, 2019.","title":"On the minimal supervision for training any binary classifier from only unlabeled data","authors":"Nan Lu, Gang Niu, Aditya K Menon, Masashi Sugiyama","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Matthews et al., 2014] Mark Matthews, Saeed Abdullah, Geri Gay, and Tanzeem Choudhury. Tracking mental well- being: Balancing rich sensing and patient needs. Com- puter, 47(4):36-43, 2014.","title":"Tracking mental wellbeing: Balancing rich sensing and patient needs","authors":"Mark Matthews, Saeed Abdullah, Geri Gay, Tanzeem Choudhury","year":"2014","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mintz et al., 2009] Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation ex- traction without labeled data. In ACL, 2009.","title":"Distant supervision for relation extraction without labeled data","authors":"Mintz","year":"2009","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep K. Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurIPS. 2013.","title":"Learning with noisy labels","authors":"Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, Ambuj Tewari","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Pas- sos, David Cournapeau, Matthieu Brucher, Matthieu Per- rot, and Édouard Duchesnay. Scikit-learn: Machine learn- ing in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Édouard Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sakai et al., 2017] Tomoya Sakai, Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Semi-supervised clas- sification based on classification from positive and unla- beled data. In ICML, 2017.","title":"Semi-supervised classification based on classification from positive and unlabeled data","authors":"Tomoya Sakai, C Marthinus, Gang Du Plessis, Masashi Niu, Sugiyama","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Scholkopf and Smola, 2001] Bernhard Scholkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2001.","title":"Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond","authors":"Smola Scholkopf, Bernhard Scholkopf, Alexander J Smola","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shinoda et al., 2019] Kazuhiko Shinoda, Masahiko Yoshii, Hayato Yamaguchi, and Hirotaka Kaji. Daytime sleepi- ness level prediction using respiratory information. In IJ- CAI, 2019.","title":"Daytime sleepiness level prediction using respiratory information","authors":"Kazuhiko Shinoda, Masahiko Yoshii, Hayato Yamaguchi, Hirotaka Kaji","year":"2019","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2019/827"},{"position":34,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. The journal of machine learning re- search, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vural et al., 2007] Esra Vural, Mujdat Cetin, Aytul Ercil, Gwen Littlewort, Marian Bartlett, and Javier Movellan. Drowsy driver detection through facial movement analy- sis. In Michael Lew, Nicu Sebe, Thomas S. Huang, and Erwin M. Bakker, editors, Human-Computer Interaction, pages 6-18. Springer, 2007.","title":"Drowsy driver detection through facial movement analysis","authors":"Esra Vural, Mujdat Cetin, Aytul Ercil, Gwen Littlewort, Marian Bartlett, Javier Movellan","year":"2007","venue":"Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Weiss et al., 2016] Gary M. Weiss, Jeffrey W. Lockhart, Tony T. Pulickal, Paul T. McHugh, Isaac H. Ronan, and Jessica L. Timko. Actitracker: A smartphone-based ac- tivity recognition system for improving health and well- being. In IEEE International Conference on Data Science and Advanced Analytics, pages 682-688, 2016.","title":"Actitracker: A smartphone-based activity recognition system for improving health and wellbeing","authors":"M Gary, Jeffrey W Weiss, Tony T Lockhart, Paul T Pulickal, Isaac H Mchugh, Jessica L Ronan, Timko","year":"2016","venue":"IEEE International Conference on Data Science and Advanced Analytics","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou, 2018] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-53, 2018.","title":"A brief introduction to weakly supervised learning","authors":"Zhi-Hua Zhou","year":"2018","venue":"National Science Review","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2003] Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, 2003.","title":"Semi-supervised learning using gaussian fields and harmonic functions","authors":"Xiaojin Zhu, Zoubin Ghahramani, John Lafferty","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}