{"type":"ScholarlyArticle","title":"Bidirectional Active Learning with Gold-Instance-Based Human Training","authors":[{"name":"Feilong Tang","slug":"feilong-tang","given":"Feilong","family":"Tang","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"}],"abstract":"Active learning was proposed to improve learning performance and reduce labeling cost. However, traditional relabeling-based schemes seriously limit the ability of active learning because human may repeatedly make similar mistakes, without improving their expertise. In this paper, we propose a Bidirectional Active Learning with human Training (BALT) model that can enhance human related expertise during labeling and improve relabelingquality accordingly. We quantitatively capture how gold instances can be used to both estimate labelers? previous performance and improve their future correctness ratio. Then, we propose the backward relabeling scheme that actively selects the most likely incorrectly labeled instances for relabeling. Experimental results on three real datasets demonstrate that our BALT algorithm significantly outperforms representative related proposals.","doi":"10.24963/ijcai.2019/830","url":"https://ijcai.org/proceedings/2019/830","pdf":"https://ijcai.org/proceedings/2019/0830.pdf","markdown":"https://ijcai.org/proceedings/2019/830.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/830","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":830,"pages":{"first":5989,"last":5996},"keywords":["Special Track on AI for Improving Human-Well Being: AI benefits to society AI applications (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: AI applications for Improving Human-Well Being (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Amir et al., 2016] Ofra Amir, Ece Kamar, Andrey Kolobov, and Barbara Grosz. Interactive teaching strategies for a- gent training. In IJCAI, pages 804-811, 2016.","title":"Interactive teaching strategies for agent training","authors":"Ofra Amir, Ece Kamar, Andrey Kolobov, Barbara Grosz","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/119"},{"position":2,"raw":"[Cakmak and Thomaz, 2014] Maya Cakmak and Andrea L Thomaz. Eliciting good teaching from humans for ma- chine learners. Artificial Intelligence, 217:198-215, 2014.","title":"Eliciting good teaching from humans for machine learners","authors":"Thomaz Cakmak, Maya Cakmak, Andrea L Thomaz","year":"2014","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2013] Jia Deng, Jonathan Krause, and Li Fei- Fei. Fine-grained crowdsourcing for fine-grained recogni- tion. In CVPR, pages 580-587, 2013.","title":"Fine-grained crowdsourcing for fine-grained recognition","authors":"Jia Deng, Jonathan Krause, Li Fei-Fei","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donmez et al., 2009] Pinar Donmez, Jaime G Carbonell. Efficiently learning the accuracy of labeling sources for selective sampling. In SIGKDD, pages 259-268, 2009.","title":"Efficiently learning the accuracy of labeling sources for selective sampling","authors":"Pinar Donmez, Jaime G Carbonell","year":"2009","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dontcheva et al., 2014] Mira Dontcheva, R Morris. Com- bining crowdsourcing and learning to improve engage- ment and performance. In CHI, pages 3379-3388, 2014.","title":"Combining crowdsourcing and learning to improve engagement and performance","authors":"Mira Dontcheva, R Morris","year":"2014","venue":"CHI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from la- belers of unknown expertise. In Advances in Neural Infor- mation Processing Systems, 2009, pp. 2035-2043.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Ting-Fan, Jacob Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gadiraju et al., 2015] Ujwal Gadiraju, Besnik Fetahu. Training workers for improving performance in crowd- sourcing microtasks. In EC-TEL, pages 100-114, 2015.","title":"Training workers for improving performance in crowdsourcing microtasks","authors":"Ujwal Gadiraju, Besnik Fetahu","year":"2015","venue":"EC-TEL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Golovin et al., 2010] Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal bayesian active learning with noisy observations. In NIPS, pages 766-774, 2010.","title":"Near-optimal bayesian active learning with noisy observations","authors":"Andreas Daniel Golovin, Debajyoti Krause, Ray","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Qi Li, Fenglong Ma, Jing Gao, Lu Su, Christopher J. Quinn. Crowdsourcing high quality labels with a tight budget In ACM International Conference on Web Search and Data Mining, pages 237-246, 2016.","title":"Crowdsourcing high quality labels with a tight budget","authors":"Qi Li, Fenglong Ma, Jing Gao, Lu Su, Christopher J Quinn","year":"2016","venue":"ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Harris, 2011] Christopher Harris. Youre hired! an examina- tion of crowdsourcing incentive models in human resource tasks. In CSDM, pages 15-18, 2011.","title":"Youre hired! an examination of crowdsourcing incentive models in human resource tasks","authors":"Christopher Harris","year":"2011","venue":"CSDM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2014] Sheng-Jun Huang, Rong Jin, and Zhi- hua Zhou. Active learning by querying informative & rep- resentative examples. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 36(10):1936-1949, 2014.","title":"Active learning by querying informative & representative examples","authors":"Sheng-Jun Huang, Rong Jin, Zhihua Zhou","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ipeirotis et al., 2014] Panagiotis G. Ipeirotis, Foster Provost, Victor S. Sheng, and Jing Wang. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28(2):402-441, 2014.","title":"Repeated labeling using multiple noisy labelers","authors":"G Panagiotis, Foster Ipeirotis, Victor S Provost, Jing Sheng, Wang","year":"2014","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johns et al., 2015] Edward Johns, Oisin Mac Aodha, and Gabriel J Brostow. Becoming the expert-interactive multi- class machine teaching. In CVPR, pages 1-9, 2015.","title":"Becoming the expert-interactive multiclass machine teaching","authors":"Edward Johns, Oisin Mac Aodha, Gabriel J Brostow","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kamar and Horvitz, 2015] Ece Kamar and Eric Horvitz. Planning for crowdsourcing hierarchical tasks. In AAMAS, pages 1191-1199, 2015.","title":"Planning for crowdsourcing hierarchical tasks","authors":"Horvitz Kamar, Ece Kamar, Eric Horvitz","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2012] Neeraj Kumar, Peter N Belhumeur, Ar- ijit Biswas, David W Jacobs, W John Kress, Ida C Lopez, and João VB Soares. Leafsnap: A computer vision system for automatic plant species identification. In ECCV. 2012.","title":"Leafsnap: A computer vision system for automatic plant species identification","authors":"Neeraj Kumar, Arijit Peter N Belhumeur, David W Biswas, John Jacobs, Ida C Kress, João Vb Lopez, Soares","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li and Guo, 2013] Xin Li and Y Guo.Adaptive active learn- ing for image classification.In CVPR,pages 859-866,2013.","title":"Adaptive active learning for image classification","authors":"Guo ; ] Xin Li, Y Li, Guo","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2013] Shoushan Li, Yunxia Xue, Zhongqing Wang, and G Zhou. Active learning for cross-domain sen- timent classification. In IJCAI, pages 2127-2133, 2013.","title":"Active learning for cross-domain sentiment classification","authors":"Shoushan Li, Yunxia Xue, Zhongqing Wang, G Zhou","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/314"},{"position":18,"raw":"[Lin and Weld, 2016] Christopher H Lin and Daniel S Weld. Re-active learning: Active learning with relabeling. In AAAI, pages 1845-1852, 2016.","title":"Re-active learning: Active learning with relabeling","authors":"H Christopher, Daniel S Lin, Weld","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2014] Christopher H Lin, Daniel S Weld, et al. To re (label), or not to re (label). In HCOMP, 2014.","title":"To re (label), or not to re (label)","authors":"Daniel S Christopher H Lin, Weld","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oleson et al., 2011] David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le. Programmatic gold: Targeted and scalable quality assurance in crowd- sourcing. Human computation, 2011.","title":"Programmatic gold: Targeted and scalable quality assurance in crowdsourcing","authors":"David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le","year":"2011","venue":"Human computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qian et al., 2013] Buyue Qian, Xiang Wang, Fei Wang, Hongfei Li, Jieping Ye, and Ian Davidson. Active learning from relative queries. In IJCAI, pages 1614-1620, 2013.","title":"Active learning from relative queries","authors":"Buyue Qian, Xiang Wang, Fei Wang, Hongfei Li, Jieping Ye, Ian Davidson","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/240"},{"position":22,"raw":"[Roy and McCallum, 2001] Nicholas Roy and Andrew Mc- Callum. Toward optimal active learning through monte carlo estimation of error reduction. ICML, 2001.","title":"Toward optimal active learning through monte carlo estimation of error reduction","authors":"Roy, Mccallum, Nicholas Roy, Andrew Mc-Callum","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sachan et al., 2015] Mrinmaya Sachan, Eduard Hovy, and Eric P Xing. An active learning approach to coreference resolution. In IJCAI, pages 1312-1318, 2015.","title":"An active learning approach to coreference resolution","authors":"Sachan","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/189"},{"position":24,"raw":"[Servajean et al., 2016] Maximilien Servajean, Alexis Joly, Dennis Shasha, Julien Champ, and Esther Pacitti. Thep- lantgame: Actively training human annotators for domain- specific crowdsourcing. In ACMMM, 2016.","title":"Theplantgame: Actively training human annotators for domainspecific crowdsourcing","authors":"Maximilien Servajean, Alexis Joly, Dennis Shasha, Julien Champ, Esther Pacitti","year":"2016","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Settles, 2010] Burr Settles. Active learning literature sur- vey. University of Wisconsin, Madison, 2010.","title":"Burr Settles. Active learning literature survey","authors":"Settles","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sheng et al., 2008] Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In SIGKDD, pages 614-622, 2008.","title":"Get another label? improving data quality and data mining using multiple, noisy labelers","authors":"Foster Victor S Sheng, Panagiotis G Provost, Ipeirotis","year":"2008","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lewis et al., 1994] David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. In SIGIR, pages 3-12, 1994.","title":"A sequential algorithm for training text classifiers","authors":"David D Lewis, William A Gale","year":"1994","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Singla et al., 2014] Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, and Andreas Krause. Near- optimally teaching the crowd to classify. In ICML, 2014.","title":"Nearoptimally teaching the crowd to classify","authors":"Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, Andreas Krause","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Truby et al., 2014] Katherine Truby, Meredith L Weiss, and David L Rousseau. Teaching the unfamiliar to a crowd. PS: Political Science & Politics, 47(1):189-194, 2014.","title":"Teaching the unfamiliar to a crowd","authors":"Katherine Truby, Meredith L Weiss, David L Rousseau","year":"2014","venue":"PS: Political Science & Politics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and P Perona. Caltech-ucsd birds 200. 2010.","title":null,"authors":"Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, Perona","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2015] Xiao-Yu Zhang, Shupeng Wang, and X- iaochun Yun. Bidirectional active learning: a two-way ex- ploration into unlabeled and labeled data set. IEEE Trans Neural Netw Learn Syst., 26(12):3034-44, 2015.","title":"Bidirectional active learning: a two-way exploration into unlabeled and labeled data set","authors":"Xiao-Yu Zhang, Shupeng Wang, Xiaochun Yun","year":"2015","venue":"IEEE Trans Neural Netw Learn Syst","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2011] Liyue Zhao, Gita Sukthankar, and Rahul Sukthankar. Incremental relabeling for active learn- ing with noisy crowdsourced annotations. In PAS- SAT/SocialCom, pages 728-733, 2011.","title":"Incremental relabeling for active learning with noisy crowdsourced annotations","authors":"Zhao","year":"2011","venue":"PAS-SAT/SocialCom","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu, 2013] Xiaojin Zhu. Machine teaching for bayesian learners in the exponential family. In NIPS, pages 1905- 1913, 2013.","title":"Machine teaching for bayesian learners in the exponential family","authors":"Xiaojin Zhu","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}