{"type":"ScholarlyArticle","title":"EZLearn: Exploiting Organic Supervision in Automated Data Annotation","authors":[{"name":"Maxim Grechkin","slug":"maxim-grechkin","given":"Maxim","family":"Grechkin","affiliation":"University of Washington"},{"name":"Hoifung Poon","slug":"hoifung-poon","given":"Hoifung","family":"Poon","affiliation":"Microsoft Research"},{"name":"Bill Howe","slug":"bill-howe","given":"Bill","family":"Howe","affiliation":"University of Washington"}],"abstract":"Many real-world applications require automated data annotation, such as identifying tissue origins based on gene expressions and classifying images into semantic categories. Annotation classes are often numerous and subject to changes over time, and annotating examples has become the major bottleneck for supervised learning methods. In science and other high-value domains, large repositories of data samples are often available, together with two sources of organic supervision: a lexicon for the annotation classes, and text descriptions that accompany some data samples. Distant supervision has emerged as a promising paradigm for exploiting such indirect supervision by automatically annotating examples where the text description contains a class mention in the lexicon. However, due to linguistic variations and ambiguities, such training data is inherently noisy, which limits the accuracy in this approach. In this paper, we introduce an auxiliary natural language processing system for the text modality, and incorporate co-training to reduce noise and augment signal in distant supervision. Without using any manually labeled data, our EZLearn system learned to accurately annotate data samples in functional genomics and scientific figure comprehension, substantially outperforming state-of-the-art supervised methods trained on tens of thousands of annotated examples.","doi":"10.24963/ijcai.2018/568","url":"https://ijcai.org/proceedings/2018/568","pdf":"https://ijcai.org/proceedings/2018/0568.pdf","markdown":"https://ijcai.org/proceedings/2018/568.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/568","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":568,"pages":{"first":4085,"last":4091},"keywords":["Natural Language Processing: Text Classification","Machine Learning: Classification","Machine Learning: Semi-Supervised Learning","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Blitzer et al., 2007] John Blitzer, Mark Dredze, Fernando Pereira, et al. Biographies, bollywood, boom-boxes and blenders: Do- main adaptation for sentiment classification. In ACL, 2007.","title":"Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification","authors":"John Blitzer, Mark Dredze, Fernando Pereira","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blum and Mitchell, 1998] Avrim Blum and Tom Mitchell. Com- bining labeled and unlabeled data with co-training. In COLT, 1998.","title":"Combining labeled and unlabeled data with co-training","authors":"Avrim Blum, Tom Mitchell","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet, 2015] Franc ¸ois Chollet. Keras. https://github.com/ fchollet/keras, 2015.","title":"Franc ¸ois Chollet. Keras","authors":"Chollet","year":"2015","venue":null,"doi":null,"url":"https://github.com/fchollet/keras,2015","internal_url":null},{"position":4,"raw":"[Clough and Barrett, 2016] Emily Clough and Tanya Barrett. The gene expression omnibus database. Methods Mol Biol, 2016.","title":"The gene expression omnibus database","authors":"Barrett ; Clough, Emily Clough, Tanya Barrett","year":"2016","venue":"Methods Mol Biol","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Daumé III, 2007] Hal Daumé III. Frustratingly easy domain adap- tation. ACL 2007, 2007.","title":"Hal Daumé III. Frustratingly easy domain adaptation","authors":"Iii Daumé","year":"2007","venue":"ACL 2007","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In CVPR 2009, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR 2009","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Edgar et al., 2002] Ron Edgar, Michael Domrachev, and Alex E Lash. Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. NAR, 2002.","title":"Gene Expression Omnibus: NCBI gene expression and hybridization array data repository","authors":"Ron Edgar, Michael Domrachev, Alex E Lash","year":"2002","venue":"NAR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fei-Fei et al., 2006] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE TPAMI, 2006.","title":"One-shot learning of object categories","authors":"Li Fei-Fei, Rob Fergus, Pietro Perona","year":"2006","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Friesike et al., 2015] Sascha Friesike, Bastian Widenmayer, et al. Opening science: towards an agenda of open science in academia and industry. J. of Tech. Transfer, 2015.","title":"Opening science: towards an agenda of open science in academia and industry","authors":"Sascha Friesike, Bastian Widenmayer","year":"2015","venue":"J. of Tech. Transfer","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gremse et al., 2011] Marion Gremse, Antje Chang, Ida Schom- burg, Andreas Grote, Maurice Scheer, Christian Ebeling, and Di- etmar Schomburg. The BRENDA Tissue Ontology (BTO): the first all-integrating ontology of all organisms for enzyme sources. Nucleic Acids Research, 2011.","title":"The BRENDA Tissue Ontology (BTO): the first all-integrating ontology of all organisms for enzyme sources","authors":"Marion Gremse, Antje Chang, Ida Schomburg, Andreas Grote, Maurice Scheer, Christian Ebeling, Dietmar Schomburg","year":"2011","venue":"Nucleic Acids Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gutierrez-Arcelus et al., 2015] Maria Gutierrez-Arcelus, Halit Ongen, Tuuli Lappalainen, et al. Tissue-specific effects of genetic and epigenetic variation on gene regulation and splicing. PLoS Genet, 2015.","title":"Tissue-specific effects of genetic and epigenetic variation on gene regulation and splicing","authors":"Maria Gutierrez-Arcelus, Halit Ongen, Tuuli Lappalainen","year":"2015","venue":"PLoS Genet","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hanahan and Weinberg, 2011] Douglas Hanahan and Robert A Weinberg. Hallmarks of cancer: the next generation. cell, 144(5):646-674, 2011.","title":"Hallmarks of cancer: the next generation","authors":"Weinberg Hanahan, Douglas Hanahan, Robert A Weinberg","year":"2011","venue":"cell","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hearst, 1991] Marti Hearst. Noun homograph disambiguation us- ing local context in large text corpora. Using Corpora, pages 185-188, 1991.","title":"Marti Hearst. Noun homograph disambiguation using local context in large text corpora","authors":"Hearst","year":"1991","venue":"Using Corpora","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Howe et al., 2017] Bill Howe, Po-shen Lee, Maxim Grechkin, Sean T Yang, and Jevin D West. Deep mapping of the visual literature. In WWW Companion, pages 1273-1277, 2017.","title":"Deep mapping of the visual literature","authors":"Howe","year":"2017","venue":"WWW Companion","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Joulin et al., 2017] Armand Joulin, Edouard Grave, Piotr Bo- janowski, and Tomas Mikolov. Bag of tricks for efficient text classification. EACL 2017, page 427, 2017.","title":"Bag of tricks for efficient text classification","authors":"Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov","year":"2017","venue":"EACL 2017","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2013] Young-suk Lee, Arjun Krishnan, Qian Zhu, and Olga G. Troyanskaya. Ontology-aware classification of tissue and cell-type signals in gene expression profiles across platforms and technologies. Bioinformatics, 29(23):3036-3044, 2013.","title":"Troyanskaya. Ontology-aware classification of tissue and cell-type signals in gene expression profiles across platforms and technologies","authors":"Lee","year":"2013","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee et al., 2017] Po-shen Lee, Jevin D West, and Bill Howe. Viziometrics: Analyzing visual information in the scientific lit- erature. IEEE Transactions on Big Data, 2017.","title":"Viziometrics: Analyzing visual information in the scientific literature","authors":"Lee","year":"2017","venue":"IEEE Transactions on Big Data","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Libbrecht and Noble, 2015] Maxwell W. Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomics. Nat. Rev. Genetics, 2015.","title":"Machine learning applications in genetics and genomics","authors":"Maxwell W Libbrecht, William Stafford, Noble","year":"2015","venue":"Nat. Rev. Genetics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Malone et al., 2010] James Malone, Ele Holloway, Tomasz Adamusiak, et al. Modeling sample variables with an experi- mental factor ontology. Bioninf., 2010.","title":"Modeling sample variables with an experimental factor ontology","authors":"Malone","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McClosky and Charniak, 2008] David McClosky and Eugene Charniak. Self-training for biomedical parsing. In ACL, 2008.","title":"Self-training for biomedical parsing","authors":"Charniak Mcclosky, David Mcclosky, Eugene Charniak","year":"2008","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mintz et al., 2009] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without la- beled data. In ACL, pages 1003-1011, 2009.","title":"Distant supervision for relation extraction without labeled data","authors":"Mike Mintz, Steven Bills, Rion Snow, Dan Jurafsky","year":"2009","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Molloy, 2011] Jennifer C Molloy. The open knowledge founda- tion: open data means better science. PLoS Biol, 2011.","title":"The open knowledge foundation: open data means better science","authors":"Jennifer C Molloy","year":"2011","venue":"PLoS Biol","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nigam and Ghani, 2000] Kamal Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In CIKM, 2000.","title":"Analyzing the effectiveness and applicability of co-training","authors":"Kamal Nigam, R Ghani","year":"2000","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Palatucci et al., 2009] Mark Palatucci, Dean Pomerleau, Geof- frey E Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. In NIPS, 2009.","title":"Zero-shot learning with semantic output codes","authors":"Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, Tom M Mitchell","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE TKDE, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peng et al., 2017] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. Cross-sentence n-ary rela- tion extraction with graph LSTMs. TACL, 5:101-115, 2017.","title":"Cross-sentence n-ary relation extraction with graph LSTMs","authors":"Peng","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Piccolo et al., 2013] Stephen Piccolo, Michelle Withers, Owen Francis, Andrea Bild, and Evan Johnson. Multiplatform single- sample estimates of transcriptional activation. PNAS, 2013.","title":"Multiplatform singlesample estimates of transcriptional activation","authors":"Stephen Piccolo, Michelle Withers, Owen Francis, Andrea Bild, Evan Johnson","year":"2013","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Piwowar and Vision, 2013] Heather Piwowar and Todd Vision. Data reuse and the open data citation advantage. PeerJ, 2013.","title":"Data reuse and the open data citation advantage","authors":"Heather Piwowar, Todd Vision","year":"2013","venue":"PeerJ","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Quirk and Poon, 2017] Chris Quirk and Hoifung Poon. Distant su- pervision for relation extraction beyond the sentence boundary. EACL-2017, 2017.","title":"Distant supervision for relation extraction beyond the sentence boundary","authors":"Poon Quirk, Chris Quirk, Hoifung Poon","year":"2017","venue":"EACL-2017","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Raina et al., 2007] Rajat Raina, Alexis Battle, Honglak Lee, Ben- jamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlabeled data. In ICML, 2007.","title":"Self-taught learning: transfer learning from unlabeled data","authors":"Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, Andrew Y Ng","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ratner et al., 2016] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data Programming: Creating large training sets, quickly. In NIPS, 2016.","title":"Data Programming: Creating large training sets, quickly","authors":"Christopher M De Alexander J Ratner, Sen Sa, Daniel Wu, Christopher Selsam, Ré","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Rung and Brazma, 2013] Johan Rung and Alvis Brazma. Reuse of public genome-wide gene expression data. Nat. Rev. Genet., 2013.","title":"Reuse of public genome-wide gene expression data","authors":"Brazma Rung, Johan Rung, Alvis Brazma","year":"2013","venue":"Nat. Rev. Genet","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christo- pher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In NIPS, 2013.","title":"Zero-shot learning through cross-modal transfer","authors":"Richard Socher, Milind Ganjoo, Christopher D Manning, Andrew Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hin- ton. Lecture 6.5-rmsprop: Divide the gradient by a running aver- age of its recent magnitude. Coursera: NN for ML, 2012.","title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman","year":"2012","venue":"Tijmen Tieleman and Geoffrey Hinton","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Torrente et al., 2016] Aurora Torrente, Margus Lukk, et al. Iden- tification of cancer related genes using a comprehensive map of human gene expression. PLOS ONE, 2016.","title":"Identification of cancer related genes using a comprehensive map of human gene expression","authors":"Torrente","year":"2016","venue":"PLOS ONE","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Verspoor et al., 2006] Karin Verspoor, Judith Cohn, Susan Mniszewski, and Cliff Joslyn. A categorization approach to automated ontological function annotation. Prot. Sc., 2006.","title":"A categorization approach to automated ontological function annotation","authors":"Karin Verspoor, Judith Cohn, Susan Mniszewski, Cliff Joslyn","year":"2006","venue":"Prot. Sc","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and compos- ing robust features with denoising autoencoders. In ICML, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhu and Ghahramani, 2002] Xiaojin Zhu and Zoubin Ghahra- mani. Learning from labeled and unlabeled data with label prop- agation. Technical Report CMU-CALD-02-107, 2002.","title":"Learning from labeled and unlabeled data with label propagation","authors":"Xiaojin Zhu, Zoubin Ghahramani","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2008] Yuelin Zhu, Sean Davis, Robert Stephens, Paul S. Meltzer, and Yidong Chen. GEOmetadb: powerful alternative search engine for the gene expression omnibus. Bioinf., 2008.","title":"GEOmetadb: powerful alternative search engine for the gene expression omnibus","authors":"Yuelin Zhu, Sean Davis, Robert Stephens, Paul S Meltzer, Yidong Chen","year":"2008","venue":"Bioinf","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}