{"type":"ScholarlyArticle","title":"Learning to Interpret Satellite Images using Wikipedia","authors":[{"name":"Burak Uzkent","slug":"burak-uzkent","given":"Burak","family":"Uzkent","affiliation":"Department of Computer Science , Stanford University"},{"name":"Evan Sheehan","slug":"evan-sheehan","given":"Evan","family":"Sheehan","affiliation":"Department of Computer Science , Stanford University"},{"name":"Chenlin Meng","slug":"chenlin-meng","given":"Chenlin","family":"Meng","affiliation":"Department of Computer Science , Stanford University"},{"name":"Zhongyi Tang","slug":"zhongyi-tang","given":"Zhongyi","family":"Tang","affiliation":"Department of Earth System Science , Stanford University"},{"name":"Marshall Burke","slug":"marshall-burke","given":"Marshall","family":"Burke","affiliation":"Department of Earth System Science , Stanford University"},{"name":"David Lobell","slug":"david-lobell","given":"David","family":"Lobell","affiliation":"Department of Earth System Science , Stanford University"},{"name":"Stefano Ermon","slug":"stefano-ermon","given":"Stefano","family":"Ermon","affiliation":"Department of Computer Science , Stanford University"}],"abstract":"Despite recent progress in computer vision, fine-grained interpretation of satellite images remains challenging because of a lack of labeled training data. To overcome this limitation, we construct a novel dataset called WikiSatNet by pairing geo-referenced Wikipedia articles with satellite imagery of their corresponding locations. We then propose two strategies to learn representations of satellite images by predicting properties of the corresponding articles from the images. Leveraging this new multi-modal dataset, we can drastically reduce the quantity of human-annotated labels and time required for downstream tasks. On the recently released fMoW dataset, our pre-training strategies can boost the performance of a model pre-trained on ImageNet by up to 4.5% in F1 score.","doi":"10.24963/ijcai.2019/502","url":"https://ijcai.org/proceedings/2019/502","pdf":"https://ijcai.org/proceedings/2019/0502.pdf","markdown":"https://ijcai.org/proceedings/2019/502.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/502","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":502,"pages":{"first":3620,"last":3626},"keywords":["Natural Language Processing: NLP Applications and Tools","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Unsupervised Learning","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Christie et al., 2018] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, 2018.","title":"Functional map of the world","authors":"Gordon Christie, Neil Fendley, James Wilson, Ryan Mukherjee","year":"2018","venue":"Proc. IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai et al., 2009] Wenyuan Dai, Ou Jin, Gui-Rong Xue, Qiang Yang, and Yong Yu. Eigentransfer: a unified frame- work for transfer learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 193-200. ACM, 2009.","title":"Eigentransfer: a unified framework for transfer learning","authors":"Wenyuan Dai, Ou Jin, Gui-Rong Xue, Qiang Yang, Yong Yu","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2018] Junwei Han, Dingwen Zhang, Gong Cheng, Nian Liu, and Dong Xu. Advanced deep-learning techniques for salient and category-specific object de- tection: a survey. IEEE Signal Processing Magazine, 35(1):84-100, 2018.","title":"Advanced deep-learning techniques for salient and category-specific object detection: a survey","authors":"Han","year":"2018","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269. IEEE, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jean et al., 2018] Neal Jean, Sherrie Wang, George Azzari, David Lobell, and Stefano Ermon. Tile2vec: Unsu- pervised representation learning for remote sensing data. arXiv preprint arXiv:1805.02855, 2018.","title":"Tile2vec: Unsupervised representation learning for remote sensing data","authors":"Neal Jean, Sherrie Wang, George Azzari, David Lobell, Stefano Ermon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kaiser et al., 2017] Pascal Kaiser, Jan Dirk Wegner, Aurélien Lucchi, Martin Jaggi, Thomas Hofmann, and Konrad Schindler. Learning aerial image segmentation from online maps. IEEE Transactions on Geoscience and Remote Sensing, 55(11):6054-6068, 2017.","title":"Learning aerial image segmentation from online maps","authors":"Pascal Kaiser, Jan Dirk Wegner, Aurélien Lucchi, Martin Jaggi, Thomas Hofmann, Konrad Schindler","year":"2017","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma et al., 2014] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In Ad- vances in Neural Information Processing Systems, pages 3581-3589, 2014.","title":"Semisupervised learning with deep generative models","authors":"Shakir Diederik P Kingma, Danilo Mohamed, Max Jimenez Rezende, Welling","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le and Mikolov, 2014] Quoc Le and Thomas Mikolov. Dis- tributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014.","title":"Distributed representations of sentences and documents","authors":"Quoc Le, Thomas Mikolov","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lei Ba et al., 2015] Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neu- ral networks using textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pages 4247-4255, 2015.","title":"Predicting deep zero-shot convolutional neural networks using textual descriptions","authors":"Lei Ba","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mahajan et al., 2018] Dhruv Mahajan, Ross Girshick, Vig- nesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Ex- ploring the limits of weakly supervised pretraining. arXiv preprint arXiv:1805.00932, 2018.","title":"Exploring the limits of weakly supervised pretraining","authors":"Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens Van Der Maaten","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. CoRR, abs/1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[NAIP, 2016] USDA-NASS Cropland Data NAIP. Published crop-specific data layer. USDA-NASS, Washington, DC, 2016.","title":"USDA-NASS Cropland Data NAIP. Published crop-specific data layer","authors":"Naip","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oshri et al., 2018] Barak Oshri, Annie Hu, Peter Adelson, Xiao Chen, Pascaline Dupas, Jeremy Weinstein, Marshall Burke, David Lobell, and Stefano Ermon. Infrastructure quality assessment in africa using satellite imagery and deep learning. In Proc. 24th SIGKDD Conference, 2018.","title":"Infrastructure quality assessment in africa using satellite imagery and deep learning","authors":"Barak Oshri, Annie Hu, Peter Adelson, Xiao Chen, Pascaline Dupas, Jeremy Weinstein, Marshall Burke, David Lobell, Stefano Ermon","year":"2018","venue":"Proc. 24th SIGKDD Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pan et al., 2010] Sinno Jialin Pan, Qiang Yang, et al. A sur- vey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Qiang Sinno Jialin Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raina et al., 2007] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learn- ing: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pages 759-766. ACM, 2007.","title":"Self-taught learning: transfer learning from unlabeled data","authors":"Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, Andrew Y Ng","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ratner et al., 2017] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak su- pervision. arXiv preprint arXiv:1711.10160, 2017.","title":"Snorkel: Rapid training data creation with weak supervision","authors":"Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, Christopher Ré","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ren et al., 2018] Hongyu Ren, Russell Stewart, Jiaming Song, Volodymyr Kuleshov, and Stefano Ermon. Adver- sarial constraint learning for structured prediction. CoRR, abs/1805.10561, 2018.","title":"Adversarial constraint learning for structured prediction","authors":"Russell Hongyu Ren, Jiaming Stewart, Volodymyr Song, Stefano Kuleshov, Ermon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/366"},{"position":22,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sheehan et al., 2019] Evan Sheehan, Chenlin Meng, Matthew Tan, Burak Uzkent, Neal Jean, David Lobell, Marshall Burke, and Stefano Ermon. Predicting economic development using geolocated wikipedia articles. In Proc. 25th SIGKDD Conference, 2019.","title":"Predicting economic development using geolocated wikipedia articles","authors":"Evan Sheehan, Chenlin Meng, Matthew Tan, Burak Uzkent, Neal Jean, David Lobell, Marshall Burke, Stefano Ermon","year":"2019","venue":"Proc. 25th SIGKDD Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stewart and Ermon, 2017] Russell Stewart and Stefano Er- mon. Label-free supervision of neural networks with physics and domain knowledge. In AAAI, 2017.","title":"Label-free supervision of neural networks with physics and domain knowledge","authors":"Russell Stewart, Stefano Ermon","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2018. [Wikipedia, 2018] Wikipedia. Wikipedia, the free encyclo- pedia, 2018.","title":"Learning two-branch neural networks for image-text matching tasks","authors":"Liwei Wang, Yin Li, Jing Huang, Svetlana Lazebnik","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}