{"type":"ScholarlyArticle","title":"Geolocating Images with Crowdsourcing and Diagramming","authors":[{"name":"Rachel Kohler","slug":"rachel-kohler","given":"Rachel","family":"Kohler","affiliation":"Department of Computer Science and Center for HCI , Virginia Tech"},{"name":"John Purviance","slug":"john-purviance","given":"John","family":"Purviance","affiliation":"Department of Computer Science and Center for HCI , Virginia Tech"},{"name":"Kurt Luther","slug":"kurt-luther","given":"Kurt","family":"Luther","affiliation":"Department of Computer Science and Center for HCI , Virginia Tech"}],"abstract":"Many types of investigative work involve verifying the legitimacy of visual evidence by identifying the precise geographic location where a photo or video was taken. Professional geolocation is often a manual, time-consuming process that can involve searching large areas of satellite imagery for potential matches. In this paper, we explore how crowdsourcing can be used to support expert image geolocation. We adapt an expert diagramming technique to overcome spatial reasoning limitations of novice crowds so that they can support an expert's search. In an experiment (n=540), we found that diagrams work significantly better than ground-level photos and allow crowds to reduce a search area by half before any expert intervention. We also discuss hybrid approaches to complex image analysis combining crowds, experts, and computer vision.","doi":"10.24963/ijcai.2018/741","url":"https://ijcai.org/proceedings/2018/741","pdf":"https://ijcai.org/proceedings/2018/0741.pdf","markdown":"https://ijcai.org/proceedings/2018/741.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/741","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":741,"pages":{"first":5299,"last":5303},"keywords":["Computer Vision: Computer Vision","Humans and AI: Human-Computer Interaction","Humans and AI: Human Computation and Crowdsourcing"],"videos":[],"references":[{"position":1,"raw":"[Barot, 2014] Trushar Barot. Verifying Images. In Verifi- cation Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. January 2014.","title":"Trushar Barot. Verifying Images","authors":"Barot","year":"2014","venue":"Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bianchetti and MacEachren, 2015] Raechel A. Bianchetti and Alan M. MacEachren. Cognitive Themes Emerging from Air Photo Interpretation Texts Published to 1960. IS- PRS International Journal of Geo-Information, 4(2):551- 571, April 2015.","title":"Cognitive Themes Emerging from Air Photo Interpretation Texts Published to 1960","authors":"A Raechel, Alan M Bianchetti, Maceachren","year":"2015","venue":"IS-PRS International Journal of Geo-Information","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bigham et al., 2010] Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: Nearly Real-time Answers to Visual Questions. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Tech- nology, UIST '10, pages 333-342, New York, NY, USA, 2010. ACM.","title":"VizWiz: Nearly Real-time Answers to Visual Questions","authors":"Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, Tom Yeh","year":"2010","venue":"Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST '10","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brandtzaeg et al., 2016] Petter Bae Brandtzaeg, Marika Lüders, Jochen Spangenberg, Linda Rath-Wiggins, and Asbjørn Følstad. Emerging Journalistic Verification Prac- tices Concerning Social Media. Journalism Practice, 10(3):323-342, April 2016.","title":"Emerging Journalistic Verification Practices Concerning Social Media","authors":"Petter Bae Brandtzaeg, Marika Lüders, Jochen Spangenberg, Linda Rath-Wiggins, Asbjørn Følstad","year":"2016","venue":"Journalism Practice","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghouaiel and Lefèvre, 2016] Nehla Ghouaiel and Sébastien Lefèvre. Coupling ground-level panoramas and aerial im- agery for change detection. Geo-spatial Information Sci- ence, 19(3):222-232, July 2016.","title":"Nehla Ghouaiel and Sébastien Lefèvre. Coupling ground-level panoramas and aerial imagery for change detection","authors":"Lefèvre Ghouaiel","year":"2016","venue":"Geo-spatial Information Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Greenberg et al., 2015] Michael D. Greenberg, Matthew W. Easterday, and Elizabeth M. Gerber. Critiki: A Scaf- folded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of ACM Creativity & Cog- nition 2015, Glasgow, Scotland, 2015. ACM.","title":"Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers","authors":"D Michael, Matthew W Greenberg, Elizabeth M Easterday, Gerber","year":"2015","venue":"Proceedings of ACM Creativity & Cognition 2015","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hara et al., 2015] Kotaro Hara, Shiri Azenkot, Megan Campbell, Cynthia L. Bennett, Vicki Le, Sean Pannella, Robert Moore, Kelly Minckler, Rochelle H. Ng, and Jon E. Froehlich. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Lo- cations with Google Street View: An Extended Analysis. ACM Trans. Access. Comput., 6(2):5:1-5:23, March 2015.","title":"Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View: An Extended Analysis","authors":"Kotaro Hara, Shiri Azenkot, Megan Campbell, Cynthia L Bennett, Vicki Le, Sean Pannella, Robert Moore, Kelly Minckler, Rochelle H Ng, Jon E Froehlich","year":"2015","venue":"ACM Trans. Access. Comput","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hays and Efros, 2008] J. Hays and A. A. Efros. IM2gps: estimating geographic information from a single image. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, June 2008.","title":"IM2gps: estimating geographic information from a single image","authors":"J Hays, A A Efros","year":"2008","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Higgins, 2014] Eliot Higgins. A Beginner's Guide to Ge- olocating Videos, July 2014.","title":"A Beginner's Guide to Geolocating Videos","authors":"Eliot Higgins, Higgins","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Just and Carpenter, 1985] Marcel Adam Just and Patricia A. Carpenter. Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial abil- ity. Psychological Review, 92(2):137-172, April 1985.","title":"Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability","authors":"Carpenter ; Just, Adam Marcel, Patricia A Just, Carpenter","year":"1985","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kerle and Hoffman, 2013] N. Kerle and R. R. Hoffman. Collaborative damage mapping for emergency response: the role of Cognitive Systems Engineering. Nat. Hazards Earth Syst. Sci., 13(1):97-113, January 2013.","title":"Collaborative damage mapping for emergency response: the role of Cognitive Systems Engineering","authors":"N Kerle, R R Hoffman","year":"2013","venue":"Nat. Hazards Earth Syst. Sci","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kohler and Luther, 2017] Rachel Kohler and Kurt Luther. Crowdsourced Image Geolocation as Collective Intelli- gence. In Collective Intelligence 2017, New York, NY, USA, 2017.","title":"Crowdsourced Image Geolocation as Collective Intelli-gence","authors":"Rachel Kohler, Kurt Luther","year":"2017","venue":"Collective Intelligence 2017","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kohler et al., 2017] Rachel Kohler, John Purviance, and Kurt Luther. Supporting Image Geolocation with Dia- gramming and Crowdsourcing. In Fifth AAAI Conference on Human Computation and Crowdsourcing, September 2017.","title":"Supporting Image Geolocation with Diagramming and Crowdsourcing","authors":"Rachel Kohler, John Purviance, Kurt Luther","year":"2017","venue":"Fifth AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kolbe et al., 2005] Thomas H. Kolbe, Gerhard Gröger, and Lutz Plümer. CityGML: Interoperable Access to 3d City Models. In Professor Dr Peter van Oosterom, Dr Siyka Zlatanova, and Elfriede M. Fendel, editors, Geo- information for Disaster Management, pages 883-899. Springer Berlin Heidelberg, 2005.","title":"CityGML: Interoperable Access to 3d City Models","authors":"H Thomas, Gerhard Kolbe, Lutz Gröger, Plümer","year":"2005","venue":"Geoinformation for Disaster Management","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2015] Tsung-Yi Lin, Yin Cui, Serge Belongie, and James Hays. Learning Deep Representations for Ground-to-Aerial Geolocalization. In Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015.","title":"Learning Deep Representations for Ground-to-Aerial Geolocalization","authors":"Tsung-Yi Lin, Yin Cui, Serge Belongie, James Hays","year":"2015","venue":"Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mehta et al., 2016] Sneha Mehta, Chris North, and Kurt Luther. An Exploratory Study of Human Performance in Image Geolocation Tasks. In HCOMP 2016 GroupSight Workshop on Human Computation for Image and Video Analysis, Austin, TX, USA, 2016. AAAI.","title":"An Exploratory Study of Human Performance in Image Geolocation Tasks","authors":"Sneha Mehta, Chris North, Kurt Luther","year":"2016","venue":"HCOMP 2016 GroupSight Workshop on Human Computation for Image and Video Analysis","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Meier, 2015] Patrick Meier. Digital Humanitarians: How Big Data Is Changing the Face of Humanitarian Response. Routledge, Boca Raton, FL, null edition edition, January 2015.","title":"Digital Humanitarians: How Big Data Is Changing the Face of Humanitarian Response","authors":"Patrick Meier","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Noronha et al., 2011] Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos. Platemate: crowdsourc- ing nutritional analysis from food photographs. In Pro- ceedings of the 24th annual ACM symposium on User in- terface software and technology, UIST '11, pages 1-12, New York, NY, USA, 2011. ACM.","title":"Platemate: crowdsourcing nutritional analysis from food photographs","authors":"Jon Noronha, Eric Hysen, Haoqi Zhang, Krzysztof Z Gajos","year":"2011","venue":"Proceedings of the 24th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Painho et al., 2010] Marcos Painho, Maribel Yasmina San- tos, and Hardy Pundt, editors. Geospatial Thinking. Springer, 2010.","title":"Geospatial Thinking","authors":null,"year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Weyand et al., 2016] Tobias Weyand, Ilya Kostrikov, and James Philbin. PlaNet -Photo Geolocation with Convolu- tional Neural Networks. arXiv:1602.05314 [cs], February 2016. arXiv: 1602.05314.","title":"PlaNet -Photo Geolocation with Convolutional Neural Networks","authors":"Weyand","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wiggins and Crowston, 2014] Andrea Wiggins and Kevin Crowston. Surveying the citizen science landscape. First Monday, 20(1), December 2014.","title":"Surveying the citizen science landscape","authors":"Crowston Wiggins, Andrea Wiggins, Kevin Crowston","year":"2014","venue":"First Monday","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wollan, 2004] Helen Wollan. Incorporating Heuristically Generated Search Patterns in Search and Rescue. Master's thesis, University of Edinburgh, 2004.","title":"Incorporating Heuristically Generated Search Patterns in Search and Rescue","authors":"Helen Wollan, Wollan","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zacks et al., 2000] Jeffrey M. Zacks, Jon Mires, Barbara Tversky, and Eliot Hazeltine. Mental spatial transforma- tions of objects and perspective. Spatial Cognition and Computation, 2(4):315-332, December 2000.","title":"Mental spatial transformations of objects and perspective","authors":"Jeffrey M Zacks, Jon Mires, Barbara Tversky, Eliot Hazeltine","year":"2000","venue":"Spatial Cognition and Computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhai et al., 2016] Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs. Predicting Ground-Level Scene Layout from Aerial Imagery. arXiv:1612.02709 [cs], December 2016. arXiv: 1612.02709.","title":"Predicting Ground-Level Scene Layout from Aerial Imagery","authors":"Zhai","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}