{"type":"ScholarlyArticle","title":"Reagent: Converting Ordinary Webpages into Interactive Software Agents","authors":[{"name":"Matthew Peveler","slug":"matthew-peveler","given":"Matthew","family":"Peveler","affiliation":"Rensselaer Polytechnic Institute , Troy , NY 12180 , USA"},{"name":"Jeffrey O. Kephart","slug":"jeffrey-o-kephart","given":"Jeffrey O.","family":"Kephart","affiliation":"IBM Thomas J. Watson Research Center , Yorktown Heights , NY 10598 , USA"},{"name":"Hui Su","slug":"hui-su","given":"Hui","family":"Su","affiliation":"Rensselaer Polytechnic Institute , Troy , NY 12180 , USA; IBM Thomas J. Watson Research Center , Yorktown Heights , NY 10598 , USA"}],"abstract":"We introduce Reagent, a technology that can be used in conjunction with automated speech recognition to allow users to query and manipulate ordinary webpages via speech and pointing. Reagent can be used out-of-the-box with third-party websites, as it requires neither special instrumentation from website developers nor special domain knowledge to capture semantically-meaningful mouse interactions with structured elements such as tables and plots. When it is unable to infer mappings between domain vocabulary and visible webpage content on its own, Reagent proactively seeks help by engaging in a voice-based interaction with the user.","doi":"10.24963/ijcai.2019/956","url":"https://ijcai.org/proceedings/2019/956","pdf":"https://ijcai.org/proceedings/2019/0956.pdf","markdown":"https://ijcai.org/proceedings/2019/956.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/956","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Demos","track":"Demos","number":956,"pages":{"first":6560,"last":6562},"keywords":["AI: Natural Language Processing","AI: Human-Computer Interactive Systems","AI: Knowledge Representation and Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Allen et al., 2019] David Allen, Rahul Divekar, Jaimie Drozdal, Lilit Balagyozyan, Shuyue Zheng, Ziyi Song, Huang Zou, Jeramey Tyler, Xiangyang Mou, Rui Zhao, Helen Zhou, Jianling Yue, Jeffrey O. Kephart, and Hui Su. The rensselaer mandarin project -a cognitive and im- mersive language learning environment. In Proceedings of AAAI 2019, 2019.","title":"The rensselaer mandarin project -a cognitive and immersive language learning environment","authors":"David Allen, Rahul Divekar, Jaimie Drozdal, Lilit Balagyozyan, Shuyue Zheng, Ziyi Song, Huang Zou, Jeramey Tyler, Xiangyang Mou, Rui Zhao, Helen Zhou, Jianling Yue, Jeffrey O Kephart, Hui Su","year":"2019","venue":"Proceedings of AAAI 2019","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2007] Sören Auer, Christian Bizer, Georgi Ko- bilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In Pro- ceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC'07/ASWC'07, pages 722-735, Berlin, Heidelberg, 2007. Springer-Verlag.","title":"Dbpedia: A nucleus for a web of open data","authors":"Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, Zachary Ives","year":"2007","venue":"Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC'07/ASWC'07","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Divekar et al., 2018] Rahul R. Divekar, Matthew Peveler, Robert Rouhani, Rui Zhao, Jeffrey O. Kephart, David Allen, Kang Wang, Qiang Ji, and Hui Su. CIRA: An archi- tecture for building configurable immersive smart-rooms. In Advances in Intelligent Systems and Computing, pages 76-95. Springer International Publishing, nov 2018.","title":"CIRA: An architecture for building configurable immersive smart-rooms","authors":"R Rahul, Matthew Divekar, Robert Peveler, Rui Rouhani, Jeffrey O Zhao, David Kephart, Kang Allen, Qiang Wang, Hui Ji, Su","year":"2018","venue":"Advances in Intelligent Systems and Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Farrell et al., 2016] Robert G Farrell, Jonathan Lenchner, Jeffrey O Kephart, Alan Webb, Michael Muller, Thomas Erickson, David Melville, Rachel Bellamy, Daniel Gruen, Jonathan Connell, Danny Soroker, Andy Aaron, Shari Trewin, Maryam Ashoori, Jason Ellis, Brian Gaucher, and Dario Gil. Symbiotic cognitive computing. AI Magazine, 37(3):81-93, 2016.","title":"Symbiotic cognitive computing","authors":"Jonathan Robert G Farrell, Jeffrey O Lenchner, Alan Kephart, Michael Webb, Thomas Muller, David Erickson, Rachel Melville, Daniel Bellamy, Jonathan Gruen, Danny Connell, Andy Soroker, Shari Aaron, Maryam Trewin, Jason Ashoori, Brian Ellis, Dario Gaucher, Gil","year":"2016","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Joshi and Liu, 2009] Parag Mulendra Joshi and Sam Liu. Web document text and images extraction using dom anal- ysis and natural language processing. In Proceedings of the 9th ACM Symposium on Document Engineering, Do- cEng '09, pages 218-221, New York, NY, USA, 2009. ACM.","title":"Web document text and images extraction using dom analysis and natural language processing","authors":"Mulendra Parag, Sam Joshi, Liu","year":"2009","venue":"Proceedings of the 9th ACM Symposium on Document Engineering, Do-cEng '09","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kephart et al., 2018] Jeffrey O. Kephart, Victor Dibia, Ja- son Ellis, Biplav Srivastava, Kartik Talamadupula, and Mishal Dholakia. A cognitive assistant for visualizing and analyzing exoplanets. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"A cognitive assistant for visualizing and analyzing exoplanets","authors":"Jeffrey O Kephart, Victor Dibia, Jason Ellis, Biplav Srivastava, Kartik Talamadupula, Mishal Dholakia","year":"2018","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Sun et al., 2011] Fei Sun, Dandan Song, and Lejian Liao. Dom based content extraction via text density. In Pro- ceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pages 245-254, New York, NY, USA, 2011. ACM.","title":"Dom based content extraction via text density","authors":"Fei Sun, Dandan Song, Lejian Liao","year":"2011","venue":"Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}