{"type":"ScholarlyArticle","title":"ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces (Extended Abstract)","authors":[{"name":"Hariharan Subramonyam","slug":"hariharan-subramonyam","given":"Hariharan","family":"Subramonyam","affiliation":"Stanford University"},{"name":"Colleen Seifert","slug":"colleen-seifert","given":"Colleen","family":"Seifert","affiliation":"University of Michigan"},{"name":"Eytan Adar","slug":"eytan-adar","given":"Eytan","family":"Adar","affiliation":"University of Michigan"}],"abstract":"When prototyping AI experiences (AIX), interface designers seek effective ways to support end-user tasks through AI capabilities. However, AI poses challenges to design due to its dynamic behavior in response to training data, end-user data, and feedback. Designers must consider AI's uncertainties and offer adaptations such as explainability, error recovery, and automation vs. human task control. Unfortunately, current prototyping tools assume a black-box view of AI, forcing designers to work with separate tools to explore machine learning models, understand model performance, and align interface choices with model behavior. This introduces friction to rapid and iterative prototyping. We propose Model-Informed Prototyping (MIP), a workflow for AIX design that combines model exploration with UI prototyping tasks. Our system, ProtoAI, allows designers to directly incorporate model outputs into interface designs, evaluate design choices across different inputs, and iteratively revise designs by analyzing model breakdowns.","doi":"10.24963/ijcai.2022/748","url":"https://ijcai.org/proceedings/2022/748","pdf":"https://ijcai.org/proceedings/2022/0748.pdf","markdown":"https://ijcai.org/proceedings/2022/748.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/748","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":748,"pages":{"first":5344,"last":5348},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987274","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985417","length_sec":541,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amershi et al., 2015] Saleema Amershi, Max Chickering, Steven M Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. Modeltracker: Redesigning performance analy- sis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 337-346, 2015.","title":"Modeltracker: Redesigning performance analysis tools for machine learning","authors":"Saleema Amershi, Max Chickering, Steven M Drucker, Bongshin Lee, Patrice Simard, Jina Suh","year":"2015","venue":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amershi et al., 2019] Saleema Amershi, Dan Weld, Mi- haela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Pro- ceedings of the 2019 CHI Conference on Human Factors in Computing Systems, page 3. ACM, 2019.","title":"Guidelines for human-ai interaction","authors":"Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen","year":"2019","venue":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beaudouin-Lafon and Mackay, 2009] Michel Beaudouin- Lafon and Wendy E Mackay. Prototyping tools and techniques. In Human-Computer Interaction, pages 137-160. CRC Press, 2009.","title":"Prototyping tools and techniques","authors":"Beaudouin-Lafon, Michel Mackay, Wendy E Beaudouin-Lafon, Mackay","year":"2009","venue":"Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellamy et al., 2018] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943, 2018.","title":"Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias","authors":"Rachel Ke Bellamy, Kuntal Dey, Michael Hind, Stephanie Samuel C Hoffman, Kalapriya Houde, Pranay Kannan, Jacquelyn Lohia, Sameep Martino, Aleksandra Mehta, Mojsilovic","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Browne, 2019] Jacob T Browne. Wizard of oz prototyping for machine learning experiences. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Com- puting Systems, pages 1-6, 2019.","title":"Wizard of oz prototyping for machine learning experiences","authors":"T Jacob, Browne","year":"2019","venue":"Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dellermann et al., 2019] Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. The future of human-ai col- laboration: a taxonomy of design knowledge for hybrid intelligence systems. 2019.","title":"The future of human-ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems","authors":"Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, Philipp Ebel","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Google, 2020] Google. Visually probe the behavior of trained machine learning models, with minimal coding., 2020.","title":"Google. Visually probe the behavior of trained machine learning models, with minimal coding","authors":"Google","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hartmann et al., 2006] Björn Hartmann, Scott R Klemmer, Michael Bernstein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of the 19th annual ACM symposium on User interface software and technology, pages 299-308, 2006.","title":"Reflective physical prototyping through integrated design, test, and analysis","authors":"Björn Hartmann, Michael Scott R Klemmer, Leith Bernstein, Brandon Abdulla, Avi Burr, Jennifer Robinson-Mosher, Gee","year":"2006","venue":"Proceedings of the 19th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Helms et al., 2018] Karey Helms, Barry Brown, Magnus Sahlgren, and Airi Lampinen. Design methods to inves- tigate user experiences of artificial intelligence. In 2018 AAAI Spring Symposium Series, 2018.","title":"Design methods to investigate user experiences of artificial intelligence","authors":"Karey Helms, Barry Brown, Magnus Sahlgren, Airi Lampinen","year":"2018","venue":"AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Holmquist, 2017] Lars Erik Holmquist. Intelligence on tap: artificial intelligence as a new design material. interac- tions, 24(4):28-33, 2017.","title":"Intelligence on tap: artificial intelligence as a new design material","authors":"Lars Erik Holmquist, Holmquist","year":"2017","venue":"interactions","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Horvitz, 1999] Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159-166, 1999.","title":"Principles of mixed-initiative user interfaces","authors":"Eric Horvitz","year":"1999","venue":"Proceedings of the SIGCHI conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klemmer et al., 2000] Scott R Klemmer, Anoop K Sinha, Jack Chen, James A Landay, Nadeem Aboobaker, and An- nie Wang. Suede: a wizard of oz prototyping tool for speech user interfaces. In Proceedings of the 13th annual ACM symposium on User interface software and technol- ogy, pages 1-10, 2000.","title":"Suede: a wizard of oz prototyping tool for speech user interfaces","authors":"Anoop K Scott R Klemmer, Jack Sinha, James A Chen, Nadeem Landay, Annie Aboobaker, Wang","year":"2000","venue":"Proceedings of the 13th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maulsby et al., 1993] David Maulsby, Saul Greenberg, and Richard Mander. Prototyping an intelligent agent through wizard of oz. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing sys- tems, pages 277-284, 1993.","title":"Prototyping an intelligent agent through wizard of oz","authors":"David Maulsby, Saul Greenberg, Richard Mander","year":"1993","venue":"Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mitchell et al., 2019] Margaret Mitchell, Simone Wu, An- drew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Pro- ceedings of the conference on fairness, accountability, and transparency, pages 220-229, 2019.","title":"Model cards for model reporting","authors":"Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa, Deborah Raji, Timnit Gebru","year":"2019","venue":"Proceedings of the conference on fairness, accountability, and transparency","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pruitt and Grudin, 2003] John Pruitt and Jonathan Grudin. Personas: practice and theory. In Proceedings of the 2003 conference on Designing for user experiences, pages 1-15, 2003.","title":"Personas: practice and theory","authors":"Grudin Pruitt, John Pruitt, Jonathan Grudin","year":"2003","venue":"Proceedings of the 2003 conference on Designing for user experiences","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on com- puter vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Subramonyam et al., 2021] Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. Towards a process model for co-creating ai experiences. In Designing Interactive Systems Conference 2021, pages 1529-1543, 2021.","title":"Towards a process model for co-creating ai experiences","authors":"Hariharan Subramonyam, Colleen Seifert, Eytan Adar","year":"2021","venue":"Designing Interactive Systems Conference 2021","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Subramonyam et al., 2022] Hariharan Subramonyam, Jane Im, Colleen Seifert, and Eytan Adar. Solving separation- of-concerns problems in collaborative design of human-ai systems through leaky abstractions. In CHI Conference on Human Factors in Computing Systems, pages 1-21, 2022.","title":"Solving separationof-concerns problems in collaborative design of human-ai systems through leaky abstractions","authors":"Hariharan Subramonyam, Jane Im, Colleen Seifert, Eytan Adar","year":"2022","venue":"CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[van Allen, 2018] Philip van Allen. Prototyping ways of pro- totyping ai. interactions, 25(6):46-51, 2018.","title":"Prototyping ways of prototyping ai","authors":"; Van Allen, Philip Van Allen","year":"2018","venue":"interactions","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Winograd et al., 1986] Terry Winograd, Fernando Flores, and Fernando F Flores. Understanding computers and cognition: A new foundation for design. Intellect Books, 1986.","title":"Understanding computers and cognition: A new foundation for design","authors":"Terry Winograd, Fernando Flores, Fernando F Flores","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wood, 1997] Larry E Wood. User interface design: Bridg- ing the gap from user requirements to design. CRC Press, 1997.","title":"User interface design: Bridging the gap from user requirements to design","authors":"E Larry, Wood","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2019] Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T Iqbal, and Jaime Teevan. Sketching nlp: A case study of exploring the right things to design with language intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Sys- tems, page 185. ACM, 2019.","title":"Sketching nlp: A case study of exploring the right things to design with language intelligence","authors":"Qian Yang, Justin Cranshaw, Saleema Amershi, T Shamsi, Jaime Iqbal, Teevan","year":"2019","venue":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2020] Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. Re-examining whether, why, and how human-ai interaction is uniquely difficult to de- sign. In Proceedings of the 2020 chi conference on human factors in computing systems, pages 1-13, 2020.","title":"Re-examining whether, why, and how human-ai interaction is uniquely difficult to design","authors":"Qian Yang, Aaron Steinfeld, Carolyn Rosé, John Zimmerman","year":"2020","venue":"Proceedings of the 2020 chi conference on human factors in computing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}