{"type":"ScholarlyArticle","title":"Towards a Resilient Intelligent Automation System","authors":[{"name":"Segev Shlomov","slug":"segev-shlomov","given":"Segev","family":"Shlomov","affiliation":"IBM Research"},{"name":"Sami Marreed","slug":"sami-marreed","given":"Sami","family":"Marreed","affiliation":"IBM Research"},{"name":"Avi Yaeli","slug":"avi-yaeli","given":"Avi","family":"Yaeli","affiliation":"IBM Research"}],"abstract":"Intelligent Process Automation (IPA) solutions must adapt to changes in user interfaces autonomously, without manual intervention. Addressing this critical challenge and aiming to advance the state-of-the-art, last year we introduced the IPA Challenge competition at IJCAI 2023. This demo paper presents IDA - our novel UI automation solution, developed to tackle complex resiliency issues. Leveraging the capabilities of large language models and employing grounded instructions, our system demonstrates a significant advancement towards resilient IPA. We provide an overview of the IPA Challenge, detail the architecture of our system, and illustrate its effectiveness in overcoming the resiliency challenges. A link to the demo video can be found at:  https://youtu.be/G5nI3V9Umjc","doi":"10.24963/ijcai.2024/1035","url":"https://ijcai.org/proceedings/2024/1035","pdf":"https://ijcai.org/proceedings/2024/1035.pdf","markdown":"https://ijcai.org/proceedings/2024/1035.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1035","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1035,"pages":{"first":8797,"last":8800},"keywords":["Natural Language Processing: NLP: Language grounding","Humans and AI: HAI: Intelligent user interfaces","Humans and AI: HAI: Applications","Agent-based and Multi-agent Systems: MAS: Trust and reputation","Humans and AI: HAI: Other","AI Ethics, Trust, Fairness: ETF: Safety and robustness"],"videos":[],"references":[{"position":1,"raw":"[Automation Anywhere, 2023] Automation Anywhere. www.automationanywhere.com, 2023.","title":"Automation Anywhere","authors":null,"year":"2023","venue":null,"doi":null,"url":"www.automationanywhere.com","internal_url":null},{"position":2,"raw":"[Fuyu-Heavy, 2024] Adept. Fuyu-Heavy. https://www.","title":"Adept. Fuyu-Heavy","authors":"Fuyu-Heavy","year":"2024","venue":null,"doi":null,"url":"https://www","internal_url":null},{"position":3,"raw":"[Guilmette, 2020] Aaron Guilmette. Workflow Automation with Microsoft Power Automate: Achieve digital transfor- mation through business automation with minimal coding. Packt Publishing Ltd, 2020.","title":"Workflow Automation with Microsoft Power Automate: Achieve digital transformation through business automation with minimal coding","authors":"Aaron Guilmette","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2024] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024.","title":"Webvoyager: Building an end-to-end web agent with large multimodal models","authors":"Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[IBM, 2022] IBM. Ibm robotic process automation. www.ibm.com/products/robotic-process-automation, 2022.","title":"IBM. Ibm robotic process automation","authors":null,"year":"2022","venue":null,"doi":null,"url":"www.ibm.com/products/robotic-process-automation","internal_url":null},{"position":6,"raw":"[Koh et al., 2024] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Gra- ham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024.","title":"Visualwebarena: Evaluating multimodal agents on realistic visual web tasks","authors":"Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Le and Gulwani, 2014] Vu Le and Sumit Gulwani. Flashex- tract: A framework for data extraction by examples. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 542-553, 2014.","title":"Flashextract: A framework for data extraction by examples","authors":"Le Vu, Sumit Gulwani","year":"2014","venue":"Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Leshed et al., 2008] Gilly Leshed, Eben M Haber, Tara Matthews, and Tessa Lau. Coscripter: automating & shar- ing how-to knowledge in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1719-1728, 2008.","title":"Coscripter: automating & sharing how-to knowledge in the enterprise","authors":"Gilly Leshed, Eben M Haber, Tara Matthews, Tessa Lau","year":"2008","venue":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Little et al., 2007] Greg Little, Tessa A Lau, Allen Cypher, James Lin, Eben M Haber, and Eser Kandogan. Koala: capture, share, automate, personalize business processes on the web. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 943-946, 2007.","title":"Koala: capture, share, automate, personalize business processes on the web","authors":"Greg Little, Tessa A Lau, Allen Cypher, James Lin, Eben M Haber, Eser Kandogan","year":"2007","venue":"Proceedings of the SIGCHI conference on Human factors in computing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[OpenAI, 2024] OpenAI. (2024). ChatGPT (3.5) OpenAI. https://chat.openai.com, 2024.","title":"ChatGPT (3.5) OpenAI","authors":"Openai","year":"2024","venue":null,"doi":null,"url":"https://chat.openai.com","internal_url":null},{"position":11,"raw":"[Schwartz et al., 2023] Sivan Schwartz, Avi Yaeli, and Segev Shlomov. Enhancing trust in llm-based ai automation agents: New considerations and future challenges, 2023.","title":"Enhancing trust in llm-based ai automation agents: New considerations and future challenges","authors":"Sivan Schwartz, Avi Yaeli, Segev Shlomov","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sugiura and Koseki, 1998] Atsushi Sugiura and Yoshiyuki Koseki. Internet scrapbook: automating web browsing tasks by demonstration. In Proceedings of the 11th annual ACM symposium on User interface software and technol- ogy, pages 9-18, 1998.","title":"Internet scrapbook: automating web browsing tasks by demonstration","authors":"Koseki ; ] Atsushi Sugiura, Yoshiyuki Sugiura, Koseki","year":"1998","venue":"Proceedings of the 11th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tripathi, 2018] Alok Mani Tripathi. Learning Robotic Pro- cess Automation: Create Software robots and automate business processes with the leading RPA tool-UiPath. Packt Publishing Ltd, 2018.","title":"Learning Robotic Process Automation: Create Software robots and automate business processes with the leading RPA tool-UiPath","authors":"Alok Mani, Tripathi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2024] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Ji- tao Sang. Mobile-agent: Autonomous multi-modal mo- bile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024.","title":"Mobile-agent: Autonomous multi-modal mobile device agent with visual perception","authors":"Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Zheng et al., 2024a] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.","title":"Gpt-4v (ision) is a generalist web agent, if grounded","authors":"Zheng","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zheng et al., 2024b] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.","title":"Gpt-4v(ision) is a generalist web agent, if grounded","authors":"Zheng","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}