{"type":"ScholarlyArticle","title":"ScreenAgent: A Vision Language Model-driven Computer Control Agent","authors":[{"name":"Runliang Niu","slug":"runliang-niu","given":"Runliang","family":"Niu","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Jindong Li","slug":"jindong-li","given":"Jindong","family":"Li","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Shiqi Wang","slug":"shiqi-wang","given":"Shiqi","family":"Wang","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Yali Fu","slug":"yali-fu","given":"Yali","family":"Fu","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Xiyu Hu","slug":"xiyu-hu","given":"Xiyu","family":"Hu","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Xueyuan Leng","slug":"xueyuan-leng","given":"Xueyuan","family":"Leng","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"He Kong","slug":"he-kong","given":"He","family":"Kong","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Yi Chang","slug":"yi-chang","given":"Yi","family":"Chang","affiliation":"School of Artificial Intelligence , Jilin University; Engineering Research Center of Knowledge-Driven Human-Machine Intelligence , Ministry of Education , China"},{"name":"Qi Wang","slug":"qi-wang","given":"Qi","family":"Wang","affiliation":"School of Artificial Intelligence , Jilin University; Engineering Research Center of Knowledge-Driven Human-Machine Intelligence , Ministry of Education , China"}],"abstract":"Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphical User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing daily computer tasks. Finally, we train a model, ScreenAgent, which achieves comparable computer control capabilities to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code and more detailed information are at https://github.com/niuzaisheng/ScreenAgent.","doi":"10.24963/ijcai.2024/711","url":"https://ijcai.org/proceedings/2024/711","pdf":"https://ijcai.org/proceedings/2024/0711.pdf","markdown":"https://ijcai.org/proceedings/2024/711.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/711","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":711,"pages":{"first":6433,"last":6441},"keywords":["Computer Vision: CV: Vision, language and reasoning","Agent-based and Multi-agent Systems: MAS: Human-agent interaction","Natural Language Processing: NLP: Resources and evaluation","Natural Language Processing: NLP: Dialogue and interactive systems"],"videos":[],"references":[{"position":1,"raw":"[Burns et al., 2022] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision language nav- igation with unknown command feasibility. In European Conference on Computer Vision (ECCV), 2022.","title":"A dataset for interactive vision language navigation with unknown command feasibility","authors":"Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A Plummer","year":"2022","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websrc: A dataset for web-based structural read- ing comprehension, 2021.","title":"Websrc: A dataset for web-based structural reading comprehension","authors":"Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, Kai Yu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2023] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mo- hamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023.","title":"Minigpt-v2: large language model as a unified interface for vision-language multi-task learning","authors":"Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, Mohamed Elhoseiny","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiang et al., 2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.","title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality","authors":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez","year":"2023","venue":null,"doi":null,"url":"https://vicuna.lmsys.org","internal_url":null},{"position":5,"raw":"[Deng et al., 2023] Xiang Deng, Yu Gu, Boyuan Zheng, Shi- jie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.","title":"Mind2web: Towards a generalist agent for the web","authors":"Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2011] Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. From one tree to a forest: a unified solu- tion for structured web data extraction. In Proceedings of the 34th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR '11, page 775-784, New York, NY, USA, 2011. Associa- tion for Computing Machinery.","title":"From one tree to a forest: a unified solution for structured web data extraction","authors":"Qiang Hao, Rui Cai, Yanwei Pang, Lei Zhang","year":"2011","venue":"Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hong et al., 2023] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zi- han Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914, 2023.","title":"Cogagent: A visual language model for gui agents","authors":"Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2020a] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. In Annual Con- ference of the Association for Computational Linguistics (ACL 2020), 2020.","title":"Mapping natural language instructions to mobile ui action sequences","authors":"Li","year":"2020","venue":"Annual Conference of the Association for Computational Linguistics (ACL 2020)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020b] Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements, 2020.","title":"Widget captioning: Generating natural language description for mobile user interface elements","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2023] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023.","title":"Monkey: Image resolution and text label are important things for large multi-modal models","authors":"Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, Xiang Bai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2015] Tsung-Yi Lin, Michael Maire, Serge Be- longie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, Piotr Dollár","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided explo- ration. In International Conference on Learning Represen- tations (ICLR), 2018.","title":"Reinforcement learning on web interfaces using workflow-guided exploration","authors":"Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang","year":"2018","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruc- tion tuning, 2023.","title":"Improved baselines with visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nakano et al., 2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.","title":"Webgpt: Browser-assisted question-answering with human feedback","authors":"Nakano","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nogueira and Cho, 2016] Rodrigo Nogueira and Kyunghyun Cho. End-to-end goal-driven web navi- gation. In Advances In Neural Information Processing Systems, pages 1903-1911, 2016.","title":"Rodrigo Nogueira and Kyunghyun Cho. End-to-end goal-driven web navigation","authors":"Cho Nogueira","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schick et al., 2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.","title":"Toolformer: Language models can teach themselves to use tools","authors":"Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2022] Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205.11029, 2022.","title":"Meta-gui: Towards multi-modal conversational agents on mobile gui","authors":"Sun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21, page 498-510, New York, NY, USA, 2021. Association for Computing Machinery.","title":"Screen2words: Automatic mobile ui summarization with multimodal learning","authors":"Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li","year":"2021","venue":"The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.","title":"Voyager: An openended embodied agent with large language models","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2023b] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023.","title":"A survey on large language model based autonomous agents","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2023c] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Ji- akai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents, 2023.","title":"A survey on large language model based autonomous agents","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2023d] Lei Wang, Jingsen Zhang, Hao Yang, Zhiyuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. When large lan- guage model based agent meets user behavior analy- sis: A novel user simulation paradigm. arXiv preprint arXiv:2306.02552, 2023.","title":"When large language model based agent meets user behavior analysis: A novel user simulation paradigm","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2023e] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023.","title":"Cogvlm: Visual expert for pretrained language models","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2023a] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appa- gent: Multimodal agents as smartphone users. 2023.","title":"Appagent: Multimodal agents as smartphone users","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2023b] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1), 2023.","title":"The dawn of lmms: Preliminary explorations with gpt-4v (ision)","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yao et al., 2023] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023.","title":"Webshop: Towards scalable real-world web interaction with grounded language agents","authors":"Yao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ye et al., 2023] Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, et al. Proagent: From robotic process automation to agentic process automation. arXiv preprint arXiv:2311.10751, 2023.","title":"Proagent: From robotic process automation to agentic process automation","authors":"Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yin et al., 2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A sur- vey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023.","title":"A survey on multimodal large language models","authors":"Yin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}