{"type":"ScholarlyArticle","title":"Captioning Bosch: A Twitter Bot","authors":[{"name":"Cornelia Ferner","slug":"cornelia-ferner","given":"Cornelia","family":"Ferner","affiliation":"Salzburg University of Applied Sciences"}],"abstract":"The artworks by Dutch painter Hieronymus Bosch are well known for their incredible wealth of details. The popular BoschBot regularly posts small segments of the digitized paintings on Twitter, thus relieving their density and making them more accessible. CaptioningBoschBot, the Twitter bot presented in this demo, reverses the creative process of the artist: It uses the out-of-context painting segments as input for an encoder-decoder model to generate captions that interpret the painted objects. As the model was only trained on realistic, photographic images, curious interpretations of the otherworldly details can be observed. The generated captions are again posted on Twitter to encourage discussions about Bosch's masterpieces and the AI technology in general.","doi":"10.24963/ijcai.2022/694","url":"https://ijcai.org/proceedings/2022/694","pdf":"https://ijcai.org/proceedings/2022/0694.pdf","markdown":"https://ijcai.org/proceedings/2022/694.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/694","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Exhibits","track":"AI and Arts - Exhibits","number":694,"pages":{"first":5011,"last":5014},"keywords":["Application domains: Images and visual arts","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Methods and resources: Applications and software frameworks","Application domains: Text, literature and creative language"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987173","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985316","length_sec":317,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Benson, 2019] Louise Benson. Pointless Profundity? How Twitter Art Bots Are Shaking Up Visual Culture. Elephant, 2019. https://elephant.art/pointless-profundity-twitter-art- bots-shaking-visual-culture/, Last accessed on 2022-04- 27.","title":"Pointless Profundity? How Twitter Art Bots Are Shaking Up Visual Culture","authors":"Louise Benson","year":"2019","venue":null,"doi":null,"url":"https://elephant.art/pointless-profundity-twitter-art-bots-shaking-visual-culture/","internal_url":null},{"position":2,"raw":"[Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv preprint arXiv:1504.00325, 2015.","title":"Microsoft COCO Captions: Data Collection and Evaluation Server","authors":"Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C Lawrence, Zitnick","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chu et al., 2012] Zi Chu, Steven Gianvecchio, Haining Wang, and Sushil Jajodia. Detecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg? IEEE Transactions on Dependable and Secure Computing, 9(6):811-824, 2012.","title":"Detecting Automation of Twitter Accounts: Are You a Human","authors":"Zi Chu, Steven Gianvecchio, Haining Wang, Sushil Jajodia","year":"2012","venue":"Bot, or Cyborg? IEEE Transactions on Dependable and Secure Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory Transformer for Image Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587, 2020.","title":"Meshed-memory Transformer for Image Captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierar- chical Image Database. In CVPR09. IEEE, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"J Deng, W Dong, R Socher, L.-J Li, K Li, L Fei-Fei","year":"2009","venue":"CVPR09","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale. In International Conference on Learning Representations. ICLR, 2021.","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fitzpatrick, 2017] L. Kelly Fitzpatrick. Anatomy of a Museum Twitter Bot. Medium, 2017. https://medium.com/berkman-klein-center/anatomy- of-a-museum-twitter-bot-2311d81de243, Last accessed on 2022-04-27.","title":"Anatomy of a Museum Twitter Bot. Medium","authors":"L Kelly Fitzpatrick","year":"2017","venue":null,"doi":null,"url":"https://medium.com/berkman-klein-center/anatomy-of-a-museum-twitter-bot-2311d81de243","internal_url":null},{"position":8,"raw":"[McMullan, 2018] Thomas McMullan. This Bonkers Bot is the Only Twitter Account Worth Following Right Now. Wired, 2018. https://www.wired.co.uk/article/ bosch-twitter-bot-digital-art, Last accessed on 2022-04- 27.","title":"This Bonkers Bot is the Only Twitter Account Worth Following Right Now","authors":"Thomas Mcmullan","year":"2018","venue":"Wired","doi":null,"url":"https://www.wired.co.uk/article/bosch-twitter-bot-digital-art","internal_url":null},{"position":9,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models Are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.","title":"Language Models Are Unsupervised Multitask Learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Varol et al., 2017] Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menczer, and Alessandro Flammini. On- line Human-Bot Interactions: Detection, Estimation, and Characterization. In Eleventh International AAAI Confer- ence on Web and Social Media, pages 280-289. AAAI, 2017.","title":"Online Human-Bot Interactions: Detection, Estimation, and Characterization","authors":"Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menczer, Alessandro Flammini","year":"2017","venue":"Eleventh International AAAI Conference on Web and Social Media","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32nd International Conference on Machine Learning, pages 2048-2057. PMLR, 2015.","title":"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}