Captioning Bosch: A Twitter Bot

Captioning Bosch: A Twitter Bot

Cornelia Ferner

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
AI and Arts - Exhibits. Pages 5011-5014. https://doi.org/10.24963/ijcai.2022/694

The artworks by Dutch painter Hieronymus Bosch are well known for their incredible wealth of details. The popular BoschBot regularly posts small segments of the digitized paintings on Twitter, thus relieving their density and making them more accessible. CaptioningBoschBot, the Twitter bot presented in this demo, reverses the creative process of the artist: It uses the out-of-context painting segments as input for an encoder-decoder model to generate captions that interpret the painted objects. As the model was only trained on realistic, photographic images, curious interpretations of the otherworldly details can be observed. The generated captions are again posted on Twitter to encourage discussions about Bosch's masterpieces and the AI technology in general.
Keywords:
Application domains: Text, literature and creative language
Application domains: Images and visual arts
Methods and resources: Applications and software frameworks
Methods and resources: Machine learning, deep learning, neural models, reinforcement learning