Re:Draw - Context Aware Translation as a Controllable Method for Artistic Production

Re:Draw - Context Aware Translation as a Controllable Method for Artistic Production

João Libório Cardoso, Francesco Banterle, Paolo Cignoni, Michael Wimmer

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence
AI, Arts & Creativity. Pages 7609-7617. https://doi.org/10.24963/ijcai.2024/842

We introduce context-aware translation, a novel method that combines the benefits of inpainting and image-to-image translation, respecting simultaneously the original input and contextual relevance -- where existing methods fall short. By doing so, our method opens new avenues for the controllable use of AI within artistic creation, from animation to digital art. As an use case, we apply our method to redraw any hand-drawn animated character eyes based on any design specifications -- eyes serve as a focal point that captures viewer attention and conveys a range of emotions; however, the labor-intensive nature of traditional animation often leads to compromises in the complexity and consistency of eye design. Furthermore, we remove the need for production data for training and introduce a new character recognition method that surpasses existing work by not requiring fine-tuning to specific productions. This proposed use case could help maintain consistency throughout production and unlock bolder and more detailed design choices without the production cost drawbacks. A user study shows context-aware translation is preferred over existing work 95.16% of the time.
Keywords:
Application domains: Images, movies and visual arts
Application domains: Computer Graphics and Animation
Methods and resources: AI systems for collaboration and co-creation
Methods and resources: Machine learning, deep learning, neural models, reinforcement learning
Theory and philosophy of arts and creativity in AI systems: Social (multi-agent) creativity and human-computer co-creation