{"type":"ScholarlyArticle","title":"VideoMaster: A Multimodal Micro Game Video Recreator","authors":[{"name":"Yipeng Yu","slug":"yipeng-yu","given":"Yipeng","family":"Yu","affiliation":""},{"name":"Xiao Chen","slug":"xiao-chen","given":"Xiao","family":"Chen","affiliation":""},{"name":"Hui Zhan","slug":"hui-zhan","given":"Hui","family":"Zhan","affiliation":""}],"abstract":"To free human from laborious video production, this paper proposes the building of VideoMaster, a multimodal system equipped with four capabilities: highlight extraction, video describing, video dubbing and video editing. It extracts interesting episodes from long game videos, generates subtitles for each episode, reads the subtitles through synthesized speech, and finally re-creates a better short video through video editing. Notably, VideoMaster takes a combination of deep learning and traditional computer vision techniques to extract highlights with fine-to-coarse labels, utilizes a novel framework named PCSG-v (probabilistic context sensitive grammar for video) for video description generation, and imitates a target speaker's voice to read the description. To the best of our knowledge, VideoMaster is the first multimedia system that can automatically produce product-level micro-videos without heavy human annotation.","doi":"10.24963/ijcai.2023/844","url":"https://ijcai.org/proceedings/2023/844","pdf":"https://ijcai.org/proceedings/2023/0844.pdf","markdown":"https://ijcai.org/proceedings/2023/844.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/844","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Demo Track","track":"Demo Track","number":844,"pages":{"first":7179,"last":7182},"keywords":["Computer Vision: CV: Video analysis and understanding","Natural Language Processing: NLP: Applications","Computer Vision: CV: Applications","Natural Language Processing: NLP: Language generation","Computer Vision: CV: Vision and language","Machine Learning: ML: Multi-modal learning","Computer Vision: CV: Segmentation","Natural Language Processing: NLP: Speech","Multidisciplinary Topics and Applications: MDA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Borsos et al., 2022] Zalán Borsos, Raphaël Marinier, et al. Audiolm: a language modeling approach to audio genera- tion. arXiv:2209.03143, 2022.","title":"Audiolm: a language modeling approach to audio generation","authors":"Borsos","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bradski and Kaehler, 2008] Gary Bradski and Adrian Kaehler. Learning OpenCV: Computer vision with the OpenCV library. O'Reilly Media, Inc., 2008.","title":"Learning OpenCV: Computer vision with the OpenCV library","authors":"Kaehler Bradski, Gary Bradski, Adrian Kaehler","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Derpanis, 2010] Konstantinos G Derpanis. Overview of the ransac algorithm. Image Rochester NY, 4(1):2-3, 2010.","title":"Overview of the ransac algorithm","authors":"G Konstantinos, Derpanis","year":"2010","venue":"Image Rochester NY","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Leutenegger et al., 2011] Stefan Leutenegger, Margarita Chli, and Roland Y Siegwart. Brisk: Binary robust in- variant scalable keypoints. In ICCV, pages 2548-2555, 2011.","title":"Brisk: Binary robust invariant scalable keypoints","authors":"Stefan Leutenegger, Margarita Chli, Roland Y Siegwart","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2021] Tingtian Li, Zixun Sun, Haoruo Zhang, Jin Li, Ziming Wu, Hui Zhan, Yipeng Yu, and Hengcan Shi. Deep music retrieval for fine-grained videos by exploiting cross-modal-encoded voice-overs. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 1880-1884, 2021.","title":"Deep music retrieval for fine-grained videos by exploiting cross-modal-encoded voice-overs","authors":"Li","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lowe, 2004] G Lowe. Sift-the scale invariant feature trans- form. Int. J, 2:91-110, 2004.","title":"Sift-the scale invariant feature transform","authors":"Lowe","year":"2004","venue":"Int. J","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeff Wu, Xu Jiang, et al. Training language models to follow instructions with hu- man feedback. arXiv:2203.02155, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeff Wu, Xu Jiang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchi- cal text-conditional image generation with clip latents. arXiv:2204.06125, 2022.","title":"Hierarchical text-conditional image generation with clip latents","authors":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In NeurIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rublee et al., 2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In ICCV, 2011.","title":"Orb: An efficient alternative to sift or surf","authors":"Rublee","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Saharia et al., 2022] Chitwan Saharia, William Chan, Saurabh Saxena, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205.11487, 2022.","title":"Photorealistic text-to-image diffusion models with deep language understanding","authors":"Saharia","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Scao et al., 2022] Teven Le Scao, Angela Fan, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv:2211.05100, 2022.","title":"A 176b-parameter open-access multilingual language model","authors":"Scao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Singer et al., 2022] Uriel Singer, Adam Polyak, Thomas Hayes, et al. Make-a-video: Text-to-video generation without text-video data. arXiv:2209.14792, 2022.","title":"Make-a-video: Text-to-video generation without text-video data","authors":"Singer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thanh et al., 2009] Nguyen Duc Thanh, Wanqing Li, and Philip Ogunbona. An improved template matching method for object detection. In ACCV, 2009.","title":"An improved template matching method for object detection","authors":"Thanh Nguyen Duc, Wanqing Li, Philip Ogunbona","year":"2009","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2017a] Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. Skyfire: Data-driven seed generation for fuzzing. In S&P, 2017.","title":"Skyfire: Data-driven seed generation for fuzzing","authors":"Wang","year":"2017","venue":"S&P","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017b] Yuxuan Wang, R.J. Skerry-Ryan, et al. Tacotron: Towards end-to-end speech synthesis. In IN- TERSPEECH, 2017.","title":"Tacotron: Towards end-to-end speech synthesis","authors":"Wang","year":"2017","venue":"IN-TERSPEECH","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Yuxuan Wang, Daisy Stanton, et al. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In ICML, 2018.","title":"Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis","authors":"Wang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiao et al., 2022] Xinyu Xiao, Zixun Sun, Tingtian Li, and Yipeng Yu. Relational graph reasoning transformer for im- age captioning. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, 2022.","title":"Relational graph reasoning transformer for image captioning","authors":"Xinyu Xiao, Zixun Sun, Tingtian Li, Yipeng Yu","year":"2022","venue":"2022 IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2022] Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In EMNLP, 2022.","title":"Re3: Generating longer stories with recursive reprompting and revision","authors":"Kevin Yang, Nanyun Peng, Yuandong Tian, Dan Klein","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2021] Yipeng Yu, Zirui Tu, Longyu Lu, Xiao Chen, Hui Zhan, and Zixun Sun. Text2video: Automatic video generation based on text scripts. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 2753-2755, 2021.","title":"Text2video: Automatic video generation based on text scripts","authors":"Yipeng Yu, Zirui Tu, Longyu Lu, Xiao Chen, Hui Zhan, Zixun Sun","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia, MM '21","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang and Krieger, 2011] Yi Zhang and Hans-Ulrich Krieger. Large-scale corpus-driven pcfg approximation of an hpsg. In IWPT, pages 198-208, 2011.","title":"Large-scale corpus-driven pcfg approximation of an hpsg","authors":"Yi Zhang, Hans-Ulrich Krieger","year":"2011","venue":"IWPT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Yaping Zhang, Shuai Nie, Wenju Liu, Xing Xu, Dongxiang Zhang, and Heng Tao Shen. Sequence-to-sequence domain adaptation network for ro- bust text image recognition. In CVPR, 2019.","title":"Sequence-to-sequence domain adaptation network for robust text image recognition","authors":"Yaping Zhang, Shuai Nie, Wenju Liu, Xing Xu, Dongxiang Zhang, Heng Tao Shen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}