{"type":"ScholarlyArticle","title":"Place Anything into Any Video","authors":[{"name":"Ziling Liu","slug":"ziling-liu","given":"Ziling","family":"Liu","affiliation":"Southern University of Science and Technology"},{"name":"Jinyu Yang","slug":"jinyu-yang","given":"Jinyu","family":"Yang","affiliation":"Southern University of Science and Technology"},{"name":"Mingqi Gao","slug":"mingqi-gao","given":"Mingqi","family":"Gao","affiliation":"Southern University of Science and Technology"},{"name":"Feng Zheng","slug":"feng-zheng","given":"Feng","family":"Zheng","affiliation":"Southern University of Science and Technology"}],"abstract":"Controllable video editing has demonstrated remarkable potential across diverse applications, particularly in scenarios where capturing or re-capturing real-world videos is either impractical or costly. This paper introduces a novel and efficient system named Place-Anything, which facilitates the insertion of any object into any video solely based on a picture or text description of the target object or element. The system comprises three modules: 3D generation, video reconstruction, and 3D target insertion. This integrated approach offers an efficient and effective solution for producing and editing high-quality videos by naturally inserting realistic objects. Through experiment, we demonstrate that our system can effortlessly place any object into any video using just a photograph of the object. Our demo video can be found at https://youtu.be/afXqgLLRnTE. Please also visit our project page https://place-anything.github.io to get more information.","doi":"10.24963/ijcai.2024/1019","url":"https://ijcai.org/proceedings/2024/1019","pdf":"https://ijcai.org/proceedings/2024/1019.pdf","markdown":"https://ijcai.org/proceedings/2024/1019.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1019","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1019,"pages":{"first":8729,"last":8732},"keywords":["Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Applications","Computer Vision: CV: Scene analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Brooks et al., 2024] Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.","title":"Video generation models as world simulators","authors":"Tim Brooks, Bill Peebles, Connor Homes, Will Depue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin, Ricky Ng, Aditya Wang, Ramesh","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Campos et al., 2021] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transac- tions on Robotics, 37(6):1874-1890, 2021.","title":"Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam","authors":"Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José Mm Montiel, Juan D Tardós","year":"2021","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Engel et al., 2017] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE trans- actions on pattern analysis and machine intelligence, 40(3):611-625, 2017.","title":"Direct sparse odometry","authors":"Engel","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hagemann et al., 2023] Annika Hagemann, Moritz Knorr, and Christoph Stiller. Deep geometry-aware camera self- calibration from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3438-3448, 2023.","title":"Deep geometry-aware camera selfcalibration from video","authors":"Annika Hagemann, Moritz Knorr, Christoph Stiller","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kerbl et al., 2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaus- sian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.","title":"3d gaussian splatting for real-time radiance field rendering","authors":"Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis","year":"2023","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Liu et al., 2023] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309, 2023.","title":"Zero-1-to-3: Zero-shot one image to 3d object","authors":"Liu","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Poole et al., 2022] Ben Poole, Ajay Jain, Jonathan T Bar- ron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.","title":"Dreamfusion: Text-to-3d using 2d diffusion","authors":"Ben Poole, Ajay Jain, Jonathan T Barron, Ben Mildenhall","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Raj et al., 2023] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Bar- ron, et al. Dreambooth3d: Subject-driven text-to-3d gen- eration. arXiv preprint arXiv:2303.13508, 2023.","title":"Dreambooth3d: Subject-driven text-to-3d generation","authors":"Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ravi et al., 2020] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501, 2020.","title":"Accelerating 3d deep learning with pytorch3d","authors":"Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, Georgia Gkioxari","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Tang et al., 2023] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023.","title":"Dreamgaussian: Generative gaussian splatting for efficient 3d content creation","authors":"Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, Gang Zeng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Teed and Deng, 2020] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.","title":"Raft: Recurrent all-pairs field transforms for optical flow","authors":"Zachary Teed, Jia Deng","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}