{"type":"ScholarlyArticle","title":"D-DPCC: Deep Dynamic Point Cloud Compression via 3D Motion Prediction","authors":[{"name":"Tingyu Fan","slug":"tingyu-fan","given":"Tingyu","family":"Fan","affiliation":"Cooperative Medianet Innovation Center , Shanghai Jiao Tong University"},{"name":"Linyao Gao","slug":"linyao-gao","given":"Linyao","family":"Gao","affiliation":"Cooperative Medianet Innovation Center , Shanghai Jiao Tong University"},{"name":"Yiling Xu","slug":"yiling-xu","given":"Yiling","family":"Xu","affiliation":"Cooperative Medianet Innovation Center , Shanghai Jiao Tong University"},{"name":"Zhu Li","slug":"zhu-li","given":"Zhu","family":"Li","affiliation":"University of Missouri , Kansas City"},{"name":"Dong Wang","slug":"dong-wang","given":"Dong","family":"Wang","affiliation":"Guangdong OPPO Mobile Telecommunications Corp. , Ltd."}],"abstract":"The non-uniformly distributed nature of the 3D Dynamic Point Cloud (DPC) brings significant challenges to its high-efficient inter-frame compression. This paper proposes a novel 3D sparse convolution-based Deep Dynamic Point Cloud Compression (D-DPCC) network to compensate and compress the DPC geometry with 3D motion estimation and motion compensation in the feature space. In the proposed D-DPCC network, we design a Multi-scale Motion Fusion (MMF) module to accurately estimate the 3D optical flow between the feature representations of adjacent point cloud frames. Specifically, we utilize a 3D sparse convolution-based encoder to obtain the latent representation for motion estimation in the feature space and introduce the proposed MMF module for fused 3D motion embedding. Besides, for motion compensation, we propose a 3D Adaptively Weighted Interpolation (3DAWI) algorithm with a penalty coefficient to adaptively decrease the impact of distant neighbours. We compress the motion embedding and the residual with a lossy autoencoder-based network. To our knowledge, this paper is the first work proposing an end-to-end deep dynamic point cloud compression framework. The experimental result shows that the proposed D-DPCC framework achieves an average 76% BD-Rate (Bjontegaard Delta Rate) gains against state-of-the-art Video-based Point Cloud Compression (V-PCC) v13 in inter mode.","doi":"10.24963/ijcai.2022/126","url":"https://ijcai.org/proceedings/2022/126","pdf":"https://ijcai.org/proceedings/2022/0126.pdf","markdown":"https://ijcai.org/proceedings/2022/126.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/126","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":126,"pages":{"first":898,"last":904},"keywords":["Computer Vision: 3D Computer Vision","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986873","length_sec":70,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985016","length_sec":335,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ballé et al., 2017] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compres- sion. In 5th International Conference on Learning Repre- sentations, ICLR 2017, 2017.","title":"End-to-end optimized image compression","authors":"Ballé","year":"2017","venue":"5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choy et al., 2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.","title":"4d spatio-temporal convnets: Minkowski convolutional neural networks","authors":"Christopher Choy, Junyoung Gwak, Silvio Savarese","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Queiroz and Chou, 2017] Ricardo L de Queiroz and Philip A Chou. Motion-compensated compression of dy- namic voxelized point clouds. IEEE Transactions on Im- age Processing, 26(8):3886-3895, 2017.","title":"Motion-compensated compression of dynamic voxelized point clouds","authors":"Ricardo L De Queiroz, Philip A Chou","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"['Eon et al., 2017] Eugene d'Eon, Bob Harrison, Taos Myers, and Philip A Chou. 8i voxelized full bodies-a voxelized point cloud dataset. ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) input doc- ument WG11M40059/WG1M74006, 7:8, 2017.","title":"8i voxelized full bodies-a voxelized point cloud dataset","authors":"Eon","year":"2017","venue":"ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) input document WG","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2021] Linyao Gao, Tingyu Fan, Jianqiang Wang, Yiling Xu, Jun Sun, and Zhan Ma. Point cloud geometry compression via neural graph sampling. In 2021 IEEE In- ternational Conference on Image Processing (ICIP), pages 3373-3377. IEEE, 2021.","title":"Point cloud geometry compression via neural graph sampling","authors":"Gao","year":"2021","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2021] Zhihao Hu, Guo Lu, and Dong Xu. Fvc: A new framework towards deep video compression in fea- ture space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1502- 1511, 2021.","title":"Fvc: A new framework towards deep video compression in feature space","authors":"Zhihao Hu, Guo Lu, Dong Xu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang and Liu, 2019] Tianxin Huang and Yong Liu. 3d point cloud geometry compression on deep learning. In Proceedings of the 27th ACM International Conference on Multimedia, pages 890-898, 2019.","title":"3d point cloud geometry compression on deep learning","authors":"Liu ; ] Tianxin Huang, Yong Huang, Liu","year":"2019","venue":"Proceedings of the 27th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Keming et al., 2018] Cao Keming, Xu Yi, Lu Yao, and Wen Ziyu. Owlii dynamic human mesh sequence dataset. Doc- ument ISO/IEC JTC1/SC29/WG11 m42816, San Diego, 2018.","title":"Owlii dynamic human mesh sequence dataset","authors":"Cao Keming, Xu Yi, Lu Yao, Wen Ziyu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations, ICLR 2015, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Zhu Li, Shan Liu, Frederic Dufaux, Li Li, Ge Li, and C.-C. Jay Kuo. Guest editorial introduction to the special issue on recent advances in point cloud pro- cessing and compression. IEEE Transactions on Circuits and Systems for Video Technology, 31(12), 2021.","title":"Guest editorial introduction to the special issue on recent advances in point cloud processing and compression","authors":"Zhu Li, Shan Liu, Frederic Dufaux, Li Li, Ge Li, C.-C. Jay Kuo","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Xingyu Liu, Charles R Qi, and Leonidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 529-537, 2019.","title":"Flownet3d: Learning scene flow in 3d point clouds","authors":"Xingyu Liu, Charles R Qi, Leonidas J Guibas","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2019] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dvc: An end-to- end deep video compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11006-11015, 2019.","title":"Dvc: An end-toend deep video compression framework","authors":"Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nguyen et al., 2021] Dat Thanh Nguyen, Maurice Quach, Giuseppe Valenzise, and Pierre Duhamel. Learning-based lossless compression of 3d point cloud geometry. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4220-4224. IEEE, 2021.","title":"Learning-based lossless compression of 3d point cloud geometry","authors":"Nguyen","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2017] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: deep hierarchical feature learning on point sets in a metric space. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pages 5105-5114, 2017.","title":"Pointnet++: deep hierarchical feature learning on point sets in a metric space","authors":"Li Charles R Qi, Hao Yi, Leonidas J Su, Guibas","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schwarz et al., 2018] Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A Chou, Robert A Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li, et al. Emerging mpeg stan- dards for point cloud compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(1):133-148, 2018.","title":"Emerging mpeg standards for point cloud compression","authors":"Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A Chou, Robert A Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li","year":"2018","venue":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial in- telligence, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"Thirty-first AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thanou et al., 2015] Dorina Thanou, Philip A Chou, and Pascal Frossard. Graph-based motion estimation and com- pensation for dynamic 3d point cloud compression. In 2015 IEEE International Conference on Image Processing (ICIP), pages 3235-3239. IEEE, 2015.","title":"Graph-based motion estimation and compensation for dynamic 3d point cloud compression","authors":"Dorina Thanou, Philip A Chou, Pascal Frossard","year":"2015","venue":"2015 IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021a] Guangming Wang, Yunzhe Hu, Xinrui Wu, and Hesheng Wang. Residual 3d scene flow learn- ing with context-aware feature extraction. arXiv preprint arXiv:2109.04685, 2021.","title":"Residual 3d scene flow learning with context-aware feature extraction","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021b] Guangming Wang, Xinrui Wu, Zhe Liu, and Hesheng Wang. Hierarchical attention learning of scene flow in 3d point clouds. IEEE Transactions on Image Processing, 30:5168-5181, 2021.","title":"Hierarchical attention learning of scene flow in 3d point clouds","authors":"Wang","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021c] Jianqiang Wang, Dandan Ding, Zhu Li, and Zhan Ma. Multiscale point cloud geometry compres- sion. In 2021 Data Compression Conference (DCC), pages 73-82. IEEE, 2021.","title":"Multiscale point cloud geometry compression","authors":"Wang","year":"2021","venue":"2021 Data Compression Conference (DCC)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021d] Jianqiang Wang, Hao Zhu, Haojie Liu, and Zhan Ma. Lossy point cloud geometry compression via end-to-end learning. IEEE Transactions on Circuits and Systems for Video Technology, 2021.","title":"Lossy point cloud geometry compression via end-to-end learning","authors":"Wang","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Yiling Xu, Ke Zhang, Lanyi He, Zhiqian Jiang, and Wenjie Zhu. Introduction to point cloud com- pression. ZTE Communications, 16(3):8, 2018.","title":"Introduction to point cloud compression","authors":"Yiling Xu, Ke Zhang, Lanyi He, Zhiqian Jiang, Wenjie Zhu","year":"2018","venue":"ZTE Communications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu, and Qigong Sun. Attan: Attention adver- sarial networks for 3d point cloud semantic segmentation. In Proceedings of the Twenty-Ninth International Confer- ence on International Joint Conferences on Artificial In- telligence, pages 789-796, 2021.","title":"Attan: Attention adversarial networks for 3d point cloud semantic segmentation","authors":"Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu, Qigong Sun","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/110"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}