{"type":"ScholarlyArticle","title":"AggPose: Deep Aggregation Vision Transformer for Infant Pose Estimation","authors":[{"name":"Xu Cao","slug":"xu-cao","given":"Xu","family":"Cao","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; New York University"},{"name":"Xiaoye Li","slug":"xiaoye-li","given":"Xiaoye","family":"Li","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; Shenzhen Baoan Women's and Childiren's Hospital , Jinan University"},{"name":"Liya Ma","slug":"liya-ma","given":"Liya","family":"Ma","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; Shenzhen Baoan Women's and Childiren's Hospital , Jinan University"},{"name":"Yi Huang","slug":"yi-huang","given":"Yi","family":"Huang","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; Shenzhen Children's Hospital"},{"name":"Xuan Feng","slug":"xuan-feng","given":"Xuan","family":"Feng","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; Shenzhen Children's Hospital"},{"name":"Zening Chen","slug":"zening-chen","given":"Zening","family":"Chen","affiliation":"New York University"},{"name":"Hongwu Zeng","slug":"hongwu-zeng","given":"Hongwu","family":"Zeng","affiliation":"Shenzhen Children's Hospital"},{"name":"Jianguo Cao","slug":"jianguo-cao","given":"Jianguo","family":"Cao","affiliation":"Shenzhen Automatic Rehabilitation Laboratory; Shenzhen Children's Hospital"}],"abstract":"Movement and pose assessment of newborns lets experienced pediatricians predict neurodevelopmental disorders, allowing early intervention for related diseases. However, most of the newest AI approaches for human pose estimation methods focus on adults, lacking publicly benchmark for infant pose estimation. In this paper, we fill this gap by proposing infant pose dataset and Deep Aggregation Vision Transformer for human pose estimation, which introduces a fast trained full transformer framework without using convolution operations to extract features in the early stages. It generalizes Transformer + MLP to high-resolution deep layer aggregation within feature maps, thus enabling information fusion between different vision levels. We pre-train AggPose on COCO pose dataset and apply it on our newly released large-scale infant pose estimation dataset. The results show that AggPose could effectively learn the multi-scale features among different resolutions and significantly improve the performance of infant pose estimation. We show that AggPose outperforms hybrid model HRFormer and TokenPose in the infant pose estimation dataset. Moreover, our AggPose outperforms HRFormer by 0.8 AP on COCO val pose estimation on average.  Our code is available at github.com/SZAR-LAB/AggPose.","doi":"10.24963/ijcai.2022/700","url":"https://ijcai.org/proceedings/2022/700","pdf":"https://ijcai.org/proceedings/2022/0700.pdf","markdown":"https://ijcai.org/proceedings/2022/700.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/700","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI for Good","number":700,"pages":{"first":5045,"last":5051},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Computer Vision: Applications","Computer Vision: Biomedical Image Analysis","Multidisciplinary Topics and Applications: Health and Medicine"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987126","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985269","length_sec":573,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abbruzzese et al., 2020] Laurel Daniels Abbruzzese, Natasha Yamane, Deborah Fein, Letitia Naigles, and Sylvie Goldman. Assessing child postural variability: Development, feasibility, and reliability of a video coding system. Physical & Occupational Therapy In Pediatrics, 41(3):314-325, 2020.","title":"Assessing child postural variability: Development, feasibility, and reliability of a video coding system","authors":"Abbruzzese","year":"2020","venue":"Physical & Occupational Therapy In Pediatrics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao and Lin, 2021] Xu Cao and Yanghao Lin. Caggnet: Crossing aggregation network for medical image segmen- tation. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 1744-1750. IEEE, 2021.","title":"Caggnet: Crossing aggregation network for medical image segmentation","authors":"Lin ; ] Cao, Cao Xu, Yanghao Lin","year":"2020","venue":"25th International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2019] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intel- ligence, 43(1):172-186, 2019.","title":"Openpose: realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of CVPR, pages 5386-5395, 2020.","title":"Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation","authors":"Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, Lei Zhang","year":"2020","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2021] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Multi-scale high-resolution vision transformer for semantic segmentation. arXiv preprint arXiv:2111.01236, 2021.","title":"Multi-scale high-resolution vision transformer for semantic segmentation","authors":"Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, David Z Pan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hesse et al., 2018] Nikolas Hesse, Christoph Bodensteiner, Michael Arens, Ulrich G Hofmann, Raphael Weinberger, and A Sebastian Schroeder. Computer vision for medical infant motion analysis: State of the art and rgb-d data set. In Proceedings of ECCV Workshops, pages 0-0, 2018.","title":"Computer vision for medical infant motion analysis: State of the art and rgb-d data set","authors":"Nikolas Hesse, Christoph Bodensteiner, Michael Arens, Raphael Ulrich G Hofmann, Sebastian Weinberger, Schroeder","year":"2018","venue":"Proceedings of ECCV Workshops","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2020] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil is in the details: Delving into unbiased data processing for human pose estimation. In Proceedings of CVPR, pages 5700-5709, 2020.","title":"The devil is in the details: Delving into unbiased data processing for human pose estimation","authors":"Junjie Huang, Zheng Zhu, Feng Guo, Guan Huang","year":"2020","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2021] Xiaofei Huang, Nihang Fu, Shuangjun Liu, and Sarah Ostadabbas. Invariant representation learn- ing for infant pose estimation with small data. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1-8. IEEE, 2021.","title":"Invariant representation learning for infant pose estimation with small data","authors":"Xiaofei Huang, Nihang Fu, Shuangjun Liu, Sarah Ostadabbas","year":"2021","venue":"16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose es- timation. In Proceedings of ICCV, pages 11313-11322, 2021.","title":"Tokenpose: Learning keypoint tokens for human pose estimation","authors":"Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, Erjin Zhou","year":"2021","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Lin","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McCay et al., 2019] Kevin D McCay, Edmond SL Ho, Claire Marcroft, and Nicholas D Embleton. Establishing pose based features using histograms for the detection of abnormal infant movements. In EMBC, pages 5469-5472. IEEE, 2019.","title":"Establishing pose based features using histograms for the detection of abnormal infant movements","authors":"Kevin D Mccay, Edmond Sl Ho, Claire Marcroft, Nicholas D Embleton","year":"2019","venue":"EMBC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Reich et al., 2021] Simon Reich, Dajie Zhang, Tomas Kul- vicius, Sven Bölte, Karin Nielsen-Saines, Florian B Poko- rny, Robert Peharz, Luise Poustka, Florentin Wörgötter, Christa Einspieler, et al. Novel ai driven approach to clas- sify infant motor functions. Scientific Reports, 11(1):1-13, 2021.","title":"Novel ai driven approach to classify infant motor functions","authors":"Simon Reich, Dajie Zhang, Tomas Kulvicius, Sven Bölte, Karin Nielsen-Saines, Florian B Pokorny, Robert Peharz, Luise Poustka, Florentin Wörgötter, Christa Einspieler","year":"2021","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Silva et al., 2021] Nelson Silva, Dajie Zhang, Tomas Kulvi- cius, Alexander Gail, Carla Barreiros, Stefanie Lindstaedt, Marc Kraft, Sven Bölte, Luise Poustka, Karin Nielsen- Saines, et al. The future of general movement assess- ment: The role of computer vision and machine learning- a scoping review. Research in developmental disabilities, 110:103854, 2021.","title":"The future of general movement assessment: The role of computer vision and machine learninga scoping review","authors":"Nelson Silva, Dajie Zhang, Tomas Kulvicius, Alexander Gail, Carla Barreiros, Stefanie Lindstaedt, Marc Kraft, Sven Bölte, Luise Poustka, Karin Nielsen-Saines","year":"2021","venue":"Research in developmental disabilities","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2020] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recogni- tion. IEEE transactions on pattern analysis and machine intelligence, 2020.","title":"Deep high-resolution representation learning for visual recognition","authors":"Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang","year":"2020","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xie et al., 2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for seman- tic segmentation with transformers. arXiv preprint arXiv:2105.15203, 2021.","title":"Segformer: Simple and efficient design for semantic segmentation with transformers","authors":"Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2021] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via trans- former. In Proceedings of ICCV, pages 11802-11812, 2021.","title":"Transpose: Keypoint localization via transformer","authors":"Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang","year":"2021","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yuan et al., 2021] Yuhui Yuan, Rao Fu, Lang Huang, Wei- hong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense predic- tion. arXiv preprint arXiv:2110.09408, 2021.","title":"Hrformer: High-resolution transformer for dense prediction","authors":"Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, Jingdong Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2020] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-aware coordinate rep- resentation for human pose estimation. In Proceedings of CVPR, pages 7093-7102, 2020.","title":"Distribution-aware coordinate representation for human pose estimation","authors":"Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, Ce Zhu","year":"2020","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}