{"type":"ScholarlyArticle","title":"Online Hybrid Lightweight Representations Learning: Its Application to Visual Tracking","authors":[{"name":"Ilchae Jung","slug":"ilchae-jung","given":"Ilchae","family":"Jung","affiliation":"ASRI , Seoul National University; CSE , POSTECH"},{"name":"Minji Kim","slug":"minji-kim","given":"Minji","family":"Kim","affiliation":"ASRI , Seoul National University; ECE & IPAI , Seoul National University"},{"name":"Eunhyeok Park","slug":"eunhyeok-park","given":"Eunhyeok","family":"Park","affiliation":"CSE , POSTECH"},{"name":"Bohyung Han","slug":"bohyung-han","given":"Bohyung","family":"Han","affiliation":"ASRI , Seoul National University; ECE & IPAI , Seoul National University"}],"abstract":"This paper presents a novel hybrid representation learning framework for streaming data, where an image frame in a video is modeled by an ensemble\r\nof two distinct deep neural networks; one is a low-bit quantized network and the other is a lightweight full-precision network. The former learns coarse\r\nprimary information with low cost while the latter conveys residual information for high fidelity to original representations. The proposed parallel architecture is effective to maintain complementary information since fixed-point arithmetic can be utilized in the quantized network and the lightweight\r\nmodel provides precise representations given by a compact channel-pruned network. We incorporate the hybrid representation technique into an online\r\nvisual tracking task, where deep neural networks need to handle temporal variations of target appearances in real-time. Compared to the state-of-the-art\r\nreal-time trackers based on conventional deep neural networks, our tracking algorithm demonstrates competitive accuracy on the standard benchmarks\r\nwith a small fraction of computational cost and memory footprint.","doi":"10.24963/ijcai.2022/140","url":"https://ijcai.org/proceedings/2022/140","pdf":"https://ijcai.org/proceedings/2022/0140.pdf","markdown":"https://ijcai.org/proceedings/2022/140.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/140","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":140,"pages":{"first":1002,"last":1008},"keywords":["Computer Vision: Representation Learning","Computer Vision: Applications","Computer Vision: Machine Learning for Vision","Computer Vision: Motion and Tracking"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986814","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984957","length_sec":361,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Baskin et al., 2018] Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M Bron- stein, and Avi Mendelson. Uniq: Uniform noise injection for non-uniform quantization of neural networks. arXiv preprint arXiv:1804.10969, 2018.","title":"Uniq: Uniform noise injection for non-uniform quantization of neural networks","authors":"Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M Bronstein, Avi Mendelson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bulat and Tzimiropoulos, 2019] Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural net- works. arXiv preprint arXiv:1909.13863, 2019.","title":"Xnor-net++: Improved binary neural networks","authors":"Adrian Bulat, Georgios Tzimiropoulos","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Che et al., 2018] Manqiang Che, Runling Wang, Yan Lu, Yan Li, Hui Zhi, and Changzhen Xiong. Channel prun- ing for visual tracking. In ECCVW, 2018.","title":"Channel pruning for visual tracking","authors":"Che","year":"2018","venue":"ECCVW","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR, 2021.","title":"Transformer tracking","authors":"Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, Huchuan Lu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Danelljan et al., 2019] Martin Danelljan, Goutam Bhat, Fa- had Shahbaz Khan, and Michael Felsberg. Atom: Accu- rate tracking by overlap maximization. In CVPR, 2019.","title":"Atom: Accurate tracking by overlap maximization","authors":"Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fan et al., 2019] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In CVPR, 2019.","title":"Lasot: A high-quality benchmark for large-scale single object tracking","authors":"Liting Heng Fan, Fan Lin, Peng Yang, Ge Chu, Sijia Deng, Hexin Yu, Yong Bai, Chunyuan Xu, Haibin Liao, Ling","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2018] Xitong Gao, Yiren Zhao, Łukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.","title":"Dynamic channel pruning: Feature boosting and suppression","authors":"Gao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In NeurIPS, 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jang et al., 2017] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jung et al., 2018] Ilchae Jung, Jeany Son, Mooyeol Baek, and Bohyung Han. Real-Time MDNet. In ECCV, 2018.","title":"Real-Time MDNet","authors":"Jung","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jung et al., 2019] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In CVPR, 2019.","title":"Learning to quantize deep networks by optimizing quantization intervals with task loss","authors":"Jung","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jung et al., 2020] Ilchae Jung, Kihyun You, Hyeonwoo Noh, Minsu Cho, and Bohyung Han. Real-time object tracking via meta-learning: Efficient model adaptation and one-shot channel pruning. AAAI, 2020.","title":"Real-time object tracking via meta-learning: Efficient model adaptation and one-shot channel pruning","authors":"Jung","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kang and Han, 2020] Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In ICML, 2020.","title":"Operation-aware soft channel pruning using differentiable masks","authors":"Minsoo Kang, Bohyung Han","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kristan et al., 2016] Matej Kristan, Aleš Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka Čehovin Zajc, Tomas Vojir, Gustav Häger, Alan Lukežič, and Gustavo Fernandez. The visual object tracking VOT2016 challenge results. In ECCVW, 2016.","title":"The visual object tracking VOT2016 challenge results","authors":"Matej Kristan, Aleš Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Čehovin Luka, Tomas Zajc, Gustav Vojir, Alan Häger, Gustavo Lukežič, Fernandez","year":"2016","venue":"ECCVW","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kristan et al., 2018] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pfugfelder, Luka Čehovin Zajc, Tomas Vojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, Gustavo Fernandez, and et al. The sixth visual object tracking vot2018 challenge results. In ECCVW, 2018.","title":"The sixth visual object tracking vot2018 challenge results","authors":"Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pfugfelder, Čehovin Luka, Tomas Zajc, Goutam Vojir, Alan Bhat, Abdelrahman Lukezic, Gustavo Eldesokey, Fernandez","year":"2018","venue":"ECCVW","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In CVPR, 2019.","title":"Siamrpn++: Evolution of siamese visual tracking with very deep networks","authors":"Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mueller et al., 2016] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for UAV tracking. In ECCV, 2016.","title":"A benchmark and simulator for UAV tracking","authors":"Matthias Mueller, Neil Smith, Bernard Ghanem","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nam and Han, 2016] Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, 2016.","title":"Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking","authors":"Nam, Han","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park and Yoo, 2020] Eunhyeok Park and Sungjoo Yoo. Profit: A novel training method for sub-4-bit mobilenet models. In ECCV, 2020.","title":"Eunhyeok Park and Sungjoo Yoo. Profit: A novel training method for sub-4-bit mobilenet models","authors":"Yoo Park","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qin et al., 2020] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for ac- curate binary neural networks. In CVPR, 2020.","title":"Forward and backward information retention for accurate binary neural networks","authors":"Qin","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2015] Y. Wu, J. Lim, and M. Yang. Object track- ing benchmark. TPAMI, 2015.","title":"Object tracking benchmark","authors":"Y Wu, J Lim, M Yang","year":"2015","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2020] Haibao Yu, Tuopu Wen, Guangliang Cheng, Jiankai Sun, Qi Han, and Jianping Shi. Low-bit quantization needs good distribution. In CVPRW, 2020.","title":"Low-bit quantization needs good distribution","authors":"Haibao Yu, Tuopu Wen, Guangliang Cheng, Jiankai Sun, Qi Han, Jianping Shi","year":"2020","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2018] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In ECCV, 2018.","title":"Distractor-aware siamese networks for visual object tracking","authors":"Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, Weiming Hu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhuang et al., 2020] Bohan Zhuang, Lingqiao Liu, Mingkui Tan, Chunhua Shen, and Ian Reid. Training quantized neu- ral networks with a full-precision auxiliary module. In CVPR, 2020.","title":"Training quantized neural networks with a full-precision auxiliary module","authors":"Bohan Zhuang, Lingqiao Liu, Mingkui Tan, Chunhua Shen, Ian Reid","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}