{"type":"ScholarlyArticle","title":"Multi-Stream Deep Similarity Learning Networks for Visual Tracking","authors":[{"name":"Kunpeng Li","slug":"kunpeng-li","given":"Kunpeng","family":"Li","affiliation":"Department of Electrical and Computer Engineering , Northeastern University , Boston , MA 02115 , USA"},{"name":"Yu Kong","slug":"yu-kong","given":"Yu","family":"Kong","affiliation":"Department of Electrical and Computer Engineering , Northeastern University , Boston , MA 02115 , USA"},{"name":"Yun Fu","slug":"yun-fu","given":"Yun","family":"Fu","affiliation":"Department of Electrical and Computer Engineering , Northeastern University , Boston , MA 02115 , USA; College of Computer and Information Science , Northeastern University , Boston , MA 02115 , USA"}],"abstract":"Visual tracking has achieved remarkable success in recent decades, but it remains a challenging problem due to appearance variations over time and complex cluttered background. In this paper, we adopt a tracking-by-verification scheme to overcome these challenges by determining the patch in the subsequent frame that is most similar to the target template and distinctive to the background context. A multi-stream deep similarity learning network is proposed to learn the similarity comparison model. The loss function of our network encourages the distance between a positive patch in the search region and the target template to be smaller than that between positive patch and the background patches. Within the learned feature space, even if the distance between positive patches becomes large caused by the appearance change or interference of background clutter, our method can use the relative distance to distinguish the target robustly. Besides, the learned model is directly used for tracking with no need of model updating, parameter fine-tuning and can run at 45 fps on a single GPU. Our tracker achieves state-of-the-art performance on the visual tracking benchmark compared with other recent real-time-speed trackers, and shows better capability in handling background clutter, occlusion and appearance change.","doi":"10.24963/ijcai.2017/301","url":"https://ijcai.org/proceedings/2017/301","pdf":"https://ijcai.org/proceedings/2017/0301.pdf","markdown":"https://ijcai.org/proceedings/2017/301.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/301","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":301,"pages":{"first":2166,"last":2172},"keywords":["Robotics and Vision: Robotics and Vision","Machine Learning: Deep Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bertinetto et al., 2016] Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional siamese networks for object tracking. In ECCV, pages 850-865, 2016.","title":"Fully-convolutional siamese networks for object tracking","authors":"Luca Bertinetto, Jack Valmadre, F João, Andrea Henriques, Philip Hs Vedaldi, Torr","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bromley et al., 1993] Jane Bromley, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a \"siamese\" time delay neural network. IJPRAI, 7(4):669-688, 1993.","title":"Signature verification using a \"siamese\" time delay neural network","authors":"Jane Bromley, Yann Lecun, Cliff Moore, Eduard Säckinger, Roopak Shah","year":"1993","venue":"IJPRAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chechik et al., 2009] Gal Chechik, Uri Shalit, Varun Sharma, and Samy Bengio. An online algorithm for large scale image similarity learning. In NIPS, pages 306-314, 2009.","title":"An online algorithm for large scale image similarity learning","authors":"Chechik","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Danelljan et al., 2016] Martin Danelljan, Gustav Hager, and Michael Felsberg. Discriminative scale space tracking. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2016.","title":"Discriminative scale space tracking","authors":"Martin Danelljan, Gustav Hager, Michael Felsberg","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan and Ling, 2016] Heng Fan and Haibin Ling. Sanet: Structure-aware network for visual tracking. arXiv:1611.06878, 2016.","title":"Sanet: Structure-aware network for visual tracking","authors":"Heng Fan, Haibin Ling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gan et al., 2015] Quan Gan, Qipeng Guo, Zheng Zhang, and Kyunghyun Cho. First step toward model-free, anony- mous object tracking with recurrent neural networks. arXiv:1511.06425, 2015.","title":"First step toward model-free, anonymous object tracking with recurrent neural networks","authors":"Quan Gan, Qipeng Guo, Zheng Zhang, Kyunghyun Cho","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hare et al., 2016] Sam Hare, Golodetz, Stephen L Hicks, and Philip HS Torr. Struck: Structured output tracking with kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):2096-2109, 2016.","title":"Struck: Structured output tracking with kernels","authors":"Sam Hare, Stephen L Golodetz, Philip Hs Hicks, Torr","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Held et al., 2016] David Held, Sebastian Thrun, and Silvio Savarese. Learning to track at 100 fps with deep regression networks. In ECCV, 2016.","title":"Learning to track at 100 fps with deep regression networks","authors":"David Held, Sebastian Thrun, Silvio Savarese","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Henriques et al., 2015] João F Henriques, Pedro Martins, and Jorge Batista. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583-596, 2015.","title":"High-speed tracking with kernelized correlation filters","authors":"Pedro João F Henriques, Jorge Martins, Batista","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2015] Zhibin Hong, Zhe Chen, Chaohui Wang, Xue Mei, Danil Prokhorov, and Dacheng Tao. Multi- store tracker (muster): A cognitive psychology inspired approach to object tracking. In CVPR, pages 749-758, 2015.","title":"Multistore tracker (muster): A cognitive psychology inspired approach to object tracking","authors":"Zhibin Hong, Zhe Chen, Chaohui Wang, Xue Mei, Danil Prokhorov, Dacheng Tao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kalal et al., 2012] Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas. Tracking-learning-detection. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 34(7):1409-1422, 2012.","title":"Tracking-learning-detection","authors":"Zdenek Kalal, Krystian Mikolajczyk, Jiri Matas","year":"2012","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kristan et al., 2016] Matej Kristan, Gustavo Fernndez, et al. The visual object tracking challenge results. 2016.","title":"The visual object tracking challenge results","authors":"Matej Kristan, Gustavo Fernndez","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Zhang, 2014] Kunpeng Li and Xin Zhang. A new fingertip detection and tracking algorithm and its applica- tion on writing-in-the-air system. In CISP, pages 457-462, 2014.","title":"A new fingertip detection and tracking algorithm and its application on writing-in-the-air system","authors":"Li, Zhang, Kunpeng Li, Xin Zhang","year":"2014","venue":"CISP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2015] Chao Ma, Xiaokang Yang, Chongyang Zhang, and Ming-Hsuan Yang. Long-term correlation tracking. In CVPR, pages 5388-5396, 2015.","title":"Long-term correlation tracking","authors":"Chao Ma, Xiaokang Yang, Chongyang Zhang, Ming-Hsuan Yang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mei and Ling, 2009] Xue Mei and Haibin Ling. Robust vi- sual tracking using l1 minimization. In ICCV, pages 1436- 1443, 2009.","title":"Robust visual tracking using l1 minimization","authors":"Xue Mei, Haibin Ling","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nam and Han, 2016] Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, pages 4293-4302, 2016.","title":"Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking","authors":"Nam, Han","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ning and Yang, 2016] Jifeng Ning and Ming-Hsuan Yang. Object tracking via dual linear structured svm and explicit feature map. In CVPR, pages 4266-4274, 2016.","title":"Object tracking via dual linear structured svm and explicit feature map","authors":"Yang ; ] Jifeng Ning, Ming-Hsuan Ning, Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qi et al., 2016] Yuankai Qi, Shengping Zhang, Lei Qin, Hongxun Yao, Qingming Huang, and Jongwoo Lim Ming- Hsuan Yang. Hedged deep tracking. In CVPR, 2016.","title":"Hedged deep tracking","authors":"Yuankai Qi, Shengping Zhang, Lei Qin, Hongxun Yao, Qingming Huang, Jongwoo Lim, Ming-Hsuan Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ross et al., 2008] David A Ross, Jongwoo Lim, Ruei-Sung Lin, and Ming-Hsuan Yang. Incremental learning for ro- bust visual tracking. IJCV, 77(1-3):125-141, 2008.","title":"Incremental learning for robust visual tracking","authors":"Jongwoo David A Ross, Ruei-Sung Lim, Ming-Hsuan Lin, Yang","year":"2008","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smeulders and Shah, 2014] Arnold WM Smeulders and Mubarak Shah. Visual tracking: An experimental survey. IEEE Transactions on Pattern Analysis and Machine In- telligence, 36(7):1442-1468, 2014.","title":"Visual tracking: An experimental survey","authors":"Shah ; Smeulders, W M Arnold, Mubarak Smeulders, Shah","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Taigman et al., 2014] Yaniv Taigman, Ming Yang, and Lior Wolf. Deepface: Closing the gap to human-level perfor- mance in face verification. In CVPR, pages 1701-1708, 2014.","title":"Deepface: Closing the gap to human-level performance in face verification","authors":"Yaniv Taigman, Ming Yang, Lior Wolf","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tao et al., 2016] Ran Tao, Efstratios Gavves, and Arnold WM Smeulders. Siamese instance search for tracking. In CVPR, pages 1420-1429, 2016.","title":"Siamese instance search for tracking","authors":"Ran Tao, Efstratios Gavves, Arnold Wm Smeulders","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2013] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. In CVPR, pages 2411-2418, 2013.","title":"Online object tracking: A benchmark","authors":"Yi Wu, Jongwoo Lim, Ming-Hsuan Yang","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2015] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1834- 1848, 2015.","title":"Object tracking benchmark","authors":"Yi Wu, Jongwoo Lim, Ming-Hsuan Yang","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zagoruyko and Komodakis, 2015] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In CVPR, pages 4353- 4361, 2015.","title":"Learning to compare image patches via convolutional neural networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818-833, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2014] Jianming Zhang, Shugao Ma, and Stan Sclaroff. Meem: robust tracking via multiple experts using entropy minimization. In ECCV, pages 188-203, 2014.","title":"Meem: robust tracking via multiple experts using entropy minimization","authors":"Jianming Zhang, Shugao Ma, Stan Sclaroff","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2015] Guibo Zhu, Jinqiao Wang, Yi Wu, and Hanqing Lu. Collaborative correlation tracking. In BMVC, pages 184-1, 2015.","title":"Collaborative correlation tracking","authors":"Zhu","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}