{"type":"ScholarlyArticle","title":"Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams","authors":[{"name":"Matteo Tiezzi","slug":"matteo-tiezzi","given":"Matteo","family":"Tiezzi","affiliation":"DIISM , University of Siena (Italy)"},{"name":"Simone Marullo","slug":"simone-marullo","given":"Simone","family":"Marullo","affiliation":"DIISM , University of Siena (Italy); DINFO , University of Florence (Italy)"},{"name":"Lapo Faggi","slug":"lapo-faggi","given":"Lapo","family":"Faggi","affiliation":"DIISM , University of Siena (Italy); DINFO , University of Florence (Italy)"},{"name":"Enrico Meloni","slug":"enrico-meloni","given":"Enrico","family":"Meloni","affiliation":"DIISM , University of Siena (Italy); DINFO , University of Florence (Italy)"},{"name":"Alessandro Betti","slug":"alessandro-betti","given":"Alessandro","family":"Betti","affiliation":"Inria , Lab I3S , MAASAI , Universitè Côte d'Azur (France"},{"name":"Stefano Melacci","slug":"stefano-melacci","given":"Stefano","family":"Melacci","affiliation":"DIISM , University of Siena (Italy)"}],"abstract":"Devising intelligent agents able to live in an environment and learn by observing the surroundings is a longstanding goal of Artificial Intelligence. From a bare Machine Learning perspective, challenges arise when the agent is prevented from leveraging large fully-annotated dataset, but rather the interactions with supervisory signals are sparsely distributed over space and time.\r\nThis paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Spatio-temporal stochastic coherence along the attention trajectory, paired with a contrastive term, leads to an unsupervised learning criterion that naturally copes with the considered setting. Differently from most existing works, the learned representations are used in open-set class-incremental classification of each frame pixel, relying on few supervisions. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream. Inheriting features from state-of-the art models is not as powerful as one might expect.","doi":"10.24963/ijcai.2022/483","url":"https://ijcai.org/proceedings/2022/483","pdf":"https://ijcai.org/proceedings/2022/0483.pdf","markdown":"https://ijcai.org/proceedings/2022/483.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/483","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":483,"pages":{"first":3480,"last":3486},"keywords":["Machine Learning: Online Learning","Machine Learning: Unsupervised Learning","Computer Vision: Machine Learning for Vision","Machine Learning: Weakly Supervised Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986571","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984714","length_sec":346,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aljundi et al., 2019] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proc. of the IEEE/CVF CVPR, pages 11254-11263, 2019.","title":"Task-free continual learning","authors":"Rahaf Aljundi, Klaas Kelchtermans, Tinne Tuytelaars","year":"2019","venue":"Proc. of the IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Betti et al., 2020] Alessandro Betti, Marco Gori, and Ste- fano Melacci. Learning visual features under motion in- variance. Neural Networks, 126:275-299, 2020.","title":"Learning visual features under motion invariance","authors":"Alessandro Betti, Marco Gori, Stefano Melacci","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borji and Itti, 2012] Ali Borji and Laurent Itti. State-of-the- art in visual attention modeling. IEEE TPAMI, 35(1):185- 207, 2012.","title":"Ali Borji and Laurent Itti. State-of-theart in visual attention modeling","authors":"Itti Borji","year":"2012","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Borji, 2019] Ali Borji. Saliency prediction in the deep learn- ing era: Successes and limitations. IEEE TPAMI, 2019.","title":"Saliency prediction in the deep learning era: Successes and limitations","authors":"Ali Borji","year":"2019","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burt et al., 2021] Ryan Burt, Nina N. Thigpen, Andreas Keil, and Jose C. Principe. Unsupervised foveal vision neural architecture with top-down attention. Neural Net- works, 141:145-159, 2021.","title":"Unsupervised foveal vision neural architecture with top-down attention","authors":"Ryan Burt, Nina N Thigpen, Andreas Keil, Jose C Principe","year":"2021","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaudhari et al., 2019] Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. An attentive sur- vey of attention models. arXiv preprint arXiv:1904.02874, 2019.","title":"An attentive survey of attention models","authors":"Sneha Chaudhari, Varun Mithal, Rohan Gungor Polatkan, Ramanath","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 40(4):834-848, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gan et al., 2020] Chuang Gan, Jeremy Schwartz, Seth Al- ter, Martin Schrimpf, et al. Threedworld: A plat- form for interactive multi-modal physical simulation. arXiv:2007.04954, 2020.","title":"Threedworld: A platform for interactive multi-modal physical simulation","authors":"Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geng et al., 2020] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE TPAMI, 2020.","title":"Recent advances in open set recognition: A survey","authors":"Geng","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hoi et al., 2018] Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive sur- vey. arXiv preprint arXiv:1802.02871, 2018.","title":"Online learning: A comprehensive survey","authors":"C H Steven, Doyen Hoi, Jing Sahoo, Peilin Lu, Zhao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jing and Tian, 2020] Longlong Jing and Yingli Tian. Self- supervised visual feature learning with deep neural net- works: A survey. IEEE TPAMI, 2020.","title":"Selfsupervised visual feature learning with deep neural networks: A survey","authors":"Tian Jing, Longlong Jing, Yingli Tian","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kornblith et al., 2019] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer bet- ter? In Proc. of the IEEE/CVF CVPR, June 2019.","title":"Do better imagenet models transfer better?","authors":"Simon Kornblith, Jonathon Shlens, Quoc V Le","year":"2019","venue":"Proc. of the IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proc. of the IEEE/CVF CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proc. of the IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Meloni et al., 2020] Enrico Meloni, Luca Pasqualini, Mat- teo Tiezzi, Marco Gori, and Stefano Melacci. Sailenv: Learning in virtual visual environments made simple. In ICPR, pages 8906-8913, 2020.","title":"Sailenv: Learning in virtual visual environments made simple","authors":"Enrico Meloni, Luca Pasqualini, Matteo Tiezzi, Marco Gori, Stefano Melacci","year":"2020","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Meloni et al., 2021] Enrico Meloni, Alessandro Betti, Lapo Faggi, Simone Marullo, Matteo Tiezzi, and Stefano Melacci. Evaluating continual learning algorithms by generating 3d virtual environments. arXiv preprint arXiv:2109.07855, 2021.","title":"Evaluating continual learning algorithms by generating 3d virtual environments","authors":"Enrico Meloni, Alessandro Betti, Lapo Faggi, Simone Marullo, Matteo Tiezzi, Stefano Melacci","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Min et al., 2021] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. arXiv preprint arXiv:2104.01538, 2021.","title":"Hypercorrelation squeeze for few-shot segmentation","authors":"Min","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parisi et al., 2019] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.","title":"Continual lifelong learning with neural networks: A review","authors":"Parisi","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ranftl et al., 2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. CoRR, abs/2103.13413, 2021.","title":"Vision transformers for dense prediction","authors":"Ranftl","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russell and Norvig, 2009] Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall Press, USA, 3rd edition, 2009.","title":"Artificial Intelligence: A Modern Approach","authors":"Stuart Russell, Peter Norvig","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Scheirer et al., 2012] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE TPAMI, 35(7):1757-1772, 2012.","title":"Toward open set recognition","authors":"J Walter, Anderson Scheirer, De Rezende, Archana Rocha, Terrance E Sapkota, Boult","year":"2012","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sherrah, 2016] Jamie Sherrah. Fully convolutional net- works for dense semantic labelling of high-resolution aerial imagery. arXiv preprint arXiv:1606.02585, 2016.","title":"Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery","authors":"Jamie Sherrah","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tiezzi et al., 2020] Matteo Tiezzi, Stefano Melacci, Alessandro Betti, Marco Maggini, and Marco Gori. Focus of attention improves information transfer in visual features. In NeurIPS, volume 33, pages 22194-22204, 2020.","title":"Focus of attention improves information transfer in visual features","authors":"Matteo Tiezzi, Stefano Melacci, Alessandro Betti, Marco Maggini, Marco Gori","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zanca et al., 2020] Dario Zanca, Stefano Melacci, and Marco Gori. Gravitational laws of focus of attention. IEEE TPAMI, 42(12):2983-2995, 2020.","title":"Gravitational laws of focus of attention","authors":"Dario Zanca, Stefano Melacci, Marco Gori","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2019] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 127(3):302-321, 2019.","title":"Semantic understanding of scenes through the ade20k dataset","authors":"Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, Antonio Torralba","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}