{"type":"ScholarlyArticle","title":"Recurrent Existence Determination Through Policy Optimization","authors":[{"name":"Baoxiang Wang","slug":"baoxiang-wang","given":"Baoxiang","family":"Wang","affiliation":"The Chinese University of Hong Kong"}],"abstract":"Binary determination of the presence of objects is one of the problems where humans perform extraordinarily better than computer vision systems, in terms of both speed and preciseness. One of the possible reasons is that humans can skip most of the clutter and attend only on salient regions. Recurrent attention models (RAM) are the first computational models to imitate the way humans process images via the REINFORCE algorithm. Despite that RAM is originally designed for image recognition, we extend it and present recurrent existence determination, an attention-based mechanism to solve the existence determination. Our algorithm employs a novel $k$-maximum aggregation layer and a new reward mechanism to address the issue of delayed rewards, which would have caused the instability of the training process. The experimental analysis demonstrates significant efficiency and accuracy improvement over existing approaches, on both synthetic and real-world datasets.","doi":"10.24963/ijcai.2019/507","url":"https://ijcai.org/proceedings/2019/507","pdf":"https://ijcai.org/proceedings/2019/0507.pdf","markdown":"https://ijcai.org/proceedings/2019/507.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/507","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":507,"pages":{"first":3656,"last":3662},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: POMDPs","Machine Learning Applications: Bio;Medicine","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2014] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual at- tention. arXiv preprint arXiv:1412.7755, 2014.","title":"Multiple object recognition with visual attention","authors":"Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2015] Jimmy Ba, Ruslan R Salakhutdinov, Roger B Grosse, and Brendan J Frey. Learning wake- sleep recurrent attention models. In Advances in Neural Information Processing Systems, pages 2593-2601, 2015.","title":"Learning wakesleep recurrent attention models","authors":"Jimmy Ba, R Ruslan, Roger B Salakhutdinov, Brendan J Grosse, Frey","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borji et al., 2014] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A survey. arXiv preprint arXiv:1411.5878, 2014.","title":"Salient object detection: A survey","authors":"Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Das et al., 2016] Abhishek Das, Harsh Agrawal, C Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do hu- mans and deep networks look at the same regions? arXiv preprint arXiv:1606.03556, 2016.","title":"Human attention in visual question answering: Do humans and deep networks look at the same regions","authors":"Abhishek Das, Harsh Agrawal, Lawrence Zitnick, Devi Parikh, Dhruv Batra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dieleman et al., 2015] Sander Dieleman, Kyle W Willett, and Joni Dambre. Rotation-invariant convolutional neu- ral networks for galaxy morphology prediction. arXiv preprint arXiv:1503.07077, 2015.","title":"Rotation-invariant convolutional neural networks for galaxy morphology prediction","authors":"Kyle W Sander Dieleman, Joni Willett, Dambre","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eslami et al., 2016] SM Ali Eslami, Nicolas Heess, Theo- phane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understand- ing with generative models. In Advances in Neural Infor- mation Processing Systems, pages 3225-3233, 2016.","title":"Attend, infer, repeat: Fast scene understanding with generative models","authors":"Eslami","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fong et al., 2004] Donald S Fong, Lloyd Aiello, Thomas W Gardner, George L King, George Blankenship, Jerry D Cavallerano, Fredrick L Ferris, and Ronald Klein. Retinopathy in diabetes. Diabetes Care, 27(suppl 1):s84- s87, 2004.","title":"Retinopathy in diabetes","authors":"Lloyd Donald S Fong, Thomas W Aiello, George L Gardner, George King, Jerry D Blankenship, Fredrick L Cavallerano, Ronald Ferris, Klein","year":"2004","venue":"Diabetes Care","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Graham, 2014] Benjamin Graham. Fractional max-pooling. arXiv preprint arXiv:1412.6071, 2014.","title":"Fractional max-pooling","authors":"Benjamin Graham","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gregor et al., 2015] Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. Draw: A recurrent neu- ral network for image generation. arXiv preprint arXiv:1502.04623, 2015.","title":"Draw: A recurrent neural network for image generation","authors":"Karol Gregor, Ivo Danihelka, Alex Graves, Daan Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hermann et al., 2015] Karl Moritz Hermann, Tomas Ko- cisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems, pages 1693-1701, 2015.","title":"Teaching machines to read and comprehend","authors":"Hermann","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Wang, 2018] Jiajin Li and Baoxiang Wang. Pol- icy optimization with second-order advantage information. arXiv preprint arXiv:1805.03586, 2018.","title":"Policy optimization with second-order advantage information","authors":"Wang ; Li, Jiajin Li, Baoxiang Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/699"},{"position":14,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading ban- dits. In International Conference on Machine Learning, pages 1245-1253, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Ad- vances in Neural Information Processing Systems, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational Conference on Machine Learning, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2015] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Advances in Neural Infor- mation Processing Systems, pages 802-810, 2015.","title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","authors":"Shi","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In Advances in Neural Information Processing Systems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton, 1984] Richard Stuart Sutton. Temporal credit as- signment in reinforcement learning. 1984.","title":"Temporal credit assignment in reinforcement learning","authors":"Richard Stuart, Sutton","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wierstra et al., 2007] Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep mem- ory pomdps with recurrent policy gradients. In Artificial Neural Networks-ICANN 2007, pages 697-706. 2007.","title":"Solving deep memory pomdps with recurrent policy gradients","authors":"Daan Wierstra, Alexander Foerster, Jan Peters, Juergen Schmidhuber","year":"2007","venue":"Artificial Neural Networks-ICANN 2007","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. In Reinforcement Learning, pages 5-32. 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdi- nov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In International Conference on Machine Learn- ing, pages 77-81, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yeung et al., 2016] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detec- tion from frame glimpses in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 2678-2687, 2016.","title":"End-to-end learning of action detection from frame glimpses in videos","authors":"Serena Yeung, Olga Russakovsky, Greg Mori, Li Fei-Fei","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Young et al., 2018] Kenny Young, Baoxiang Wang, and Matthew E Taylor. Metatrace actor-critic: Online step-size tuning by meta-gradient descent for reinforcement learn- ing control. arXiv preprint arXiv:1805.04514, 2018.","title":"Metatrace actor-critic: Online step-size tuning by meta-gradient descent for reinforcement learning control","authors":"Kenny Young, Baoxiang Wang, Matthew E Taylor","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/581"},{"position":26,"raw":"[Zagoruyko et al., 2016] Sergey Zagoruyko, Adam Lerer, Tsung-Yi Lin, Pedro O Pinheiro, Sam Gross, Soumith Chintala, and Piotr Dollár. A multipath network for ob- ject detection. arXiv preprint arXiv:1604.02135, 2016.","title":"A multipath network for object detection","authors":"Sergey Zagoruyko, Adam Lerer, Tsung-Yi Lin, Pedro O Pinheiro, Sam Gross, Soumith Chintala, Piotr Dollár","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}