{"type":"ScholarlyArticle","title":"Learning to Recognize Transient Sound Events using Attentional Supervision","authors":[{"name":"Szu-Yu Chou","slug":"szu-yu-chou","given":"Szu-Yu","family":"Chou","affiliation":"Graduate Institute of Networking and Multimedia , National Taiwan University , Taipei , Taiwan; Research Center for Information Technology Innovation , Academia Sinica , Taipei , Taiwan"},{"name":"Jyh-Shing Jang","slug":"jyh-shing-jang","given":"Jyh-Shing","family":"Jang","affiliation":"Graduate Institute of Networking and Multimedia , National Taiwan University , Taipei , Taiwan"},{"name":"Yi-Hsuan Yang","slug":"yi-hsuan-yang","given":"Yi-Hsuan","family":"Yang","affiliation":"Research Center for Information Technology Innovation , Academia Sinica , Taipei , Taiwan"}],"abstract":"Making sense of the surrounding context and ongoing\r\nevents through not only the visual inputs but\r\nalso acoustic cues is critical for various AI applications.\r\nThis paper presents an attempt to learn\r\na neural network model that recognizes more than\r\n500 different sound events from the audio part of\r\nuser generated videos (UGV). Aside from the large\r\nnumber of categories and the diverse recording conditions\r\nfound in UGV, the task is challenging because\r\na sound event may occur only for a short\r\nperiod of time in a video clip. Our model specifically\r\ntackles this issue by combining a main subnet\r\nthat aggregates information from the entire clip\r\nto make clip-level predictions, and a supplementary\r\nsubnet that examines each short segment of the clip\r\nfor segment-level predictions. As the labeled data\r\navailable for model training are typically on the clip\r\nlevel, the latter subnet learns to pay attention to segments\r\nselectively to facilitate attentional segment-level\r\nsupervision. We call our model the M&mnet,\r\nfor it leverages both “M”acro (clip-level) supervision\r\nand “m”icro (segment-level) supervision derived\r\nfrom the macro one. Our experiments show\r\nthat M&mnet works remarkably well for recognizing\r\nsound events, establishing a new state-of-theart\r\nfor DCASE17 and AudioSet data sets. Qualitative\r\nanalysis suggests that our model exhibits strong\r\ngains for short events. In addition, we show that the\r\nmicro subnet is computationally light and we can\r\nuse multiple micro subnets to better exploit information\r\nin different temporal scales.","doi":"10.24963/ijcai.2018/463","url":"https://ijcai.org/proceedings/2018/463","pdf":"https://ijcai.org/proceedings/2018/0463.pdf","markdown":"https://ijcai.org/proceedings/2018/463.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/463","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":463,"pages":{"first":3336,"last":3342},"keywords":["Machine Learning: Deep Learning","Machine Learning Applications: Applications of Supervised Learning","Machine Learning Applications: Other Applications"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chou et al., 2017] Szu-Yu Chou, Jyh-Shing Jang, and Yi- Hsuan Yang. FrameCNN: A weakly-supervised learning framework for frame-wise acoustic event detection and classification. In Proc. DCASE, 2017.","title":"FrameCNN: A weakly-supervised learning framework for frame-wise acoustic event detection and classification","authors":"Szu-Yu Chou, Jyh-Shing Jang, Yi-Hsuan Yang","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Diba et al., 2016] Ali Diba, Vivek Sharma, Ali Mohammad Pazandeh, Hamed Pirsiavash, and Luc Van Gool. Weakly supervised cascaded convolutional networks. In Proc. CVPR, 2016.","title":"Weakly supervised cascaded convolutional networks","authors":"Ali Diba, Vivek Sharma, Ali Mohammad Pazandeh, Hamed Pirsiavash, Luc Van Gool","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dieleman and Schrauwen, 2013] Sander Dieleman and Benjamin Schrauwen. Multiscale approaches to music audio feature learning. In Proc. ICASSP, 2013.","title":"Multiscale approaches to music audio feature learning","authors":"Schrauwen Dieleman, Sander Dieleman, Benjamin Schrauwen","year":"2013","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eghbal-Zadeh et al., 2016] Hamid Eghbal-Zadeh, Bernhard Lehner, Matthias Dorfer, and Gerhard Widmer. CP-JKU submissions for DCASE-2016: a hybrid approach using binaural i-vectors and deep convolutional neural networks. In Proc. DCASE, 2016.","title":"CP-JKU submissions for DCASE-2016: a hybrid approach using binaural i-vectors and deep convolutional neural networks","authors":"Hamid Eghbal-Zadeh, Bernhard Lehner, Matthias Dorfer, Gerhard Widmer","year":"2016","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gemmeke et al., 2017] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP, 2017.","title":"Audio set: An ontology and human-labeled dataset for audio events","authors":"F Jort, Gemmeke, P W Daniel, Dylan Ellis, Aren Freedman, Wade Jansen, R Channing Lawrence, Manoj Moore, Marvin Plakal, Ritter","year":"2017","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girdhar and Ramanan, 2017] Rohit Girdhar and Deva Ra- manan. Attentional pooling for action recognition. In Proc. NIPS, 2017.","title":"Attentional pooling for action recognition","authors":"Rohit Girdhar, Deva Ramanan","year":"2017","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hershey et al., 2017] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. CNN architectures for large-scale audio classifi- cation. In Proc. ICASSP. 2017.","title":"CNN architectures for large-scale audio classification","authors":"Shawn Hershey, Sourish Chaudhuri, P W Daniel, Jort F Ellis, Aren Gemmeke, Channing Jansen, Manoj Moore, Devin Plakal, Rif A Platt, Bryan Saurous, Malcolm Seybold, Ron Slaney, Kevin Weiss, Wilson","year":"2017","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Yu-Siang Huang, Szu-Yu Chou, and Yi- Hsuan Yang. Music thumbnailing via neural attention modeling of music emotion. In Proc. APSIPA ASC, 2017.","title":"Music thumbnailing via neural attention modeling of music emotion","authors":"Yu-Siang Huang, Szu-Yu Chou, Yi-Hsuan Yang","year":"2017","venue":"Proc. APSIPA ASC","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In Proc. ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kendall et al., 2017] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXiv, 2017.","title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","authors":"Alex Kendall, Yarin Gal, Roberto Cipolla","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2018] Anurag Kumar, Maksim Khadkevich, and Christian Fügen. Knowledge transfer from weakly la- beled audio using convolutional neural network for sound events and scenes. In Proc. ICASSP, 2018.","title":"Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes","authors":"Anurag Kumar, Maksim Khadkevich, Christian Fügen","year":"2018","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2017a] Donmoon Lee, Subin Lee, Yoonchang Han, and Kyogu Lee. Ensemble of convolutional neural networks for weakly-supervised sound event detection us- ing multiple scale input. In Proc. DCASE, 2017.","title":"Ensemble of convolutional neural networks for weakly-supervised sound event detection using multiple scale input","authors":"Lee","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2017b] Jongpil Lee, Jiyoung Park, and Juhan Nam. Combining multi-scale features using sample-level deep convolutional neural networks for weakly supervised sound event detection. In Proc. DCASE, 2017.","title":"Combining multi-scale features using sample-level deep convolutional neural networks for weakly supervised sound event detection","authors":"Lee","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2016] Tsung-Yi Lin, Piotr Dollár, Ross B. Gir- shick, Kaiming He, Bharath Hariharan, and Serge J. Be- longie. Feature pyramid networks for object detection. In Proc. CVPR, 2016.","title":"Belongie. Feature pyramid networks for object detection","authors":"Tsung-Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He, Bharath Hariharan, Serge J","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and Yang, 2016] Jen-Yu Liu and Yi-Hsuan Yang. Event localization in music auto-tagging. In Proc. MM, 2016.","title":"Event localization in music auto-tagging","authors":"Jen-Yu Liu, Yi-Hsuan Yang","year":"2016","venue":"Proc. MM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nietok. librosa: Audio and music signal analysis in Python. In Proc. scipy, 2015.","title":"librosa: Audio and music signal analysis in Python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P W Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nietok","year":"2015","venue":"Proc. scipy","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mesaros et al., 2016] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. TUT database for acoustic scene classification and sound event detection. In Proc. EU- SIPCO, 2016.","title":"TUT database for acoustic scene classification and sound event detection","authors":"Annamaria Mesaros, Toni Heittola, Tuomas Virtanen","year":"2016","venue":"Proc. EU-SIPCO","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mesaros et al., 2017] Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Em- manuel Vincent, Bhiksha Raj, and Tuomas Virtanen. DCASE 2017 Challenge setup: Tasks, datasets and base- line system. In Proc. DCASE, 2017.","title":"DCASE 2017 Challenge setup: Tasks, datasets and baseline system","authors":"Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, Tuomas Virtanen","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Piczak, 2015] Karol J. Piczak. ESC: Dataset for environ- mental sound classification. In Proc. ACM MM, 2015.","title":"ESC: Dataset for environmental sound classification","authors":"J Karol, Piczak","year":"2015","venue":"Proc. ACM MM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Salamon et al., 2014] J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In Proc. ACM MM, 2014.","title":"A dataset and taxonomy for urban sound research","authors":"J Salamon, C Jacoby, J P Bello","year":"2014","venue":"Proc. ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shen et al., 2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. DiSAN: Di- rectional self-attention network for RNN/CNN-free lan- guage understanding. In Proc. AAAI, 2018.","title":"DiSAN: Directional self-attention network for RNN/CNN-free language understanding","authors":"Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, Chengqi Zhang","year":"2018","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[van den Oord et al., 2013] Aaron van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-based music recommendation. In Proc. NIPS, 2013.","title":"Deep content-based music recommendation","authors":"Oord Van Den","year":"2013","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vu et al., 2017] Toan Vu, An Dang, and Jia-Ching Wang. Deep learning for DCASE2017 Challenge. In Proc. DCASE, 2017.","title":"Deep learning for DCASE2017 Challenge","authors":"Toan Vu, An Dang, Jia-Ching Wang","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2017a] Yong Xu, Qiuqiang Kong, Qiang Huang, Wenwu Wang, and Mark D. Plumbley. Attention and lo- calization based on a deep convolutional recurrent model for weakly supervised audio tagging. In Proc. INTER- SPEECH, 2017.","title":"Attention and localization based on a deep convolutional recurrent model for weakly supervised audio tagging","authors":"Xu","year":"2017","venue":"Proc. INTER-SPEECH","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2017b] Yong Xu, Qiuqiang Kong, Wenwu Wang, and Mark D. Plumbley. Surrey-CVSSP system for DCASE2017 Challenge Task4. In Proc. DCASE, 2017.","title":"Surrey-CVSSP system for DCASE2017 Challenge Task4","authors":"Xu","year":"2017","venue":"Proc. DCASE","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2016] Han Zhang, Tao Xu, Mohamed Elho- seiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. SPDA-CNN: Unifying semantic part detection and abstraction for fine-grained recognition. In Proc. CVPR, 2016.","title":"SPDA-CNN: Unifying semantic part detection and abstraction for fine-grained recognition","authors":"Han Zhang, Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, Dimitris Metaxas","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2017] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial reg- ularization with image-level supervisions for multi-label image classification. In Proc. CVPR, 2017.","title":"Learning spatial regularization with image-level supervisions for multi-label image classification","authors":"Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, Xiaogang Wang","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}