{"type":"ScholarlyArticle","title":"k-SDPP: Fixed-Size Video Summarization via Sequential Determinantal Point Processes","authors":[{"name":"Jiping Zheng","slug":"jiping-zheng","given":"Jiping","family":"Zheng","affiliation":"College of Computer Science & Technology , Nanjing University of Aeronautics & Astronautics; Collaborative Innovation Center of Novel Software Technology and Industrialization"},{"name":"Ganfeng Lu","slug":"ganfeng-lu","given":"Ganfeng","family":"Lu","affiliation":"College of Computer Science & Technology , Nanjing University of Aeronautics & Astronautics"}],"abstract":"With the explosive growth of video data, video summarization which converts long-time videos to key frame sequences has become an important task in information retrieval and machine learning. Determinantal point processes (DPPs) which are elegant probabilistic models have been successfully applied to video summarization. However, existing DPP-based video summarization methods suffer from poor efficiency of outputting a specified size summary or neglecting inherent sequential nature of videos. In this paper, we propose a new model in the DPP lineage named k-SDPP in vein of sequential determinantal point processes but with fixed user specified size k. Our k-SDPP partitions sampled frames of a video into segments where each segment is with constant number of video frames. Moreover, an efficient branch and bound method (BB) considering sequential nature of the frames is provided to optimally select k frames delegating the summary from the divided segments. Experimental results show that our proposed BB method outperforms not only k-DPP and sequential DPP (seqDPP) but also the partition and Markovian assumption based methods.","doi":"10.24963/ijcai.2020/108","url":"https://ijcai.org/proceedings/2020/108","pdf":"https://ijcai.org/proceedings/2020/0108.pdf","markdown":"https://ijcai.org/proceedings/2020/108.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/108","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":108,"pages":{"first":774,"last":781},"keywords":["Computer Vision: Big Data and Large Scale Methods","Computer Vision: Other"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25263/file_25263.m3u8","length_sec":269,"provider":"hls"}],"references":[{"position":1,"raw":"[Affandi et al., 2012] Raja Hafiz Affandi, Alex Kulesza, and Emily B. Fox. Markov determinantal point processes. In UAI, pages 26-35, 2012.","title":"Markov determinantal point processes","authors":"Raja Hafiz Affandi, Alex Kulesza, Emily B Fox","year":"2012","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsimas and Tsitsiklis, 1997] Dimitris Bertsimas and John Tsitsiklis. Introduction to Linear Optimization. Athena Scientific, 1997.","title":"Introduction to Linear Optimization","authors":"Dimitris Bertsimas, John Tsitsiklis","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Börzsöny et al., 2001] Stephan Börzsöny, Donald Koss- mann, and Konrad Stocker. The skyline operator. In ICDE, pages 421-430, 2001.","title":"The skyline operator","authors":"Stephan Börzsöny, Donald Kossmann, Konrad Stocker","year":"2001","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Celis et al., 2018] Elisa Celis, Vijay Keswani, Damian Straszak, Amit Deshpande, Tarun Kathuria, and Nisheeth Vishnoi. Fair and diverse DPP-based data summarization. In ICML, pages 716-725, 2018.","title":"Fair and diverse DPP-based data summarization","authors":"Elisa Celis, Vijay Keswani, Damian Straszak, Amit Deshpande, Tarun Kathuria, Nisheeth Vishnoi","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chan, 1996] Timothy M. Chan. Optimal output-sensitive convex hull algorithms in two and three dimensions. Dis- crete & Computational Geometry, 16(4):361-368, 1996.","title":"Optimal output-sensitive convex hull algorithms in two and three dimensions","authors":"Timothy M Chan","year":"1996","venue":"Discrete & Computational Geometry","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[de Avila et al., 2011] Sandra Eliza Fontes de Avila, Ana Paula Brandão Lopes, Antonio da Luz, Jr., and Arnaldo de Albuquerque Araújo. Vsumm: A mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recognition Letters, 32(1):56-68, 2011.","title":"Vsumm: A mechanism designed to produce static video summaries and a novel evaluation method","authors":"Sandra Eliza Fontes De Avila, Ana Paula Brandão, Antonio Lopes, Jr Da Luz, Arnaldo De, Albuquerque Araújo","year":"2011","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eri, 2019] Ericsson mobility report. https://www.ericsson. com/en/mobility-report, November 2019. Accessed April 25, 2020.","title":"Ericsson mobility report","authors":"Eri","year":"2019","venue":null,"doi":null,"url":"https://www.ericsson.com/en/mobility-report","internal_url":null},{"position":8,"raw":"[Gong et al., 2014] Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha. Diverse sequential subset selec- tion for supervised video summarization. In NIPS, pages 2069-2077, 2014.","title":"Diverse sequential subset selection for supervised video summarization","authors":"Gong","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hough et al., 2006] J. Ben Hough, Manjunath Krishnapur, Yuval Peres, and Bálint Virág. Determinantal processes and independence. Probability Surveys, 3(1):206-229, 2006.","title":"Determinantal processes and independence","authors":"J Ben Hough, Manjunath Krishnapur, Yuval Peres, Bálint Virág","year":"2006","venue":"Probability Surveys","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin and Sendhoff, 2008] Yaochu Jin and Bernhard Send- hoff. Pareto-based multiobjective machine learning: An overview and case studies. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 38(3):397-415, 2008.","title":"Yaochu Jin and Bernhard Sendhoff. Pareto-based multiobjective machine learning: An overview and case studies","authors":"Jin, Sendhoff","year":"2008","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part C","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kathuria and Deshpande, 2016] Tarun Kathuria and Amit Deshpande. On sampling and greedy MAP inference of constrained determinantal point processes. CoRR, abs/1607.01551, 2016.","title":"On sampling and greedy MAP inference of constrained determinantal point processes","authors":"Tarun Kathuria, Amit Deshpande","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kellerer et al., 2004] Hans Kellerer, Ulrich Pferschy, and David Pisinger. Knapsack problems. Springer, 2004.","title":"Knapsack problems","authors":"Kellerer","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krause et al., 2008] Andreas Krause, Ajit Singh, and Carlos Guestrin. Near-optimal sensor placements in gaussian pro- cesses: Theory, efficient algorithms and empirical studies. JMLR, 9:235-284, 2008.","title":"Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies","authors":"Andreas Krause, Ajit Singh, Carlos Guestrin","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kulesza and Taskar, 2011a] Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In ICML, pages 1193-1200, 2011.","title":"k-dpps: Fixed-size determinantal point processes","authors":"Taskar Kulesza, Alex Kulesza, Ben Taskar","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kulesza and Taskar, 2011b] Alex Kulesza and Ben Taskar. Learning determinantal point processes. In UAI, pages 419-427, 2011.","title":"Learning determinantal point processes","authors":"Taskar Kulesza, Alex Kulesza, Ben Taskar","year":"2011","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulesza and Taskar, 2012] Alex Kulesza and Ben Taskar. Determinantal Point Processes for Machine Learning. Now Publishers Inc., Hanover, MA, USA, 2012.","title":"Determinantal Point Processes for Machine Learning","authors":"Alex Kulesza, Ben Taskar","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016] Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point pro- cesses. In AISTATS, pages 1328-1337, 2016.","title":"Efficient sampling for k-determinantal point processes","authors":"Li","year":"2016","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lowe, 2004] David G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"G David, Lowe","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Macchi and Odile, 1975] Macchi and Odile. The coinci- dence approach to stochastic point processes. Advances in Applied Probability, 7(1):83-122, 1975.","title":"The coincidence approach to stochastic point processes","authors":"Odile ; Macchi, Odile Macchi","year":"1975","venue":"Advances in Applied Probability","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Michie, 1968] Donald Michie. Memo functions and ma- chine learning. Nature, 218:19-22, 1968.","title":"Memo functions and machine learning","authors":"Donald Michie","year":"1968","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mirzasoleiman et al., 2018] Baharan Mirzasoleiman, Ste- fanie Jegelka, and Andreas Krause. Streaming non- monotone submodular maximization: Personalized video summarization on the fly. In AAAI, pages 1379-1386, 2018.","title":"Baharan Mirzasoleiman, Stefanie Jegelka, and Andreas Krause. Streaming nonmonotone submodular maximization: Personalized video summarization on the fly","authors":"Mirzasoleiman","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Morrison et al., 2016] David R. Morrison, Sheldon H. Ja- cobson, Jason J. Sauppe, and Edward C. Sewell. Branch- and-bound algorithms: A survey of recent advances in searching, branching, and pruning. Discrete Optimization, 19:79-102, 2016.","title":"Branchand-bound algorithms: A survey of recent advances in searching, branching, and pruning","authors":"David R Morrison, Sheldon H Jacobson, Jason J Sauppe, Edward C Sewell","year":"2016","venue":"Discrete Optimization","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Perronnin and Dance, 2007] Florent Perronnin and Christo- pher Dance. Fisher kernels on visual vocabularies for im- age categorization. In CVPR, pages 1-8, 2007.","title":"Fisher kernels on visual vocabularies for image categorization","authors":"Florent Perronnin, Christopher Dance","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sharghi et al., 2016] Aidean Sharghi, Boqing Gong, and Mubarak Shah. Query-focused extractive video summa- rization. In ECCV, pages 3-19, 2016.","title":"Aidean Sharghi, Boqing Gong, and Mubarak Shah. Query-focused extractive video summarization","authors":"Sharghi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sharghi et al., 2018] Aidean Sharghi, Ali Borji, Chengtao Li, Tianbao Yang, and Boqing Gong. Improving sequential determinantal point processes for supervised video sum- marization. In ECCV, pages 533-550, 2018.","title":"Improving sequential determinantal point processes for supervised video summarization","authors":"Sharghi","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vivekraj et al., 2019] V.K. Vivekraj, Debashis Sen, and Bal- asubramanian Raman. Video skimming: Taxonomy and comprehensive survey. ACM Computing Surveys, 52(5), 2019.","title":"Balasubramanian Raman. Video skimming: Taxonomy and comprehensive survey","authors":"V K Vivekraj, Debashis Sen","year":"2019","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wolsey, 1998] Laurence A. Wolsey. Integer programming. Wiley, 1998.","title":"Integer programming","authors":"Laurence A Wolsey","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zemel, 1980] Eitan Zemel. The linear multiple choice knap- sack problem. Operations Research, 28(6):1412-1423, 1980.","title":"The linear multiple choice knapsack problem","authors":"Eitan Zemel","year":"1980","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman. Video summarization with long short- term memory. In ECCV, pages 766-782, 2016.","title":"Video summarization with long shortterm memory","authors":"Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2010] Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. Solving the apparent diversity-accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences, PNAS, 107(10):4511-4515, 2010.","title":"Solving the apparent diversity-accuracy dilemma of recommender systems","authors":"Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, Yi-Cheng Zhang","year":"2010","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}