{"type":"ScholarlyArticle","title":"A Deep Reinforcement Learning Approach to Balance Viewport Prediction and Video Transmission in 360° Video Streaming","authors":[{"name":"Guanghui Zhang","slug":"guanghui-zhang","given":"Guanghui","family":"Zhang","affiliation":""},{"name":"Jing Guo","slug":"jing-guo","given":"Jing","family":"Guo","affiliation":""}],"abstract":"360° video streaming has seen tremendous growth in past years. However, our measurement reveals a dilemma that severely limits QoE. On the one hand, viewport prediction requires the shortest possible prediction distance for high predicting accuracy; On the other hand, video transmission requires more buffered data to compensate for bandwidth fluctuations otherwise substantial playback rebuffering would be incurred. Since no existing method can break this dilemma, the QoE optimization was naturally bottlenecked. This work tackles this challenge by developing QUTA – a novel learning-based streaming system. Specifically, our measurement shows that three kinds of internal streaming parameters have significant impacts on the prediction distance, namely, download pause, data rate threshold, and playback rate. On top of this, we design a new long-term-planning (LTP) learning method that tunes the parameters dynamically based on the network and streaming context. Evaluations with large-scale streaming trace data show that QUTA not only improves the prediction accuracy and QoE by up to 68.4% but also exhibits strong robustness.","doi":"10.24963/ijcai.2024/680","url":"https://ijcai.org/proceedings/2024/680","pdf":"https://ijcai.org/proceedings/2024/0680.pdf","markdown":"https://ijcai.org/proceedings/2024/680.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/680","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":680,"pages":{"first":6152,"last":6160},"keywords":["Multidisciplinary Topics and Applications: MTA: Transportation"],"videos":[],"references":[{"position":1,"raw":"[Yaqoob et al., 2020] A. Yaqoob, T. Bi and G. Muntean, \"A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities,\" IEEE Communications Surveys & Tutorials, vol. 22(4), Fourth Quarter 2020, pp. 2801-2838.","title":"A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities","authors":"A Yaqoob, T Bi, G Muntean","year":null,"venue":"Fourth Quarter 2020","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Stockhammer et al., 2011] T. Stockhammer, \"Dynamic Adaptive Streaming over HTTP: Standards and Design Principles,\" Proc. ACM Multimedia Systems Conference (MMSys'11), San Jose, USA, Feb 2011, pp.133-144.","title":"Dynamic Adaptive Streaming over HTTP: Standards and Design Principles","authors":"T Stockhammer","year":"2011","venue":"Proc. ACM Multimedia Systems Conference (MMSys'11)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Qian et al., 2018] F. Qian, B. Han, Q. Xiao, and V. Gopalakrishnan, \"Flare: Practical Viewport-adaptive 360- degree Video Streaming for Mobile Devices,\" Proc. Annual International Conference on Mobile Computing and Networking (MobiCom'18), New Delhi, India, Oct. 2018, pp. 99-114.","title":"Flare: Practical Viewport-adaptive 360degree Video Streaming for Mobile Devices","authors":"F Qian, B Han, Q Xiao, V Gopalakrishnan","year":"2018","venue":"Proc. Annual International Conference on Mobile Computing and Networking (MobiCom'18)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guan et al., 2019] Y. Guan, C. Zheng, X. Zhang, Z. Guo, and J. Jiang. \"Pano: Optimizing 360° Video Streaming with A Better Understanding of Quality Perception,\" Proc. ACM Special Interest Group on Data Communication (SIGCOMM'19), Beijing, China, Aug 2019, pp. 394-407.","title":"Pano: Optimizing 360° Video Streaming with A Better Understanding of Quality Perception","authors":"Y Guan, C Zheng, X Zhang, Z Guo, J Jiang","year":"2019","venue":"Proc. ACM Special Interest Group on Data Communication (SIGCOMM'19)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Zhang et al., 2021] L. Zhang, Y. Suo, X. Wu, F. Wang, Y. Chen, L. Cui, J. Liu, and Z. Ming, \"TBRA: Tiling and Bitrate Adaptation for Mobile 360-Degree Video Streaming,\" Proc. ACM International Conference on Multimedia (MM'21), Chengdu, China, Oct. 2021, pp. 4007-4015.","title":"TBRA: Tiling and Bitrate Adaptation for Mobile 360-Degree Video Streaming","authors":"L Zhang, Y Suo, X Wu, F Wang, Y Chen, L Cui, J Liu, Z Ming","year":"2021","venue":"Proc. ACM International Conference on Multimedia (MM'21)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Zhang et al., 2019] Y. Zhang, P. Zhao, K. Bian, Y. Liu, L. Song, and X. Li. \"DRL360: 360-degree Video Streaming with Deep Reinforcement Learning,\" Proc. IEEE Conference on Computer Communications (INFOCOM 2019), Paris, France, April 2019, pp. 1252-1260.","title":"DRL360: 360-degree Video Streaming with Deep Reinforcement Learning","authors":"Y Zhang, P Zhao, K Bian, Y Liu, L Song, X Li","year":"2019","venue":"Proc. IEEE Conference on Computer Communications (INFOCOM 2019)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Wu et al., 2021] C. Wu, Z. Wang, and L. Sun, \"PAAS: A Preference-aware Deep Reinforcement Learning Approach for 360° Video Streaming,\" Proc. ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'21), Oct 2021, Istanbul, Turkey, pp. 34-41.","title":"PAAS: A Preference-aware Deep Reinforcement Learning Approach for 360° Video Streaming","authors":"C Wu, Z Wang, L Sun","year":"2021","venue":"Proc. ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'21)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Nasrabadi et al., 2020] A. Nasrabadi, A. Samiei, and R. Prakash, \"Viewport Prediction for 360° Videos: A Clustering Approach,\" Proc. ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'20), Istanbul, Turkey, June 2020, pp. 34-39.","title":"Viewport Prediction for 360° Videos: A Clustering Approach","authors":"A Nasrabadi, A Samiei, R Prakash","year":"2020","venue":"Proc. ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'20)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Xu et al., 2019] M. Xu, Y. Song, J. Wang, M. Qiao, L. Huo, and Z. Wang, \"Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach,\" IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41(11), Nov. 2019, pp. 2693-2708.","title":"Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach","authors":"M Xu, Y Song, J Wang, M Qiao, L Huo, Z Wang","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nguyen et al., 2018] A. Nguyen, Z. Yan, and K. Nahrstedt. \"Your attention is Unique: Detecting 360-degree Video Saliency in Head-mounted Display for Head Movement Prediction,\" Proc. ACM International Conference on Multimedia (MM'18), Seoul, South Korea, Oct. 2018, pp. 1190-1198.","title":"Your attention is Unique: Detecting 360-degree Video Saliency in Head-mounted Display for Head Movement Prediction","authors":"A Nguyen, Z Yan, K Nahrstedt","year":"2018","venue":"Proc. ACM International Conference on Multimedia (MM'18)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Xu et al., 2018] Y. Xu, Y. Dong, J. Wu, Z. Sun, Z. Shi, J. Yu, and S. Gao, \"Gaze Prediction in Dynamic 360 Immersive Videos,\" IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, June 2018, pp. 5333-5342.","title":"Gaze Prediction in Dynamic 360 Immersive Videos","authors":"Y Xu, Y Dong, J Wu, Z Sun, Z Shi, J Yu, S Gao","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rondon et al., 2021] M. F. R. Rondon, L. Sassatelli, R. Aparicio-Pardo and F. Precioso, \"TRACK: A New Method from a Re-examination of Deep Architectures for Head Motion Prediction in 360 Videos,\" IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44(9), April 2021, pp. 5681-5699.","title":"TRACK: A New Method from a Re-examination of Deep Architectures for Head Motion Prediction in 360 Videos","authors":"M F R Rondon, L Sassatelli, R Aparicio-Pardo, F Precioso","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chopra et al., 2021] L. Chopra, S. Chakraborty, A. Mondal, and S. Chakraborty, \"PARIMA: Viewport Adaptive 360- Degree Video Streaming,\" Proc. The Web Conference (WWW'21), Virtual, April 2021, pp. 2379-2391.","title":"PARIMA: Viewport Adaptive 360-Degree Video Streaming","authors":"L Chopra, S Chakraborty, A Mondal, S Chakraborty","year":"2021","venue":"Proc. The Web Conference (WWW'21)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2022] S. Wang, S. Yang, H. Li, X. Zhang, C. Zhou, C. Xu, F. Qian, N Wang, and, Z. Xu, \"SalientVR: Saliency-Driven Mobile 360-Degree Video Streaming with Gaze Information,\" Proc. Annual International Conference on Mobile Computing and Networking (MobiCom'22), Sydney, NSW, Australia, Oct. 2022, pp. 1-14.","title":"SalientVR: Saliency-Driven Mobile 360-Degree Video Streaming with Gaze Information","authors":"S Wang, S Yang, H Li, X Zhang, C Zhou, C Xu, F Qian, Z Wang, Xu","year":"2022","venue":"Proc. Annual International Conference on Mobile Computing and Networking (MobiCom'22)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[YouTube Instance, 2023] An Actual 360° Video for Experiencing the Playback Rate Change on YouTube: https://www.youtube.com/watch?v=hNAbQYU0wpg","title":"An Actual 360° Video for Experiencing the Playback Rate Change on YouTube","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://www.youtube.com/watch?v=hNAbQYU0wpg","internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, \"Asynchronous Methods for Deep Reinforcement Learning,\" International Conference on Machine Learning (ICML 2016), New York City, NY, USA, Jun 2016, pp. 1928-1937.","title":"Asynchronous Methods for Deep Reinforcement Learning","authors":"V Mnih, A P Badia, M Mirza, A Graves","year":"2016","venue":"International Conference on Machine Learning (ICML 2016)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mao et al., 2017] H. Mao, R. Netravali, M. Alizadeh, \"Neural Adaptive Video Streaming with Pensieve,\" Proc. ACM Special Interest Group on Data Communication (SIGCOMM'17), Los Angeles, CA, USA, Aug 2017, pp. 197-210.","title":"Neural Adaptive Video Streaming with Pensieve","authors":"H Mao, R Netravali, M Alizadeh","year":"2017","venue":"Proc. ACM Special Interest Group on Data Communication (SIGCOMM'17)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, et al. \"Human-level Control through Deep Reinforcement Learning,\" Nature, Feb 2015, vol. 518(7540), pp. 529-533.","title":"Human-level Control through Deep Reinforcement Learning","authors":"V Mnih, K Kavukcuoglu, D Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lillicrap et al., 2015] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, \"Continuous Control with Deep Reinforcement Learning,\" arXiv Preprint, Sep 2015.","title":"Continuous Control with Deep Reinforcement Learning","authors":"T Lillicrap, J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton et al., 1988] RS. Sutton, \"Learning to Predict by the Methods of Temporal Differences, Machine Learning, 1988, pp. 9-44.","title":"Learning to Predict by the Methods of Temporal Differences","authors":"Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2022] Y. Lu, Y. Zhu, Z. Wang, \"Personalized 360- Degree Video Streaming: A Meta-Learning Approach,\" Proc. ACM International Conference on Multimedia (MM'22), Lisboa, Portugal, Oct. 2022, pp. 3143-3151.","title":"Personalized 360-Degree Video Streaming: A Meta-Learning Approach","authors":"Y Lu, Y Zhu, Z Wang","year":"2022","venue":"Proc. ACM International Conference on Multimedia (MM'22)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shi et al., 2022] J. Shi, L. Pu, X. Yuan, Q. Gong, J. Xu, \"Sophon: Super-Resolution Enhanced 360° Video Streaming with Visual Saliency-aware Prefetch,\" Proc. ACM International Conference on Multimedia (MM'22), Lisboa, Portugal, Oct. 2022, pp. 3124-3133.","title":"Sophon: Super-Resolution Enhanced 360° Video Streaming with Visual Saliency-aware Prefetch","authors":"J Shi, L Pu, X Yuan, Q Gong, J Xu","year":"2022","venue":"Proc. ACM International Conference on Multimedia (MM'22)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Emulator, 2021] Trace Data Driven Emulator: http://gitlab.com/miguelfromeror/head-motion- prediction [YouTube, 2023] Encoding Recommendations: https://support.google.com/youtube/answer/2853702","title":"Trace Data Driven Emulator","authors":"Emulator","year":"2021","venue":null,"doi":null,"url":"https://support.google.com/youtube/answer/2853702","internal_url":null},{"position":24,"raw":"Trace 2023] TCP Throughput Trace Data: https://github.com/Streaming-code/TraceData/releases /tag/TraceData","title":"TCP Throughput Trace Data","authors":null,"year":null,"venue":null,"doi":null,"url":"https://github.com/Streaming-code/TraceData/releases/tag/TraceData","internal_url":null},{"position":25,"raw":"[Nguyen et al., 2022] D. Nguyen, L. Ngan, L. Thuong and T. Huong, \"LL-VAS: Adaptation Method for Low-Latency 360-degree Video Streaming over Mobile Networks,\" Proc. IEEE Symposium on Computers and Communications (ISCC), Rhodes, Greece, Oct. 2022, pp. 1-6.","title":"LL-VAS: Adaptation Method for Low-Latency 360-degree Video Streaming over Mobile Networks","authors":"D Nguyen, L Ngan, L Thuong, T Huong","year":"2022","venue":"Proc. IEEE Symposium on Computers and Communications (ISCC)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] G. Zhang, and J. Lee, \"LAPAS: Latency- Aware Playback-Adaptive Streaming,\" Proc. IEEE Wireless Communications and Networking Conference (WCNC), Marrakesh, Morocco, Apr. 2019, pp.1-6.","title":"LAPAS: Latency-Aware Playback-Adaptive Streaming","authors":"G Zhang, J Lee","year":"2019","venue":"Proc. IEEE Wireless Communications and Networking Conference (WCNC)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2021] B. Zhang, T. Teixeira, and Y. Reznik, \"Performance of Low-Latency HTTP-based Streaming Players,\" Proc. ACM Multimedia Systems Conference (MMSys '21), Istanbul, Turkey, Sep 2021, pp. 356-362.","title":"Performance of Low-Latency HTTP-based Streaming Players","authors":"B Zhang, T Teixeira, Y Reznik","year":"2021","venue":"Proc. ACM Multimedia Systems Conference (MMSys '21)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Dummynet, 2023] An Improved Version of Dummynet: https://github.com/mclab-cuhk/netmap-ipfw [","title":"An Improved Version of Dummynet","authors":"Dummynet","year":"2023","venue":null,"doi":null,"url":"https://github.com/mclab-cuhk/netmap-ipfw","internal_url":null},{"position":29,"raw":"[Fujimoto et al., 2018] S. Fujimoto, H. Hoof, D. Meger, \"Addressing Function Approximation Error in Actor- Critic Methods,\" Proc. International Conference on Machine Learning, Stockholm, Sweden, July 2018, pp. 1587-1596.","title":"Addressing Function Approximation Error in Actor-Critic Methods","authors":"S Fujimoto, H Hoof, D Meger","year":"2018","venue":"Proc. International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}