Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU

Guo, Shuxi; Xu, Zikang; Liu, Jiahao; Zhang, Jinyi; Qi, Qi; Sun, Haifeng; Huang, Jun; Liao, Jianxin; Wang, Jingyu

doi:10.24963/ijcai.2025/318

Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU

Shuxi Guo, Zikang Xu, Jiahao Liu, Jinyi Zhang, Qi Qi, Haifeng Sun, Jun Huang, Jianxin Liao, Jingyu Wang

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 2856-2864.

https://doi.org/10.24963/ijcai.2025/318

Keywords

Data Mining: DM: Recommender systems
Planning and Scheduling: PS: Scheduling
Data Mining: DM: Information retrieval

Abstract

Deep learning-based recommendation systems are increasingly important in the industry. To meet strict SLA requirements, serving frameworks must efficiently handle concurrent queries. However, current serving systems fail to serve concurrent queries due to the following problems: (1) inefficient operator (op) scheduling due to the query-wise op launching mechanism, and (2) heavy contention caused by the mutable nature of recommendation model inference. This paper presents RecOS, a system designed to optimize concurrent recommendation model inference on GPUs. RecOS efficiently schedules ops from different queries by monitoring GPU workloads and assigning ops to the most suitable streams. This approach reduces contention and enhances inference efficiency by leveraging inter-op parallelism and op characteristics. To maintain correctness across multiple CUDA streams, RecOS introduces a unified asynchronous tensor management mechanism. Evaluations demonstrate that RecOS improves online service performance, reducing latency by up to 68%.

Shuxi Guo, Zikang Xu, Jiahao Liu, Jinyi Zhang, Qi Qi, Haifeng Sun, Jun Huang, Jianxin Liao, and Jingyu Wang. 2025. Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 2856–2864. https://doi.org/10.24963/ijcai.2025/318

@inproceedings{ijcai2025p318,
  title     = {Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU},
  author    = {Guo, Shuxi and Xu, Zikang and Liu, Jiahao and Zhang, Jinyi and Qi, Qi and Sun, Haifeng and Huang, Jun and Liao, Jianxin and Wang, Jingyu},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {2856--2864},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/318},
  url       = {https://doi.org/10.24963/ijcai.2025/318},
}

TY  - CPAPER
TI  - Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU
AU  - Guo, Shuxi
AU  - Xu, Zikang
AU  - Liu, Jiahao
AU  - Zhang, Jinyi
AU  - Qi, Qi
AU  - Sun, Haifeng
AU  - Huang, Jun
AU  - Liao, Jianxin
AU  - Wang, Jingyu
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 2856
EP  - 2864
DO  - 10.24963/ijcai.2025/318
UR  - https://doi.org/10.24963/ijcai.2025/318
ER  -