Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation

1 Introduction

Live game commentary describes in-game events and provides contextual explanations to help viewers immerse themselves in gameplay [Behrens and Uhrich, 2022; da Silva and Scelles, 2025]. Audio commentary is integrated with the livestreamed video and delivered to users via online streaming platforms. Advances in multimodal language models and audio synthesis have accelerated research on automatic audio commentary generation [Zheng et al., 2025]. Prior work has explored language generation from structured data [Taniguchi et al., 2019; Ishigaki et al., 2021], video [Yu et al., 2018; Kim and Choi, 2020; Rao et al., 2024; Zhou et al., 2024; Chen et al., 2025; Afzal et al., 2026], and multimodal inputs [Mori et al., 2025; Someya et al., 2025; Wang and Yoshinaga, 2025], as well as expressive speech synthesis for engaging commentary[Iura et al., 2025]; however, these two lines of research have mostly progressed independently.

Building on these separate research directions, several works have begun to combine language generation and speech synthesis into end-to-end audio commentary systems [Kumano et al., 2019; Jumneanbun et al., 2020; Xu et al., 2021; Ishigaki et al., 2023]. However, existing systems rely on structured game state inputs and lightweight language models, which prevent latency from becoming a critical bottleneck. Extending this setting to raw gameplay video and modern multimodal large language models (LLMs) introduces substantial latency challenges [Mori et al., 2025].

A major bottleneck in real-time audio commentary is its pipeline design. Most pipelines are sequential: they process game inputs, generate text, synthesize speech, and start generating the next utterance only after the current speech playback has completed. As a result, waiting time accumulates across turns and leads to long silence between utterances, which substantially degrades perceived commentary quality.

To address this latency bottleneck, we propose an audio commentary system with parallel text generation and a lightweight control of video delay. While one utterance is being spoken, our system continues generating candidate utterances in the background process for newly arriving video segments and stores them in a buffer. When the current speech playback ends, the system immediately selects a buffered candidate and synthesizes speech from it, thereby avoiding silence caused by waiting for generation to finish. Since inference latency is inevitable in practice, we additionally implement a lightweight video delay control that intentionally delays the outgoing video stream to better align generated speech with the displayed visuals.

Experiments on fast-paced game videos [Saito et al., 2020] show that our system reduces mean inter-utterance silence from 9.6 seconds to 0.3 seconds, a gain further supported by a human evaluation with 120 participants.

2 Real-Time Game Audio Commentary

A typical real-time audio commentary pipeline converts a gameplay stream into audio commentary by executing (i) video acquisition and segmentation, (ii) text generation, and (iii) speech synthesis. The video stream is buffered and grouped into short frame segments. Let fi denote the frame captured at time i; then a segment is Fk = {fi, . . . , fi+N-1}, which consists of N consecutive frames. A common design is to run these modules sequentially. Specifically, the system waits until the current speech playback ends, and then calls a multimodal LLM on the latest segment to generate text using a simple prompt (e.g., "Describe the game situation. If you have nothing to say, stay silent.") [Afzal et al., 2026]. The generated text is then converted into audio by a text-to-speech (TTS) module [Iura et al., 2025]. Because the next generation is triggered only after playback finishes, this strict sequentiality accumulates latency and causes long silences between utterances.

3 Proposed System

We propose a low-latency system design based on two key ideas: parallel text generation (our main mechanism to reduce waiting time) and lightweight video delay control (to maintain temporal consistency).

3.1 Parallel Text Generation

Unlike conventional systems that trigger text generation only after speech playback ends, our system initiates text generation as soon as a new video segment becomes available, even while the current utterance is still being spoken (Figure 1). As a result, multiple candidate utterances are generated ahead of playback and stored in a buffer. When the current utterance ends, the system selects one buffered candidate and synthesizes it immediately, eliminating idle silence caused by sequential waiting.

We consider three lightweight selection policies: Latest (most recent segment), Oldest (earliest buffered), and Random. We adopt these lightweight policies to avoid additional decision latency in real-time settings.

3.2 Video Delay Control

We consider a streaming setup for live streaming platforms such as YouTube Live, where gameplay video and automatically generated speech are integrated into a single audiovisual stream on our server and then uploaded to the platform. This setting allows us to control video playback timing to mitigate misalignment caused by generation latency.

Since text generation and TTS inevitably introduce delays that cause commentary to lag, we absorb this latency by intentionally delaying the video. Specifically, we buffer the video stream and start playback only when the first utterance begins, matching the initial end-to-end generation latency.

4 Experiments

Demo Setups: In the demonstration, conference attendees voluntarily operate a Nintendo Switch console on-site, and the gameplay video is captured in real time using a capture device (Elgato HD60 X 1 ) and streamed to a commentary server. The system generates spoken commentary online based on the live video input. The demo is conducted exclusively on-site, and no gameplay video or audio is recorded or distributed. Attendees observe the gameplay together with automatically generated commentary, allowing them to directly perceive how parallel candidate generation affects inter-utterance silence and speaking rhythm. The gameplay machine sends video at 25 fps, and captured frames are grouped into segments of N consecutive frames. In all experiments, we set N = 32. Each segment Fk = {fi, . . . , fi+N-1} is sent to the commentary server as soon as it becomes available. The language model receives each segment as base64-encoded tokens and generates commentary text. We use GPT-4.1-mini 2 for text generation and follow the TTS configuration described in [Iura et al., 2025]. Our experiments vary the max new tokens parameter in {20, 40, 60, 80, 100}.

Dataset: We randomly selected 8 videos from the Smash Corpus [Saito et al., 2020] whose target game is Super Smash Bros. Ultimate. We select this game for our experiments because it is a fast-paced game where latency is particularly noticeable. This corpus provides gameplay video. We collected commentaries from skilled commentators for the selected videos, enabling a comparison between skilled humanand system-generated commentaries.

Compared Models: We compare five methods. All methods use the same video delay control (Section 3.2). Baseline After-Audio is a fully sequential pipeline that starts generating the next utterance only after the current speech playback ends, following a common design in prior real-time commentary systems [Ishigaki et al., 2023]. Baseline After-Text is a semi-sequential variant that starts generating the next utterance immediately after text generation finishes, without waiting for speech playback to end. Our system uses parallel generation with buffering, and we test three lightweight selection policies: Parallel Latest, Prallel Oldest, and Parallel Random.

1 https://www.elgato.com/jp/ja/p/game-capture-hd60-x

2 https://platform.openai.com/docs/models/gpt-4.1-mini

-

9.5 ±0.9 0.01 ±0.06

6.8 ±0.9 0.10 ±0.08

0.4 ±0.1 0.59 ±0.04

0.3±0.1 0.60±0.04

0.3±0.1 0.60±0.03

Table 2: Effect of max new tokens on speaking duration and text length (Latest policy).

Evaluation Metrics: We evaluate commentaries in terms of timing behavior and content adequacy. For timing, we compare cumulative silence, mean silence between utterances, and utterance length. To capture similarity of speaking and silence patterns to human-spoken commentaries, we represent each commentary stream as a 1Hz binary sequence (1 = speaking, 0 = silence) and compute the mean Intersection over Union (mIoU) between automatically generated commentaries and skilled human-spoken ones. Higher mIoU indicates fewer mismatches in speaking and silence regions compared to skilled human-spoken commentaries. For content adequacy, we compute ROUGE [Lin, 2004] scores against reference commentary text. Because exact temporal alignment is difficult in real-time settings, ROUGE is computed within fixed 10-second windows.

For human evaluation, we conduct a user study with 120 crowdworkers recruited via Lancers 3 . We present 30-second clips sampled from 20 scenes. Each clip is paired with commentary generated by one of 25 conditions (5 methods × 5 max new tokens settings). Participants rate on 5-point Likert scales, and we report Mean Opinion Scores (MOS) for the following criteria: (Q1) commentary rhythm naturalness, (Q2) alignment with the video, and (Q3) overall quality.

5 Results

Table 1 summarizes silence-related statistics and mIoU for each method. The sequential baselines (After-Audio and After-Text) exhibit approximately twice the cumulative silence of human commentary and extremely low mIoU scores, indicating that sequential processing introduces long and unnatural pauses between utterances. In contrast, all proposed parallel methods substantially reduce silence, achieving mIoU values close to human commentary. Differences among the selection policies are relatively small, suggesting that the main benefit comes from parallel generation itself.

3 https://www.lancers.jp/

Table 2 analyzes the effect of max new tokens under the Latest policy. While a setting of 40 best matches human speaking duration, 60 yields utterance lengths closest to human commentary, indicating that generation length affects different aspects of temporal behavior. ROUGE scores (Figure 2) show consistent improvements of parallel methods over sequential baselines.

Figure 3 reports the results of the human evaluation. The proposed methods significantly outperform the baselines across all criteria (Q1-Q3), demonstrating that reducing unnatural silence leads to higher perceived commentary quality.

6 Conclusion

We presented a real-time game audio commentary system that reduces latency by parallel generation. Experiments show substantial reductions in inter-utterance silence and improved similarity to professional speaking/silence patterns.

Acknowledgements

This paper is based on results obtained from a project, Programs for Bridging the gap between R&D and the IDeal society (society 5.0) and Generating Economic and social value (BRIDGE)/Practical Global Research in the AI × Robotics Services, implemented by the Cabinet Office, Government of Japan. This paper is also based on results obtained from AIST policy-based budget project "R&D on Generative AI Foundation Models for the Physical Domain".