Benchmarking Real-Time Question Answering via Executable Code Workflows

Abstract

Retrieving real time information is a fundamental capability for agents with search capabilities in real world applications. However, existing benchmarks are predominantly static and therefore fail to capture the temporal dynamics of information and the continuously evolving nature of real world knowledge. To address this limitation, we propose RT-QA, a dynamic evaluation framework that leverages executable code workflows to retrieve current answers at evaluation time. Specifically, we construct an agent based pipeline that autonomously generates code for web crawling and DOM based answer extraction to produce real time ground truth. To ensure robust evaluation over time, the pipeline further incorporates a repair mechanism to adapt to changes in web page structures. RT-QA spans 12 domains, such as Finance and Sports, with 320 Chinese questions categorized into three difficulty levels. Extensive evaluations of advanced models, such as GPT-5.2 and GLM-4.7, reveal significant limitations in real time adaptability: even the best models achieve only 46% accuracy. Our analysis highlights two primary failure modes: (1) Lazy Retrieval, where agents rely on search snippets instead of deeply scanning specific websites for information, accounting for 20% of failures; and (2) Temporal Confusion, a cognitive error where agents retrieve a historical date, for example an event in 2024, and fail to anchor again to the current time, 2026, for subsequent reasoning. These findings suggest that future agents require not just better retrieval strategies, but robust temporal state management.