SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning

Humans are natural any-horizon reasoners—we can iteratively skim long videos or watch short ones in full when necessary. Yet, most SOTA models follow the DIRECT paradigm, processing entire long videos in a single turn, which is resource-heavy and inefficient.

To move toward the AGENT paradigm, we explore the following question:

We identify three significant aspects required to answer this:

• System Design: Existing agents often rely solely on temporal grounding, which can be ineffective for long videos.
  Solution: SAGE, a system equipped with tools like Web Search and Speech Transcription to leverage non-visual signals.
• Data: Training long-video agents requires high-quality QnA pairs. Human annotation is expensive for long videos.
  Solution: A cost-effective synthetic pipeline using Gemini-2.5-Flash.
• RL Optimization: The variable duration of videos makes training "any-horizon" behavior difficult. Standard RL recipes (e.g., for math) often lack verifiable rewards for open-ended video tasks.
  Solution: A multi-reward RL recipe using a strong reasoning LLM as a judge to handle open-ended video tasks resembling real-world entertainment use cases.

SAGE operates in two stages based on the role of the orchestrator (SAGE-MM):

• Stage-1 (Context VLM): In this single-step stage, SAGE-MM accepts system inputs (tools, sampled frames, query, metadata) and provides information about the video's context along with either a final answer prediction or a tool call to be executed before the next step.
• Stage-2 (Iterative Reasoner): In this multi-step stage, SAGE-MM uses the video context and tool call results from previous steps to decide either to predict the final answer or call another tool in an iterative reasoning process.

We introduce an easy synthetic data generation pipeline using Gemini-2.5-Flash to train the orchestrator.

• QnA Pairs: We leverage the long-context modeling abilities of Gemini-2.5-Flash to generate high-quality QnA pairs covering the entire video.
• Tool Call Trajectories: We use a SAGE system with Gemini-2.5-Flash as the orchestrator to synthesize tool call trajectories for a cold-start SFT stage.

To instill any-horizon reasoning, we employ Group Relative Policy Optimization (GRPO) for trajectory-level optimization. We utilize a multi-reward recipe that combines step-level rewards with a final accuracy reward.

Reward Function

Driven by the limitations of current video reasoning benchmarks (primarily their focus on Multiple Choice Questions), we curated SAGE-Bench. This evaluation set consists of 1,744 manually verified samples with an average video duration of over 700 seconds.

Key Features:

• Source: Videos from 13 popular YouTube channels covering diverse genres including Sports, Food, Comedy, Education, and Travel.
• Open-Ended Focus: SAGE-Bench utilizes open-ended questions with an unbounded answer space, better simulating real-world user interactions.
• Diagnostic & Practical: Includes both diagnostic questions to test fundamental capabilities and practical questions that a user might naturally ask.

We empirically validate the effectiveness of our system, comparing it against both DIRECT and AGENT baselines.

Key Findings:

• Overall Performance: SAGE achieves improvements of up to 6.1% on open-ended video reasoning tasks compared to base models.

• Long Video Reasoning: On videos longer than 10 minutes, SAGE shows an impressive 8.2% improvement.
• Effect of Tools: Incorporating Gemini-2.5-Flash as a tool (SAGE-Flash) further boosts gains to 14.6% on long videos, highlighting the system's ability to leverage stronger backend tools.