谷歌提出长视频生成新框架,解决语义漂移难题
针对现有智能体管线在长视频生成中面临的语义漂移与级联失效问题,谷歌团队提出了一种新的自动化框架。该研究通过优化提示词结构与错误追溯机制,显著提升了跨镜头的角色一致性与叙事连贯性,减少了人工干预需求。
报道正文
本文目录8 个章节
近期视频扩散模型的进展展示了令人瞩目的高保真生成能力,相关模型能够在数秒内渲染出逼真的场景。然而,尽管扩散模型能够生成高保真的视频片段,但将其转化为连贯的长篇叙事引擎仍具挑战性。
现有的大多数智能体(agentic)流水线通过链式模块来自动化这一过程,但由于依赖独立且手工设计的提示词,往往面临语义漂移(角色着装或场景在不同镜头间发生细微变化)和级联失败(例如上游资产中的瑕疵导致下游视频合成出错)等问题。由于早期错误会不断传播并破坏长时程一致性,该过程通常需要大量的人工干预。从结构角度来看,这反映了经典的信用分配问题,因为终端故障难以追溯至具体的提示词。此外,现有方法还受困于_特征漂移_(实体和环境逐渐发生非预期变化)或_内容坍缩_(叙事无法有意义地推进)。
今天,我们推出了一项关于 AI 视频联合导演(AI video co-director)的研究,这是一个统一的、多智能体框架,旨在显式规划多镜头叙事中的视觉连续性。该框架构建在 Gemini 和 Veo 之上作为编排层,原生继承了诸如 SynthID 水印 等安全机制。通过将长格式生成视为全局优化和世界状态跟踪问题,我们开发了一系列框架——包括 Co-Director(将在 COLM 2026 发表)、CANVAS(将在 EMNLP 2026 发表)、A²RD 以及 VQQA——这些框架将高层的人类创意规范转化为执行指令,自动化了从多模型提示、镜头串联到闭环视觉优化的重复性编排任务。
我们设计这些框架旨在充当响应式的创意伙伴,将维持视觉连续性的负担抽象化,从而让用户能够专注于叙事艺术本身。该架构通过将质量建模为推理时目标(test-time objective),实现了创意合成与一致性之间的解耦。在全面评估中,我们的框架在多镜头叙事一致性和角色持久性方面展现出显著提升,成功生成了长达数分钟的视频,同时有效缓解了视觉漂移和流水线误差传播问题。
工作原理
为解决长时序视频这一多维度的难题,我们将研究分解为四大基础支柱,每个支柱分别针对生成流水线中的特定瓶颈。
1. 通过 AI 视频联合导演编排创意意图
为确保整个视频的语义连贯性,我们推出了 AI 视频联合导演,这是一个分层多智能体框架,它将视频叙事形式化为一个全局优化问题。我们不依赖僵化的线性提示链,而是引入了分层参数化机制:利用 多臂老虎机 (MAB) 算法在全局范围内识别具有潜力的创意方向。
这将创意过程形式化为在探索新颖叙事策略与利用高效创意配置之间寻找最佳平衡点的搜索过程。系统采样抽象的创意轨迹——例如结合信息型策略、小品式叙事模式和特定的美学原型——并将这些轨迹动态注入子智能体的系统提示词中。这种自上而下的引导确保了整个流水线在统一的愿景下运行。由于我们的 AI 视频联合导演框架作为编排层运作,它通过将结构化提示词直接输入到底层的 Gemini 和 Veo 模型来实现这一愿景(尽管其模型无关的架构允许它叠加在任何基础生成模型之上)。该架构确保所有生成的图像、视频和音频天然具备安全保护机制,包括 SynthID 水印技术。在生产环境中,还可以对最终视频应用额外的安全分类器,以防止单个安全片段之间出现意外的上下文交互。
该流水线通过两个相互关联的循环执行全局优化:策略引导与多阶段生产。首先,编排器 Agent(Orchestrator Agent)利用 MAB 算法评估输入,并在三个维度上选择创意配置:(1) 创意策略(意图)、(2) 叙事模式(故事结构)以及 (3) 美学原型(视觉基调与摄影风格)。该配置驱动整个生产层级。预制作 Agent(Pre-Production Agent)将逐场景的故事线概要和视觉资产综合为统一的分镜脚本。随后,制作 Agent(Production Agent)利用专门的子 Agent 将该分镜脚本转化为具体的视听媒体:关键帧 Agent(Keyframe Agent)锚定角色和场景视觉,视频 Agent(Video Agent)添加运动效果,音频 Agent(Audio Agent)则叠加匹配的旁白和配乐。
最后,一个多模态 LLM(MLLM)评审员会在指定的三个维度上对剪辑完成的成片进行批评性评估,并将分解后的奖励信号反馈给 MAB,从而在连续的生成循环中迭代优化选择。
2. 基于 CANVAS 的视觉分镜
即使拥有统一的剧本,生成长序列镜头往往会导致角色漂移和环境不稳定。为解决这一问题,我们引入了 Continuity-Aware Narratives via Visual Agentic Storyboarding(CANVAS),这是一个在多镜头叙事中显式规划视觉连续性的多智能体框架。
CANVAS 通过在叙事演进过程中维护角色、地点和物体状态的结构化表示来强制保持一致性。作为构建在 Gemini 之上的编排层,它依赖于持久化的视觉记忆。通过从记忆中检索视觉锚点或在需要时初始化新的锚点,CANVAS 确保了同一场景内的平滑过渡。这种显式的世界状态建模保证了角色在再次出现时保留其身份特征,同时环境也保持其空间结构的稳定性。
要直观了解其实际效果,下图展示了一段多镜头博物馆盗窃场景序列,并将 CANVAS 与代表性基线方法进行了对比:直接使用底层基础模型(Gemini-3.1-Pro)进行生成,以及另一种多智能体框架(AutoStudio)。图底部的提示词突出了需要保持视觉特征一致性的重复元素——例如窃贼、展厅和宝石。请注意各方法如何处理连续转场(如角色服装的变化)与非连续转场(如镜头在绕行后回到主厅)。仅使用 Gemini-3.1-Pro 生成时会出现道具不一致(文物发生变化)和背景漂移(房间布局改变),这显示了无引导生成的局限性。AutoStudio 在镜头切换过程中也会出现质量下降,导致角色漂移(窃贼的帽子消失)和背景不一致。相比之下,CANVAS 的持久视觉记忆确保了角色、空间几何结构和物体状态在整个叙事过程中始终保持完美连贯。
3. 通过 A²RD 扩展时间动态
为了将故事板转化为长达数分钟的实际视频,我们开发了 A²RD,这是一种基于智能体的自回归视频生成架构。A²RD 采用逐段生成方式,并辅以多模态视频记忆来追踪片段上下文和动态变化。
对于每个片段,它在“检索-合成-优化-更新”循环中运行。该循环的一个关键部分在于智能体如何自适应地确定片段生成模式。它在外推(extrapolation)和内插(interpolation)之间平滑切换:外推允许自然的叙事推进,而内插则将片段锚定到现有的实体和环境中。这有效地平衡了故事向前发展的需求与维持场景物理现实的需要。
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
To test this system, the ten-minute movie below showcases a long-form generation, which demands consistent narrative progression across minutes-long temporal gaps. The video highlights how the system dynamically shifts between its two operational modes: using extrapolation to push the plot into new narrative beats, and interpolation to anchor returning characters and environments to their original designs. While standard video generators suffer from severe visual decay — where characters mutate and locations morph — A²RD continuously queries its multimodal video memory to maintain character identity, costume details, and structural geometry from the opening shot to the final frame.
4. Closed-loop refinement with VQQA
Finally, we needed a way for the system to autonomously identify and fix visual artifacts. Existing test-time optimization methods are typically either computationally expensive orrequire white-box access to model internals. To address this, we developed Video Quality Question Answering (VQQA), a unified, multi-agent framework generalizable across diverse input modalities and video generation tasks.
VQQA dynamically generates visual questions tailored to the specific prompt and uses the resulting Vision-Language Model (VLM) critiques as semantic gradients (provides natural language directional feedback to guide iterative refinement, analogous to numerical gradients in backpropagation). This replaces traditional, passive evaluation metrics with human-interpretable, actionable feedback. The system can then execute a highly efficient, iterative feedback loop (where the model generates a video, evaluates it via visual questions, and refines the text prompt based on the critique) via a natural language interface. To prevent semantic drift during this refinement, VQQA employs a Global Selection mechanism: rather than blindly taking the final iteration's output, a global VLM rater evaluates every video generated across the optimization trajectory against the original, unedited prompt. The system then selects the highest-scoring candidate, ensuring localized corrections do not compromise the broader context.
In practice, VQQA operates as a black-box prompt optimizer rather than a pixel-level editor. Instead of modifying pixels directly, it iteratively refines the text prompt to correct high-level compositional defects like attribute binding errors or inconsistent character attributes. This updated prompt guides the generator to sample a new path in its latent space. In the examples below, VQQA does not mask or paint over the original frames; rather, it resolves the model’s material binding struggle by rendering a realistic mylar balloon texture onto a strict cuboid geometry, and fixes a chaotic mid-performance instrument change by keeping the violinist and pianist consistently anchored to their respective instruments across cuts.
Key results & benchmarks
To rigorously evaluate our frameworks, we developed three specialized benchmarks designed to mirror the challenges of professional video production.
- GenAD-Bench: Uses a human-in-the-loop pipeline that pairs Gemini 3 Pro with specialized image models to construct 50 fictional brands containing four distinct products each. Across 400 unique scenarios, it tests exact marketing constraints without relying on copyright conflicts or training priors.
- HardContinuityBench: Designed to evaluate spatial and environmental continuity. Generated using intricate GPT-5.2 multi-shot narrative storyboards, this benchmark challenges models with massive gaps between scene reappearances, frequent costume and accessory changes for actors, and complex state changes for interactive props.
- LVBench-C: Addresses long-horizon temporal dynamics. This dataset contains 120 text-only scenarios grouped into evolving character states, changing object properties, and progressive environmental revelations, maintaining a strict gap rule where critical visual assets must disappear for at least 10 segments before returning with realistic, narrative-driven changes.
Our evaluations demonstrate measurable performance improvements over existing video generation architectures. By navigating the creative strategy search space, AI video co-director achieves a peak quality score of 81.4 on GenAD-Bench and enhanced story consistency on ViStoryBench. CANVAS leverages structured visual memory to mitigate scene drift, yielding significant continuity gains across scene reappearances on ST-Bench and HardContinuityBench. For long-duration temporal dynamics, A²RD minimizes layout drift to improve character and environment consistency over continuous multi-minute runs on VBench-Long and LVBench-C. Finally, VQQA's closed-loop prompt optimization resolves physical and compositional inconsistencies, delivering notable absolute quality gains across T2V-CompBench, VBench2, and VBench-I2V . Please consult the individual papers for comprehensive details on our model architectures, training configurations, and baseline evaluations.
Future directions
These frameworks represent a foundational step toward unlocking coherent, long-horizon visual storytelling for creators. As we continue to refine these agentic architectures, we are exploring how to integrate human-in-the-loop workflows. Our ultimate goal is not to replace human storytelling but to empower creators by abstracting away the tedious complexities of temporal consistency and world-state tracking, ensuring they remain the control of creative direction and narrative design.
Acknowledgements
This work was made possible by the dedicated efforts of our broader research teams across Google. We would like to thank Andrew Pan, Brett Slatkin, Burak Gokturk, Carina Claassen, Daniel Vlasic, Do Xuan Long, Ishani Mondal, Jasmine Leon, Jingyun Liu, Joe Timmons, Jordan Boyd-Graber, Khanh G. LeViet, Kuang Su, Long T Le, Mihir Parmar, Min-Yen Kan, Nathan Hodson, Nick Losier, Palash Goyal, Rhyard Zhu, Sebastian Ko, Scott Penberthy, Yan Xu, Yang Li, Ye Jin, Zack Chomyn, and Tomas Pfister for their invaluable contributions to this suite of research.