Google AMIE 升级:实现专家级视听临床问诊
Google Research 宣布其医疗 AI 系统 AMIE 已从纯文本对话升级为具备视听感知能力的多模态系统,能够整合患者的步态、呼吸及面部表情等非语言线索。该系统在模拟临床咨询中达到了专家级诊断水平,旨在通过捕捉传统文本模型忽略的细微体征来提升诊断准确性与患者信任度。
报道正文
本文目录7 个章节
当医生与患者会面时,问诊远不止于言语交流。医生会观察患者的步态,留意明显的不适迹象,注意其呼吸状况,并引导患者进行体格检查操作。这些持续的视觉和听觉信息流与口述的临床病史无缝整合。这些非语言的视觉和听觉线索对于有效诊断、建立患者信任以及临床沟通至关重要。
具备临床推理和对话能力的 AI 系统有望大幅提升医疗专业知识和护理的可及性,从而推动实现一个让医生能够将时间专注于医患互动中最具意义方面的未来。在早期工作中,我们的研究 AI 系统——用于临床推理和对话的 Articulate Medical Intelligence Explorer(AMIE)——在基于文本的诊断对话中展现了专家级性能,并被证明是有效的临床医生鉴别诊断辅助工具。最近,我们将 AMIE 的能力从诊断拓展至长期疾病的治疗与管理。
我们还拓展了 AMIE 的能力,使其能够进行肿瘤学、心脏病学和眼科学领域的专家级评估,并在由患者演员参与的模拟环境中,实现针对图像和临床文档的多模态诊断推理。与此同时,我们已开始将这些研究成果转化为临床实践,通过建立以医生为中心的监管框架,并开展首批真实世界临床研究,其中包括与贝斯以色列女执事医疗中心合作进行的临床可行性研究,以及与 Included Health 合作开展的全国性随机对照研究。
尽管取得了这些进展,我们的研究仍面临一个根本性限制:基于文本的界面丢弃了临床实践中的视觉和听觉维度。患者必须将复杂的身体症状转化为文字描述,这一过程会丢失诊断信息,并可能对数字素养或健康素养有限的患者产生负面影响。纯文本系统无法独立观察为临床推理提供依据的视觉和听觉线索,也无法指导患者执行那些有助于形成鉴别诊断的体格检查动作。
今天,在文章《迈向专家级医疗 AI:面向实时视频问诊_》中,我们推出了 AMIE 的实时视频配置版本——AMIE (Video),旨在解决上述局限性。基于 Gemini 和 Project Astra,AMIE (Video) 能够进行同步的临床视频问诊,实时感知非语言临床线索,引导患者演员完成虚拟体格检查,并进行诊断推理。在一项包含 100 个场景、300 次实时问诊以及由 30 名委员会认证初级保健医师(PCPs)组成的多臂随机研究中,我们首次展示了 AI 系统在实时临床视频问诊中达到专家级水平的能力。
AMIE (Video):一种异步多智能体架构
通过视频进行有效的临床对话需要平衡相互竞争的需求:系统必须以自然的对话速度响应患者,同时执行细致的临床推理,并持续处理视觉和听觉流。目前,单一智能体无法满足所有这些要求。深度推理需要时间,但对话中的停顿会削弱患者的信任和融洽关系。
为应对这一挑战,AMIE (Video) 采用了一种异步多智能体架构,将工作分配给三个并行持续运行的专用智能体:
- Talker 智能体: 面向患者的智能体负责驱动响应迅速、低延迟的语音交互。它在保持自然对话流畅性的同时,整合来自其他智能体的指导。
- Planner 智能体: 该智能体在后台运行,持续优化系统的临床推理,更新鉴别诊断和管理计划,识别信息缺口,并重新排列多样化临床目标的优先级。
- Perception 智能体: 该智能体持续审查音频和视觉流,识别具有临床意义的非语言线索(如可见的痛苦迹象、体格检查结果或听觉信号),并将这些观察结果置于正在进行的对话语境中进行解读。
这种解耦设计使 AMIE(视频版)能够在执行诊断推理和视听感知的同时,保持自然的对话延迟,否则这些任务会引入不可接受的延迟。自动化评估证实,该三智能体架构中的每个智能体都对临床指标的提升做出了重要贡献,例如病史采集能力、临床推理和治疗建议,以及对话质量相关指标,包括以患者为中心的沟通技巧和响应延迟。
通过自动化评估指导开发
构建视听医疗 AI 的一个关键挑战在于大规模表征系统的感知和推理能力。为了指导开发,我们从医学文献中推导出了一个与远程医疗相关的临床视听能力分类体系,涵盖非语言视觉线索、听觉信号和体格检查操作。随后,我们基于这一分类体系构建了自动化评估套件。
该评估框架结合了针对性的单轮视听评估和多轮模拟音频问诊。单轮视听评估测试特定的临床感知和推理实例(例如,正确识别解剖学侧别或识别呼吸窘迫的迹象)。而多轮模拟音频问诊则评估端到端的对话性能,同时将视觉线索作为文本描述注入模拟环境中(例如,针对帕金森病场景的 AI 患者模拟器被提示展示其笔迹时,可能会注入一段口头描述“[向镜头举起纸张,显示字迹拥挤、细小]”)。这些互补的评估共同支持了系统设计的快速迭代,并在人工评估之前充分刻画了 AMIE(视频版)的能力特征和失败模式。
通过随机化视频研究进行评估
为了在更具挑战性且更真实的端到端视听临床问诊场景中评估临床胜任力,我们使用同步视频问诊界面进行了一项大规模的随机 客观结构化临床考试 (OSCE) 研究。
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
To cover a breadth of medical conditions in our evaluation, the study spanned 100 clinical scenarios covering five body systems, including cardiopulmonary, abdominal, head/eyes/ears/nose/throat (HEENT), neurological/psychiatric, and musculoskeletal conditions. Fifteen trained patient actors carried out 300 standardized consultations across three study arms:
- AMIE (Video): The video configuration of AMIE conducting real-time video consultations.
- AMIE (Text): A text-only version serving as a baseline to isolate the contribution of audio-visual capabilities.
- PCP (Video): Ten board-certified PCPs consulting via the same video interface.
An independent panel of 20 experienced primary care physicians evaluated all consultations using established clinical rubrics, including both general clinical competency scales and detailed case-specific scoring criteria tailored to each scenario.
Key results
Expert-level clinical performance: Across core clinical competencies, history-taking thoroughness, diagnostic accuracy, management appropriateness, and communication quality, clinical evaluators rated AMIE (Video) on par with PCPs. AMIE (Video) also matched or exceeded AMIE (Text) on these dimensions.
Strength in physical observation and examination: AMIE (Video) was rated significantly higher, on average, than both PCPs and AMIE (Text) eliciting physical signs and proactively guiding patient actors through virtual examination maneuvers. This advantage was also reflected in case-specific perception and examination rubric scores.
Patient actors preferred the video experience: Patient actors strongly preferred the synchronous video interface over text-based chat, rating it as significantly easier to use and more effective for communicating health concerns. They also rated AMIE (Video) favorably on empathy, rapport, and confidence in care compared to both PCPs and AMIE (Text).
Limitations and responsible development
This research has important limitations and it is critical to interpret these results within the context of these limitations. This study was conducted entirely with professional patient actors in simulated clinical settings, not with real patients presenting with their own health conditions. Patient actors, however skilled, cannot fully replicate the complexity and unpredictability of real clinical encounters, and the scenarios were limited to conditions that can be authentically portrayed through acting, omitting important clinical presentations where audio-visual perception would be diagnostically consequential. Beyond the scope of the study, targeted automated evaluations revealed occasional perceptual and reasoning errors, despite overall high-quality conversation and diagnostic accuracy, and the system still exhibits intermittent technical issues that can disrupt conversational naturalness. Given the prototype nature of Project Astra, this includes technical considerations that future development may address at a system level that go beyond the specific medical application explored in this work. Assessing these findings in studies with real patients and real clinical conditions is an essential next step before any conclusions about real-world utility can be drawn.
Looking ahead
This work demonstrates that the transition from text-based to audio-visual clinical AI is achievable at expert-level quality. AMIE (Video) engages with the perceptual richness of clinical practice, observing non-verbal cues, guiding physical examination, and conversing naturally through spoken dialogue — capabilities that more closely approximate the experience of a telehealth video encounter.
Important questions remain on the path towards responsible real-world evidence. Our findings need to be validated with real patients, expanded to encompass clinical presentations that cannot be enacted, and supported by robust safety frameworks. We have already taken early steps in this direction: a real-world feasibility study with Beth Israel Deaconess Medical Center provided initial evidence for the safety and utility of text-based AMIE in clinical practice, and our ongoing nationwide randomized study with Included Health is further evaluating AI in real-world virtual care. Together, these research experiences will help inform how audio-visual capabilities might be responsibly integrated into clinical practice. While much remains to be done, these results mark an important milestone towards AI systems that could one day augment care by engaging with the sensory complexity of clinical practice.
Acknowledgements
The research described here is joint work across many teams at Google Research and Google Deepmind. We are grateful to all our co-authors - Mahvish Nagda, Jihyeon Lee, Matthew Thompson, CJ Park, Tim Strother, Valentin Liévin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fukuzawa, Sam Schmidgall, Craig Schiff, Joseph Xu, Aliya Rysbek, Yana Lunts, Jan Freyberg, Rebecca Hemenway, Sunny Virmani, David Racz, Carey Radebaugh, Joëlle Barral, Kavi Goel, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Gregory Wayne, Tao Tu, Yun Liu, Ethan Goh, Christina Chen, Ryutaro Tanno, and Cameron Chen.