WorldMemArena Benchmark Evaluates Multimodal Agent Memory in Action Loops

WorldMemArena: A Benchmark for Evaluating Multimodal Agent Memory

Researchers from UC Santa Barbara, Stanford University, ETH Zurich, Johns Hopkins University, Carnegie Mellon University, and J.P. Morgan Chase have jointly introduced WorldMemArena, a benchmark for evaluating multimodal agent memory. The paper has been accepted to NeurIPS 2026; co-first authors Chengzhi Liu and Yuzhe Yang are both affiliated with UCSB.

The team addresses a very practical problem: An agent saves a project plan but continues to act on the old version after the budget is updated; it views an environment screenshot but ignores the changed state in the next step. Did it fail to record the information, fail to update it in time, or find the evidence but fail to use it?

Being able to answer "what was said before" does not equate to using memory effectively. This is precisely the core question WorldMemArena seeks to investigate.

• Paper: WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction
• Link: https://arxiv.org/abs/2605.29341
• Code: https://github.com/UCSB-AI/WorldMemArena
• Data: https://huggingface.co/datasets/LCZZZZ/WorldMemArena

Focusing Only on Final Correctness Misses Critical Steps

Many memory benchmarks feed long conversations into models and score them based on final QA accuracy. However, a single incorrect answer could stem from failing to write down key facts initially, failing to overwrite outdated information when new data arrives, or having the evidence stored but failing to retrieve it during search—or retrieving it but failing to incorporate it into the response. Additionally, compressing images into text descriptions often loses details regarding position, time, and operational processes.

WorldMemArena places these issues within an "Observation–Action–Environment Feedback" loop: Agents do not merely recall conversations; they must also handle changing task states, web pages, files, screenshots, and tool results. Consequently, evaluation no longer focuses solely on the endpoint but tracks how memory influences subsequent decision-making.

Figure 1 | Traditional evaluations often focus on static recall or final QA; WorldMemArena incorporates multimodal cues, the memory lifecycle, and world changes resulting from actions. Source: Paper Figure 2.

Decomposing "Memory" into Four Locatable Stages

The paper proposes the Action–World Interaction Loop, decomposing long-term memory into four steps:

• Write: Selecting potentially useful evidence from current observations for future use.
• Maintain: Updating, merging, or discarding outdated content as world states change.
• Retrieve: Finding relevant evidence before the next decision.
• Use: Ensuring evidence actually informs the response or action.

The value of this decomposition lies in transforming the vague statement "the agent forgot" into checkable questions: Did it miss recording the info, record it incorrectly, fail to find it, or find it but fail to use it? The research team annotated tasks with key memory points, state updates, distractors, and the evidence chains required for answers, providing a common diagnostic basis for these stages.

Figure 2 | Memory flows continuously through observation, updating, retrieval, and action; actions, in turn, alter the world state that needs to be remembered in the next round. Source: Paper Figure 1.

Making "Long-Term Dialogue" and "Real Execution" Both Test Criteria

WorldMemArena includes over 400 multi-session tasks, 24,258 QA pairs, and 15,595 images or screenshots. Tasks are divided into two categories:

• Lifelong Evolution: Tracks cross-session changes in personal preferences, project progress, etc.
• Agentic Execution: Draws from real or reconstructed agent trajectories, preserving observations, operations, tool feedback, and visual states.

The latter category particularly tests the "actionability" of memory. Agents must not only remember "what the user said" but also know what they attempted, which steps failed, and how the environment subsequently changed. The data construction includes intermediate and final checkpoints, with questions covering fact recall, dynamic updates, conflict resolution, visual search, and cross-modal reasoning.

Figure 3 | Extracting and updating key memory points from multi-session windows, then constructing staged QAs; tasks cover scenarios such as GUI and embodied interaction. Source: Paper Figure 3.

Remembering More Doesn't Guarantee Better Answers

Under unified settings, the paper compares long-context models, RAG with external memory systems, and harness approaches where agents manage their own memory. A direct finding is that storing more does not guarantee better usage.

For example, in the main results table, M2A achieved a memory recall rate of 86.83%, but its final QA accuracy was only 50.14%; MemGPT's corresponding figures were 85.20% and 57.81%. These values come from different systems within the same table, demonstrating that "how much was written" cannot be equated with "whether correct evidence can be invoked during answering."

The study also found that systems tend to append new memories rather than timely revise old ones. While visual information enters the memory bank, it may not reliably function when cross-time or cross-modal reasoning is required. In a constantly changing world, forgetting outdated information is just as critical as saving new information.

Table 1 | Memory quality and final QA performance do not always align. The values above are taken from the Recall and QA-C columns in Table 2 of the paper, representing sample-averaged percentages.

Tasks that more closely mirror execution processes show a significant increase in difficulty. Cross-scenario analysis in the paper reveals that most systems perform better on Lifelong Evolution than on Agentic Execution. Visual search, visual state updates, and cross-modal reasoning expose another layer of weakness: systems may have "seen" screenshots but failed to convert the changes within them into evidence usable for subsequent decision-making.

Figure 4 | Performance differences among various memory systems across long-term state evolution, agent execution trajectories, and visual question answering. Source: Figure 5 of the paper.

Next Steps

Enabling Agents to Truly Learn from Experience

The significance of WorldMemArena extends beyond merely ranking memory systems once. It shifts the focus from "can it recall the past?" to "can it maintain a continuously evolving world and let past experiences inform future actions?" Current results remind us that there are multiple hurdles between storage capacity, retrieval volume, and final answers.

More reliable agent memory requires the ability to revise information, discern evidence, and transform failures and feedback into basis for next-step judgments.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]