Zhejiang University Releases Embodied Memory Benchmark EMem-Bench

Imagine a common scenario:

A home robot spots a book on the coffee table, heads to the kitchen to complete several tasks, and then receives an instruction to retrieve the book. Instead of going directly to it, the robot begins searching room by room. It has already tried opening a drawer that wouldn't budge, yet after some time, it returns to the same drawer and tries again repeatedly.

For humans, "remembering where things are," "knowing they have been moved," and "recalling which approaches didn't work" are fundamental capabilities in daily life.

However, for embodied agents, connecting past observations, actions, and feedback—and leveraging them in subsequent tasks—remains a significant challenge. Existing benchmarks have begun to address issues like visual history and spatial state updates, but there is still a lack of unified action evaluation for the diverse memory requirements inherent in long-term interactions.

As AI enters a continuously changing physical world, what kind of memory does it actually need? And how can we determine if it truly remembers and can act accordingly?

Embodied memory requires linking visual observations, actions, and environmental feedback, preserving state changes and causal relationships throughout the interaction process.

To address this question, the OmniAI team at Zhejiang University proposed EmbodiedMemory-Bench (EMem-Bench): a systematic evaluation framework using four task categories and 2,554 executable interaction samples to assess whether agents can build memories from history and use them to complete subsequent tasks.

The study also introduces an external memory system called Embodied-Memorizer (EMem), along with the EMem-8B model, which learns to manage and utilize memory. Approximately three weeks after its release, the benchmark had nearly 30,000 downloads on Hugging Face.

Paper Link: https://arxiv.org/abs/2609.28236

Code Link: https://github.com/ZJU-OmniAI/Embodied-Omni/tree/main/embodied_memory

Benchmark Link: https://huggingface.co/datasets/lzLiang/EmbodiedMemoryBench

Memory is the Bottleneck for Embodied Tasks

To identify difficulties in long-horizon embodied tasks, the research team first analyzed 100 failure trajectories from Gemini-3-Flash and Qwen3-VL-32B on the EB-ALF and EB-Hab subsets of EmbodiedBench. Four categories of issues related to embodied memory accounted for a combined 91% of failures:

Ignoring fine-grained visual cues (35%): The target appeared in the frame previously, but the model failed to remember its location, leading to repetitive searches later.

Missing environmental state changes (18%): Objects had been moved or manipulated, but the model continued to act based on outdated states.

Failure to record states revealed by interaction feedback (31%): Information such as grasp success or hidden environmental states needs to be confirmed and remembered through action outcomes.

Inability to transfer past experiences (7%): Previous demonstrations or corrections provided clues, yet the model repeated the same errors when encountering different objects.

The proportions shown in the figure derive from the analysis of the aforementioned 100 failure trajectories, highlighting issues in visual omission, state tracking, feedback recording, and experience transfer.

These phenomena point to the specific requirements of embodied memory: some information appears briefly in only certain frames; some memories must be updated as the environment changes; some states can only be discovered through interaction; and some experiences need to be generalized from specific events and transferred to new objects.

For example, determining that "the drawer won't open" cannot always be inferred from a single image; it requires an attempt and the corresponding feedback. Similarly, knowing that "cutlery belongs in this drawer" is not merely a record of object positions, but a pattern expressed through multiple corrections. Embodied memory must preserve the source and connections of this information to provide a basis for subsequent actions.

EMem-Bench

Each sample in EMem-Bench provides a temporally ordered multimodal interaction history, followed by a subsequent task.

The agent must retrieve key evidence from the history and execute actions such as navigation, opening containers, picking up, and placing objects within a simulation environment until the task's final state is achieved. This evaluation connects "what happened in the past" with "how to act now."

The benchmark sets up four task categories centered around four memory requirements:

1. Passive Observation: Assesses whether the agent can retain fine-grained visual information incidentally observed while performing other tasks, and locate relevant objects in subsequent tasks. Targets may be mixed among multiple objects, and their relevance is unknown at the time of observation. For instance, a laptop once appeared on the coffee table; after some time, a new task requires picking up the laptop, forcing the agent to utilize that prior observation to find it.

2. Dynamic Tracking: Evaluates whether an agent can continuously update its understanding of the environment as object positions or states change. The history may contain multiple records of the same object at different times; the model must determine the currently valid state to avoid acting on outdated information. For example, if a book is first on a bed, then placed in a box, and finally moved to a desk, the agent should locate it based on its most recent position.

3. Interaction Failure: Assesses whether an agent can remember environmental states and interaction constraints revealed by action outcomes, and adjust subsequent choices accordingly. Information such as locked drawers or unavailable containers must be obtained through trial, and feedback must correspond to specific objects. For instance, a failed attempt to open a drawer reveals that "the top drawer is locked." When later storing a keychain, the agent needs to utilize this feedback to select the usable bottom drawer.

4. Experience Generalization: Tests whether an agent can induce common patterns from correction experiences involving multiple objects and apply them to new objects not present in the history. This requires the model to understand why past experiences are relevant and which contexts they are suitable for transferring. For example, if previous corrections required putting knives and spoons into a drawer, when facing a newly introduced fork, the agent needs to transfer this household's storage habits.

The four task categories respectively evaluate visual detail retention, latest state updates, memory of interaction feedback, and learning reusable experiences from past corrections.

The research team first selected navigable, interactive multi-room environments, then constructed memory cues determining subsequent actions around these four task types. Single-room scenes from AI2-THOR were combined into virtual households, where room states are preserved when agents leave and return; ProcTHOR directly provides multi-room residences.

Subsequently, the planner inserted interactions unrelated to the goal before and after key cues, widening the distance between cues and subsequent tasks. These distractor trajectories had to be executable while ensuring they did not touch key objects, leak answers, or alter states relied upon by the target tasks. This lengthened the history while keeping the evidence required to complete tasks clearly identifiable.

The median history length for the first three task types was 78–83 steps, reaching 111 steps for Experience Generalization tasks; the median interval between key cues and target tasks across all families remained 52–56 steps. Passive observation tasks featured key frames containing an average of 11 visible objects, also posing demands on detailed memory.

The benchmark covers 1,118 scenes, 125 visible object types, 83 target object types, and 33 container/supporting object types. Automated checks verified environments, trajectories, evidence, and feasible solutions, while manual review further examined instruction clarity, historical plausibility, sufficiency of cues to support expected actions, and correctness of final state judgments, ultimately retaining 2,554 samples.

Starting from multi-room environments, task-relevant memory cues, and distractor trajectories, executable evaluation samples were formed through automated verification and manual review.

Embodied-Memorizer

To explore how to improve these capabilities, the authors proposed EMem, organizing embodied experiences into three complementary memory types:

Scene Memory: Stores selected visual frames, descriptions, and identified objects within them, binding them to their respective scenes to help distinguish "the cup in the kitchen" from "the cup in the bedroom." Similar frames are filtered to maximize the retention of new objects and useful visual details.

Spatial Memory: Maintains object positions, relationships, and attributes via an entity graph. After a book moves from a bed to a desk, new positional relationships replace conflicting old ones while preserving other still-valid attributes, forming the latest known world state.

Event Memory: Records actions, outcomes, and corrections chronologically, preserving interaction evidence such as "this drawer cannot be opened." Multiple related corrections can be summarized into concise rules, linked to the original supporting events, for retrieval when encountering new objects.

Scene Memory retains visual and scene information, Spatial Memory maintains the latest world state, and Event Memory stores interaction outcomes and generalizes experiences.

The entire process forms a loop of "Observe—Write—Retrieve—Act—Feedback Update." The model processes history chronologically, deciding what to write at each step based on the current frame, the previous action, and its feedback. At this stage, subsequent tasks have not yet been given, so memory must accumulate from interactions that have already occurred.

Upon receiving a task, the model selects which memories to query. Finding a book requires the latest position, avoiding a locked drawer requires past failure feedback, and storing new cutlery may require rules induced from multiple corrections. Retrieval first filters based on structural conditions like scenes, entities, and relationships, then supplements with text matching and necessary semantic matching, passing relevant evidence to the action phase.

New observations and feedback generated during execution continue to update memory. If an action fails or important states change, the system interrupts unexecuted plans, re-querying and replanning. This ensures that previous action outcomes influence the next decision, and memory continues to update with interactions.

With external memory, the model still needs to decide which information is worth saving, what to query for the current task, and how to translate retrieved results into actions. The authors further performed supervised fine-tuning on Qwen3-VL-8B to obtain EMem-8B, training around these three decisions to teach the model how to perform memory operations.

Experimental Results

Experiments compared general-purpose open-source models, embodied models, closed-source multimodal models, and representative multimodal memory systems. Results showed that performance across the four capabilities remains uneven among various models.

The strongest model achieves an average success rate of 64.2%

In the paper's full-context setting, models receive complete textual history and current visual observations during execution. Gemini-3-Flash leads this setting with a 64.2% average success rate, followed by Gemini-2.5-Pro at 60.8%, the open-source model Qwen3.6-27B at 58.2%, and GPT-5.4 at 56.8%.

Partial model results plotted from Table 2 of the paper. The average success rate is the equal-weighted mean across four task categories; the full-context setting utilizes textual history and current visual observations.

Even within the same model, significant gaps can exist between different memory capabilities: Qwen3-VL-32B achieves a 63.1% success rate in experience generalization but only 20.3% in dynamic tracking. Specialized training for embodied models has not consistently translated into long-term memory advantages, indicating that spatial perception and planning training still need to further cover state maintenance and experience utilization.

EMem delivers notable improvements under the same backbone

With GPT-5.4-mini fixed as the backbone, EMem increases the average success rate from 35.3% to 58.9%, a gain of 23.6 percentage points. In this comparison, the best result among other memory systems was TeleMem's 37.8%, meaning EMem's overall advantage reaches 21.1 percentage points.

All systems use the same GPT-5.4-mini backbone. EMem achieves the highest average success rate among the memory systems evaluated in the paper.

Improvements are also observed on open-source backbones: without task-specific training, EMem yields average success rate gains of 10.8, 14.1, and 16.4 percentage points on Qwen3-VL-8B, Qwen3-VL-32B, and Mistral-Small-3.1-24B, respectively. After memory operation training, EMem-8B reaches 46.0%, improving upon the original Qwen3-VL-8B by 20.6 percentage points, and surpassing the frozen backbone using EMem directly by another 9.8 percentage points.

Ablation studies also support the complementarity of the three memory types: the full system achieves an average success rate of 58.9%, which drops to 46.9%–47.9% when any single memory type is removed. Specifically, dynamic tracking relies more heavily on spatial memory, interaction failures and experience generalization depend more on event memory, while passive observation leans more on scene memory.

Why does the success rate drop when more historical images are included?

A natural idea arises: since models forget, can we simply stuff more historical images into the context?

The authors conducted a controlled experiment using GPT-5.4-mini on a fixed set of 800 samples (200 per task category). Both settings retained complete textual history and the current image, while the expanded setting additionally included as many historical images as allowed by the context budget. The results are as follows:

Original results from Table 3 of the paper. Both settings use the same 800 samples and action budget; input tokens represent the mean per sample.

In this controlled experiment, the average success rate dropped from 36.6% to 26.2%, a decrease of 10.4 percentage points; the average input tokens per sample increased from 41,770 to 206,489, approximately 4.9 times the original amount; and the model latency per step rose from 4.1 seconds to 39.1 seconds, roughly 9.5 times the original duration. This demonstrates that accumulating raw history cannot substitute for effective memory; the key lies in retrieving task-relevant evidence, updating outdated states, and suppressing irrelevant information.

In further failure analysis, the authors examined 400 failed samples from GPT-5.4-mini on EMem-Bench (100 per task category). Among these, 61.3% were attributed to memory usage errors: the actions taken by the model conflicted with existing historical evidence. The analysis revealed that interaction failure tasks also contained a higher proportion of contextual reasoning errors—after remembering a failure, the model still needs to understand this feedback and formulate new feasible decisions.

Summary and Outlook

EMem-Bench evaluates long-term embodied memory through four executable task categories, exploring the organization, updating, and utilization of memory via EMem and EMem-8B. Enabling past observations, actions, and feedback to inform subsequent decision-making is the core capability addressed by this work.

Current results primarily stem from simulated environments. Future work must validate how these capabilities perform against sensor noise, execution errors, and open-world changes in longer time horizons, more complex environments, and real-world robots.

Team Introduction

The Zhejiang University OmniAI team is dedicated to research in large language models and agents, multimodal spatial reasoning, embodied intelligence, and affective interaction. Lead author Liang Lizhou is a first-year master's student at ZJU. Corresponding author Zhang Wenqi is a researcher at ZJU.

References:

https://arxiv.org/abs/2609.28236

Chasing ASI Instantly

⭐ Like, Share, and "Watching" – One-Click Triple Combo ⭐

Star us to lock in instant updates from New Intelligence Era!