Rohan Paul (@rohanpaul_ai) · 10/19/2026, 14:02:18
Amazon Study Reveals Blind Spots in Agent Memory Evaluation
A new Amazon paper reveals that end-to-end QA scores alone fail to accurately evaluate agent memory systems. Using the FEB-30 benchmark, the study found significant discrepancies between QA results and actual stored fact states, where strong LLMs can mask broken memory or retrieval misses penalize correct storage. Developers are advised to directly verify stored facts after each update alongside standard QA testing.
SOURCE COVERAGEFull Original Report
Rohan Paul (@rohanpaul_ai) · 2026-10-19 06
New Amazon paper shows that teams shipping agent memory should check what actually got stored, since QA scores alone did not reveal which pipeline stage was failing.
Agent memory benchmarks grade the agent's answers, which mixes memory with retrieval and answer writing. A strong LLM can hide a broken memory store, and a retrieval miss can make a correct one look bad.
They built FEB-30, 30 scripted fact histories turned into 147 documents, so the correct memory state is known after every document. On a memory system built like Mem0 and Hindsight, they compared the stored facts with that truth and with QA scores.
The 2 scores differed by 16.8 points on average, and pass/fail verdicts disagreed on 21% of fact histories. The system picked the right memory action, like update or reinforce, only 20.1% of the time, which QA never showed.
If you build agent memory, test the stored facts directly after each update, alongside end-to-end QA.
