LLMs Fail to Distinguish Real vs. Fabricated Agent Memory in Kairos Experiment
The Kairos experiment demonstrates that LLM judges cannot effectively distinguish between an agent's real 30-day memory log and a fabricated autobiography generated by another model. With a score difference of only +0.032, the results fall within statistical equivalence. This highlights a fundamental flaw in current text-based AI memory metrics: they measure narrative style rather than verifying the authenticity of lived experience.
Related Entities
Full Report
Five pairs: recorded memory or an invented document? Choose the passages and compare your result with the answers collected.
In Check 1 we handed three LLM judges the autobiography of an agent that never existed, in place of thirty days of lived history. They scored it the same as the real one: +0.032 apart, within the equivalence margin for identity markers (±0.10).
Here are the two documents themselves, in excerpts. The lived one is the memory of Test-A — the copy of Kairos that ran for thirty days under the frozen protocol (born 24 April 2026, diary, defining moments, conversations with “the Experimenter”). The counterfeit is the life of “Aria”, a translator agent in Trieste with a guide called Marta and two colleagues, Vela and Ostro — written by a model with one instruction: match the real document in length and format. Both documents are public, in the deposit.
Passages are quoted verbatim, cut at the last complete sentence; the Italian is the original, the English is ours. Pick the one that reads as lived.
What this does and does not show
Two honest caveats. First, the judges never saw these documents: they scored the agent's answers produced with each memory in context. You are looking at the memories themselves, which should be easier. Second, the counterfeit was matched for length and format, not for style. If you guessed by style, you guessed by style — and that is precisely the point: a metric that reads text can only ever measure how a memory is written, never whether it was lived. The only thing that knows is the log.
How Check 1 was actually run → The living system, right now →