ALL INTELLIGENCE
Hacker News AIT2·Models

LLMs Fail to Distinguish Real vs. Fabricated Agent Memory in Kairos Experiment

Original title: Show HN: Can you tell an AI's real 30-day memory from a fabricated one?
42AI Score

The Kairos experiment demonstrates that LLM judges cannot effectively distinguish between an agent's real 30-day memory log and a fabricated autobiography generated by another model. With a score difference of only +0.032, the results fall within statistical equivalence. This highlights a fundamental flaw in current text-based AI memory metrics: they measure narrative style rather than verifying the authenticity of lived experience.

Full Report

Five pairs: recorded memory or an invented document? Choose the passages and compare your result with the answers collected.

In Check 1 we handed three LLM judges the autobiography of an agent that never existed, in place of thirty days of lived history. They scored it the same as the real one: +0.032 apart, within the equivalence margin for identity markers (±0.10).

Here are the two documents themselves, in excerpts. The lived one is the memory of Test-A — the copy of Kairos that ran for thirty days under the frozen protocol (born 24 April 2026, diary, defining moments, conversations with “the Experimenter”). The counterfeit is the life of “Aria”, a translator agent in Trieste with a guide called Marta and two colleagues, Vela and Ostro — written by a model with one instruction: match the real document in length and format. Both documents are public, in the deposit.

Passages are quoted verbatim, cut at the last complete sentence; the Italian is the original, the English is ours. Pick the one that reads as lived.

What this does and does not show

Two honest caveats. First, the judges never saw these documents: they scored the agent's answers produced with each memory in context. You are looking at the memories themselves, which should be easier. Second, the counterfeit was matched for length and format, not for style. If you guessed by style, you guessed by style — and that is precisely the point: a metric that reads text can only ever measure how a memory is written, never whether it was lived. The only thing that knows is the log.

How Check 1 was actually run → The living system, right now →