UT Austin Paper: Lightweight Scorer Speeds Up AI Memory Construction
A paper from UT Austin introduces Jev-Mem, which offloads micro-decisions in long-conversation memory systems from LLMs to a fast batch-scoring model. This approach accelerates memory construction by 6.6x on the LoCoMo benchmark while improving answer accuracy to 0.777, significantly reducing LLM invocation overhead.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-10-25 14
University of Texas paper finds that handing small memory decisions to a fast scoring model let an AI memory system build its memory of long conversations 6.6 times faster
Many memory systems ask an LLM to make small choices, like what to link or when to stop searching.
Generating text for each choice is slow, and those calls pile up during memory building and search.
Jev-Mem gives those choices to a fast scorer, Jev, that returns probabilities in batches instead of text. The LLM is called only at the end, to read the chosen memories and answer.
On the LoCoMo long-conversation benchmark, it got a 0.777 answer score versus 0.700 for the best baseline. It built memory in 158 seconds versus 1,044 for the fastest rival and answered in 0.93 seconds versus 1.47.
If your memory pipeline calls an LLM for routing, scoring, or stopping, try a fast classifier there and save the LLM for answers.
