Tencent Yuanbao and KCL Release Group Chat Agent Memory Benchmark

Original title: 8万条群聊消息做成考卷:腾讯元宝×KCL发布首个基于群聊构建的Agent记忆基准
78AI Score

Tencent Yuanbao and King's College London's NLP team jointly released GroupAssistBench, built from 50 real group chats and 80,299 historical messages into 2,395 evaluation tasks, splitting agent memory into L1 factual recall and L2 open-ended service. Across 14 configurations, stronger recall did not imply better service: EverMemOS scored 0.621 on L1 versus 0.570 for full context, yet only 0.247 on L2 versus 0.431. The paper, code and dataset are all public.

Full Report

Contents13 sections

AI remembering is not the same as using it well

Contributed by Tencent Yuanbao × King's College London NLP Team

QbitAI | WeChat Official Account QbitAI

A nine-person travel group initially agreed to depart on May 1, but later changed it to April 30. When someone asks an AI assistant to send a ticket-purchase reminder, it will likely still notify based on the old date.

Although it "remembered" the history, it did not truly serve this group well.

In real group chat scenarios, there are often more subtle requirements: who wants a window seat, who has budget constraints, which arrangements are merely proposals, and which have become joint decisions.

For an AI assistant to serve a group of people, the key is not just retrieving memories, but judging whose memory and which memory should be used at this moment.

To address this problem, Tencent Yuanbao and the King's College London NLP team jointly introduced GroupAssistBench. The research is based on 50 real group chats and 80,299 historical messages, constructing 2,395 evaluation questions to examine whether Agent memory can truly move from "remembering" to "using well."

The research team compared 14 configurations, including full context, retrieval augmentation, and external memory. One representative contrast is: EverMemOS scored 0.621 on the L1 memory recall evaluation, higher than the full context's 0.570 using the same task model; but on the L2 open-ended service evaluation, it scored 0.247, lower than the full context's 0.431.

Being better at recalling does not necessarily mean being better at serving.

The paper has been made public on arXiv, and the code and dataset have also been released.

△ Main experimental results of the 14 configurations: L1 memory recall on the left, L2 open-ended service on the right. Comparisons should be made within each metric.

From a segment of group chat to two kinds of tests: memories must be findable and also used correctly

Existing Agent Memory evaluations often test historical recall through factual question answering, and recent work has begun to focus on downstream memory use. But real group chats pose more complex problems: the same piece of information may be supplemented by different members, arrangements may be updated through discussion, and a single reply may simultaneously be constrained by multiple people.

GroupAssistBench constructs data from interactions with real deployed group assistants, preserving naturally occurring topic switches, state changes, and interwoven multi-person information, and divides the evaluation into two interrelated task levels.

L1 and L2: From answering facts to completing services

L1 memory recall has clear reference facts, such as "which day was the initial agreed departure." L2 memory use requires completing a service, such as "help everyone send a travel and ticket-purchase reminder." The answer has no fixed wording, but must comply with valid historical decisions and member requirements. The two types of questions number 1,374 and 1,021 respectively.

The research further divides them by the memory structure they rely on into M1 single memory, M2 multi-memory integration, and M3 update and conflict handling. M3 especially requires identifying: when an old decision is replaced by a new decision, and which members' constraints still apply. These categories describe forms of information dependency rather than a simple difficulty ranking.

△ Overview of the evaluation framework: L1 factual recall and L2 open-ended service questions are constructed based on the same group chat history, and dependency types are divided into single memory, multi-memory, and update conflict.

Every question must truly depend on history

To avoid systems being able to guess correctly based on common sense alone, the team first compared answers "with history provided" and "without history provided," screening for requests that genuinely require group chat memory; then established Memory Anchors for the necessary facts, locating the source messages and evidence snippets, and setting respective history cutoffs to prevent models from reading information after the question was asked.

The data underwent anonymization and privacy review, and the public version does not contain original group chat identity information. The construction process combined programmatic rules, semantic review, and manual verification; in sampled verification, 95.8% of the questions were judged valid by majority vote.

△ Data construction process: screen requests that genuinely depend on history, locate memory anchors, construct evaluation questions, and then gatekeep through rules and manual verification.

Services have no standard answers, so how are scores assigned?

When a group assistant helps everyone arrange activities, reminds about ticket purchases, or coordinates disagreements, there can often be multiple suitable replies. Simple string matching cannot determine whether a service truly satisfies historical constraints. Therefore, the team constructed customized scoring rubrics (Rubric) bound to the corresponding historical evidence for each L2 question.

This Rubric consists of three parts: Grounding checks whether the answer explicitly reflects the necessary historical facts; Service checks whether the task is completed; and Penalty penalizes errors such as violating existing decisions, ignoring member restrictions, or using outdated information. For example, a ticket-purchase reminder must adopt the new date while also accounting for the relevant number of people, transportation arrangements, and individual seat preferences; treating an obsolete date as the final arrangement should be penalized.

The design of Penalty also draws on corrective feedback from real users to group assistants. The research conducted manual verification using 1,152 rubric judgments across 60 L2 questions and 180 responses, and the agreement rate between automatic judging and manual majority vote was 89.3%.

497 paired memories: Why are facts found but still not used?

To avoid mistaking gaps between different question types for differences in memory ability, the research tracked 497 paired memories: the same segment of history was both directly asked about in L1 and genuinely depended on by an L2 service. This makes it possible to observe what happens at the service stage to memories that have already been recalled.

Among the analyzed external memory systems, for memories that were recalled in L1 but not judged as effectively used in L2, about 36% did not enter the L2 model input, while the remaining about 64% entered the input but were not judged as effectively applied in the response. These percentages only describe the destination of this type of paired memory, not the failure rate of all L2 tasks.

The former situation is related to requests lacking clear retrieval cues: when a user says "help everyone arrange it," they do not point out one by one which historical preferences need to be retrieved. The latter indicates that evidence having been provided to the model does not automatically mean the model knows how it should constrain the current response.

△ The destination of 497 paired memories: comparing whether memories successfully recalled in L1 were applied in L2, entered the input but were not judged as applied, or did not enter the input.

Two interventions: stringing scattered evidence together and turning historical requirements into constraints

Based on the above phenomena, the team explored two lightweight intervention directions. RER (Relational Evidence Refinement) organizes the relationships among people, events, times, and decisions after retrieval, and supplements relevant candidate evidence when needed; ACB (Assistance Constraint Binding) organizes the obtained facts into response constraints with subject, scope, and current valid state.

In the experiments, RER improved LightMem's L2 average score by 6.28 percentage points; ACB improved xMemory and EverMemOS by 3.94 and 3.16 percentage points, respectively. When the two were combined, the L2 scores of LightMem, xMemory, and EverMemOS increased by 7.03, 4.64, and 4.43 percentage points relative to their respective baselines.

These results show that evidence organization and constraint explicitation hold promise for improving memory-driven service performance. The independent contribution of each of the two mechanisms still needs further disaggregation, and all improvements cannot simply be attributed to "retrieving a few more memories."

△ Effects of RER and ACB interventions: performance improvements on different memory systems comparing relational evidence organization, service constraint binding, and their combination.

From remembering history to using history at the right time

Real group chats are a testing ground for Personal Agents and multi-user assistants. As agents begin to take on more long-term services, the value of a memory system is not only "how much it stores and how fast it searches," but also depends on whether it can understand when to use, whom it applies to, and which information has become invalid. GroupAssistBench attempts to make these differences observable, analyzable, and comparable.

*Paper link: https://arxiv.org/abs/2611.04867 Hugging Face paper: https://huggingface.co/papers/2611.04867 Code: https://github.com/Tencent/GroupAssistBench Dataset: https://huggingface.co/datasets/tencent/GroupAssistBench

One-click triple**"Like" "Share" "Little Heart"**

Welcome to leave your thoughts in the comments!

— End —

[Academic submissions] Please send an email on weekdays to: [email protected], with [Submission] in the subject line, and tell us: who you are, where you are from, attach the project/homepage link to the submission content, and provide contact information.

🎓 We will (try our best to) reply to you promptly :)

🌟 Light up the star 🌟

See frontier tech progress every day

Read original article

Open in WeChat