ALL INTELLIGENCE
腾讯混元 (Tencent Hunyuan)T1·Models

Tencent Hunyuan Releases CL-bench: Exposing LLM Context Learning Bottlenecks with New Benchmark and Dataset

By Shihan Dou and Pluto ZhouOriginal title: Learning from context is harder than we thought
78AI Score

Tencent Hunyuan introduces CL-bench to evaluate Large Language Models' ability to learn instantly from new contexts. The study reveals that while frontier models excel at static benchmarks, they struggle significantly in scenarios simulating human-like immediate learning from documentation or rules. Code, data, and the paper are open-sourced, providing a new quantitative standard for improving AI context adaptability.

Full Report

CL-bench project page:www.clbench.com CL-bench paper link: download CL-bench paper CL-bench code repository:Tencent-Hunyuan/CL-bench CL-bench data repository:tencent/CL-bench ## We need AI to be context learners Over the past few years, language models have become astonishingly capable. Frontier systems can now solve International Mathematical Olympiad problems, navigate complex coding challenges, and pass professional exams that take humans years to prepare for. They excel at taking tests, spinning out long chains of reasoning to win at benchmarks. But as impressive as these feats are, they obscure a simple truth: being a "test-taker" is not what most people need from an AI. Look at our own daily work. A developer skims documentation for a tool they've never seen and immediately starts debugging. A player picks up a rulebook for a new game and learns by playing. A scientist sifts through complex experimental logs to derive a new theorem from fresh data. In all these cases, humans aren't relying solely on a fixed body of knowledge learned years ago. We are learning, in real-time, from the context right in front of us. image#100%#center !12 Three examples from real-world work and life scenarios. Case1: converting natural-language flight requests into safe, compliant SDK pseudocode given the SkyNet drone SDK documentation (~70K words); Case2: learning a new game by reading its full rulebook (~15K words) and analyzing a hidden-room scenario with possible outcomes; and Case 3: analyzing 300 raw experiment logs to derive the underlying relation and estimate the resonance constant. ! Current language models do not handle context this way. They rely primarily on parametric knowledge—information compressed into their weights during massive pre-training runs. At inference time, they function largely by recalling this static, internal memory, rather than actively learning from new information provided in the moment. This creates a structural mismatch. We have optimized models to excel at reasoning over what they already know yet users need them to solve tasks that depend on messy, constantly evolving context. We built models that rely on what they know from the past, but we need context learners that rely on what they can absorb from the environment in the moment. To bridge this gap, we must fundamentally change our optimization direction. image#100%#center !12 Paradigm shift in language models. ! ## Measuring context learning with CL-bench To evaluate how far current models are from being true context learners, we built CL-bench, a benchmark that tests whether LMs can learn new knowledge from context and apply it correctly. CL-bench contains 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. It tests a simple but demanding requirement: solving each task must require learning something from context that is not already in the model's pre-training. CL-bench spans four broad types of real-world context learning scenarios: image#100%#center !12 Context taxonomy of CL-bench. ! 1. Domain knowledge reasoning: The context provides specific domain knowledge (such as a fictional legal system, innovative financial tools, or niche expertise). The model is required to use this knowledge to reason and solve concrete problems. 2. Rule system application: The context provides a newly defined formal system (such as a new game mechanism, mathematical formalism, programming syntax, or technical standards). The model must understand and apply these rules to execute tasks. 3. Procedural task execution: The context provides complex process systems (such as workflows, product manuals, and operational guides). The model must understand and apply this procedural information to complete tasks. 4. Empirical discovery & simulation: The context provides experimental data, observational records, or simulated environments within a complex system. Unlike the previous categories, which involve deductive reasoning, this category is focused on inductive reasoning and is the most challenging. The model must discover potential patterns or laws from the data and apply them to solve tasks. image#100%#center !12 Examples from CL-bench. Solving these tasks requires LMs to learn from the provided context. These four cases span: (1) Adjudicating a real dispute under a 23k-word newly enacted statute; (2) Given a new programming language specification, implement a time-based control program with a fixed termination condition; (3) Executing code in an unseen programming framework, and (4) Given the technical specifications and long-term environmental policy scenarios, simulate the sustainable global supply of a critical technology metal. More cases are available at huggingface: CL-bench. ! Together, these categories capture both deductive and inductive forms of context learning that arise naturally in real-world work. More details on CL-bench can be found in our paper CL-bench. ### Design principles and features of CL-bench CL-bench is built around a simple but strict design principle: every task must require learning new knowledge from context. Each context in CL-bench is fully self-contained. All information needed to solve its tasks is provided explicitly within the context itself—no external retrieval is required, and no hidden assumptions are allowed. image#100%#center !12 In CL-bench, solving tasks requires models to learn new knowledge from the corresponding context. ! To ensure that performance genuinely reflects context learning, rather than memorization or data leakage, CL-bench adopts a contamination-free design: 1. Fictional Creation: Experts create entirely fictional content, such as designing a complete legal system for a fictional country, including novel case precedents and legal principles, or creating a new programming language with unique syntax and semantics. 2. Modification of Existing Content: Experts modify real-world content to create variations, such as altering historical events, changing scientific and mathematical definitions, or modifying technical documents and standards. 3. Integration of Niche and Emerging Content: Experts incorporate niche or recently emerging content that is largely underrepresented in pre-training datasets, such as cutting-edge research discoveries, newly released product manuals, or technical documents, as well as domain-specific knowledge from specialized fields. Without any context provided, the state-of-the-art model, GPT-5.1 (High), is only able to solve less than 1% of tasks. This starkly demonstrates that the data is contamination-free, as the model is almost entirely incapable of solving the tasks without learning from the context. Moreover, CL-bench is designed to be high in complexity and sequence-dependent. 51.1% of tasks require sequential dependencies, meaning the solution to a later task depends on the outcome of earlier interactions. This multi-turn design significantly increases the difficulty of the tasks. On average, domain experts spend about 20 hours annotating each context to ensure the quality and depth of the task construction. Each task in CL-bench is fully verifiable. On average, each context is associated with 63.2 verification rubrics, and each task contains 16.6 evaluation criteria. The correctness of every task is evaluated from multiple angles, ensuring comprehensive assessment. ### Empirical findings We evaluate ten state-of-the-art LMs on CL-bench, and results show that context learning remains a significant challenge, and some insights into it. image#100%#center !12 Task solving rate of ten frontier LLMs on the CL-bench. All LMs are evaluated in reasoning mode, with results reported as mean ± std (%) across three runs. ! On average, models solve only 17.2% of tasks. Even the best-performing model, GPT-5.1 (High), achieves just 23.7%. In other words, despite having access to all the information required to solve each task, frontier models fail on the vast majority of them. This indicates that current SOTA models largely fail at context learning. image#100%#center !12 Distribution of error types across models. ! Several additional patterns stand out: - Ignoring or misusing context is the dominant failure mode. Many errors do not stem from missing information, but from models overlooking critical details in the context or applying them incorrectly. In many cases, models revert to pre-trained assumptions even when the context explicitly defines new rules, concepts, or procedures. - Long-context reasoning and instruction following are necessary, but not sufficient. Case analyses show that models struggle to track dependencies across long contexts or to precisely follow novel constraints tend to perform worse. However, even models that handle long inputs well and follow instructions reliably still fail many tasks. Context learning requires more than long-context reasoning and instruct-following abilities. - Inductive reasoning from empirical data exhibits greater difficulty than deductive application. Deductive tasks let models apply rules and procedures explicitly given in context, while empirical discovery & simulation tasks require inductive reasoning—inferring laws and patterns from data or exploring virtual environments. Models perform significantly worse on these tasks, often under 10% success in observational and simulation cases, with high variance across runs, showing that discovering laws and patterns is much harder than applying rules. image#100%#center!12 Performance comparison of GPT-5.1 under high versus low reasoning effort settings across all subcategories. ! - Higher reasoning effort generally improves context learning. Increasing reasoning effort improves performance for some models, allowing deeper engagement with complex context. For example, GPT-5.1 gains ~6% on management and experimental tasks. Other models show little or negative effect, indicating that reasoning alone isn’t sufficient—models must also correctly absorb and organize context information. image#100%#center!12 Performance across different input length ranges, with similar trends observed as context changes. ! - Context learning difficulty correlates with context length, but short context can also be complex. Longer contexts generally make tasks harder for all models, confirming long-context processing as a key bottleneck. Yet even short contexts can be complex when they contain dense information, subtle rules and dependencies, or strict constraints, meaning difficulty comes from both length and complexity of contextual information. More findings can be found in our paper CL-bench. Taken together, CL-bench makes one conclusion unavoidable: today's frontier LMs are still far from reliable context learners. This helps explain a common frustration in real-world deployments. Even with increasingly sophisticated context engineering, models often fail in subtle but consequential ways. Providing context is not enough if the model cannot truly learn from it. Context learning as a foundational capability has been largely overlooked. Until it improves, LMs will remain brittle precisely in the settings where we most want them to help: messy, dynamic, real-world environments. ## The Future If context learning improves significantly, the role of humans in AI systems would shift. Rather than primarily providing training data, humans would focus on context engineering, designing and supplying the richest and most relevant context for each task. Humans' role would transition from training data providers toward context providers. But there's a catch. Even then, context learning is ephemeral: models adapt within a window and forget once it clears. The deeper question is: how can we make knowledge from context persistent? This knowledge goes beyond facts—it includes skills, experience, and patterns that help models generalize across tasks. image#100%#center !12 Memory is the key to consolidating the experience that language models acquire through context learning. ! For now, the focus is clear. it's time to make context learning a reality! ## Acknowledgements I would like to thank Pluto Zhou. The concept of context learning originated from his proposal in 2024. I would also like to express my sincere gratitude to Shunyu Yao for his encouragement and invaluable feedback. His repeated and insightful suggestions and revisions have greatly helped us refine this work. I also want to thank Shichun Liu and Junjie Shan for their help with the writing and presentation of this blog. Finally, I'm deeply grateful to my advisors, Tao Gui, Qi Zhang, and Xuanjing Huang, for their unwavering support of my research. Written on January, 2026, as the new year begins.