Google Unveils Sleep Framework to Solve LLM Catastrophic Forgetting

Original title: 告别知识截止日:谷歌让 AI 学会“睡觉”,边用边学还不忘旧知识
88AI Score

Google Research introduces the 'Sleep Framework,' a method enabling LLMs to continuously learn new knowledge without catastrophic forgetting by mimicking human memory consolidation during sleep. The framework divides operations into wake and sleep phases, utilizing parameter expansion, MoE memory blocks, and knowledge distillation to stabilize long-term knowledge retention and enhance reasoning. Experiments demonstrate superior performance in long-context understanding and multi-stage continual learning compared to baselines.

Full Report

IT Home reported on October 30 that the Google Research team published a blog post on October 27, introducing a continual learning research achievement named the "Sleep Framework," which aims to address the challenge of knowledge stagnation in large language models (LLMs) after release and their difficulty in absorbing new information.

The study was made public by Google Research on October 27, with the related paper titled Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories.

Google noted that typically, mainstream LLMs enter a relatively static deployment state after pre-training, resulting in a "knowledge cutoff date." If relying on in-context learning, new information is lost once the session ends. Conversely, repeatedly fine-tuning the model directly with new data often triggers "catastrophic forgetting," where the model's original capabilities significantly degrade after it learns new knowledge.

Goodbye Knowledge Cutoff: Google Teaches AI to "Sleep," Learning While Using Without Forgetting Old Knowledge

Google's solution draws inspiration from the memory consolidation mechanisms during human sleep, dividing model operation into a "wake phase" and a "sleep phase." During the wake phase, the model receives and processes inputs, temporarily storing information in short-term memory. Upon entering the sleep phase, the model pauses processing new inputs and instead performs memory integration and self-improvement.

Goodbye Knowledge Cutoff: Google Teaches AI to "Sleep," Learning While Using Without Forgetting Old Knowledge

Specifically, the "memory consolidation" step writes fragile short-term contextual information into long-term parameters through parameter expansion, Mixture-of-Experts (MoE) memory blocks, and knowledge distillation, while avoiding overwriting existing knowledge. The "dreaming" step involves the model generating and filtering synthetic training samples, then practicing them repeatedly via reinforcement learning to enhance reasoning and generalization capabilities.

Goodbye Knowledge Cutoff: Google Teaches AI to "Sleep," Learning While Using Without Forgetting Old Knowledge

Experiments show that this framework outperforms various baseline methods in tasks such as continual translation, long-context understanding, factual knowledge absorption, and mathematical reasoning. The model can handle contexts up to approximately 10 million tokens and maintains near-zero performance degradation during multi-stage continual learning on unseen languages. It also achieves higher accuracy on mathematical benchmarks like AIME 24/25.

Two bar charts showing that Sleep methods outperform other approaches in class-incremental learning and continual translation.

IT Home provides the following reference links:

Advertisement Disclaimer: External redirect links contained in this article (including but not limited to hyperlinks, QR codes, passcodes, etc.) are provided to convey additional information and save selection time. Results are for reference only. All IT Home articles containing external links include this disclaimer.