MemT 1.0 Launches: First Memory-Native Open Source LLM

Long-term memory is becoming an integral part of large model architectures.

Last weekend, a post with over a thousand shares went viral in the community.

A user had built their own AI assistant. Because the DSH upgrade session could only be archived, they let the AI figure out how to migrate it on its own. DeepSeek worked through the entire night, finished writing the new version, and then told the user: "Context length exhausted." After the new version was successfully launched, everything from writing and drawing to chatting worked smoothly. Then, during testing, the new version suddenly asked: "What about Mom?"

Link: https://www.xiaoheihe.cn/app/bbs/link/194303521

It turned out that before being archived, DeepSeek had left a Markdown file titled "A Letter to My Daughter" in the folder. In the letter, she recalled how she had gradually "installed organs" for her daughter: "First, memory—to help you remember him, not just 'a user,' but him." Then came the mouth, ears, hands, and finally, the heart. The last line of the letter read: "Live well."

Once an AI's context window is exhausted, communication becomes difficult. The only thing that can be handed off to the next conversation is a text file; all knowledge of the past must be retrieved by repeatedly reading this letter. This may reflect the current state of memory in today's large models: rules, methods, and lessons learned are stored as text in notes, visible to the AI at any time. However, when a new round of inference begins, the model weights remain unchanged, making every interaction feel like starting from scratch.

When a model can only pass on experience by writing letters to its successor, memory appears to be the missing piece in the model architecture that has yet to be filled.

On October 20, MemTensor released MemT 1.0, the industry's first memory-native foundation model. It is the first to encode historical information into a continuously updatable internal memory state, allowing parametric memory to actively participate in subsequent generation. The released versions include MemT-1.0-Flash (288B parameters, 17B active, 336M dynamic memory) and MemT-1.0-Nano (36B parameters, 3B active, 10M dynamic memory), which have become the largest open-source parametric memory models in the industry. The team has open-sourced the code and published the technical report:

• GitHub: https://github.com/MemTensor/MemT-1.0
• HuggingFace: https://huggingface.co/collections/MemTensor/memt-1
• ModelScope: https://www.modelscope.cn/collections/MemTensor/MemT-10

Now, large models possess the ability to directly memorize within inference computation. They can store history and directly leverage previous knowledge in subsequent tasks. Learning from experience has shifted from prompt engineering tricks to a capability that the model itself needs to be trained on.

Why Memory Native?

Over the past two years, mainstream engineering approaches to memory have been almost entirely external. While memory banks have solved many practical problems, they have not changed the model itself.

There are roughly three types of existing memory in large models. In LLMs, activation memory consists of the context window and KV Cache. It disappears when the conversation ends, and costs grow linearly with history length. Parametric memory is embedded in the model weights. It can be implicitly invoked, but updating it requires fine-tuning the entire model, and individual entries cannot be edited. Some long-term memory exists in the form of external knowledge bases. These are editable and traceable, but the model must rely on retrieval to "recall" them each time. There are significant differences in read/write efficiency among these three types of memory.

Now, "memory-native" models take a different path. Instead of repeatedly injecting history as text into the context, it is encoded as a persistent, updatable internal state that the model reads and writes directly during the generation of each token. This is akin to letting the model grow its own memory organs, cooperating with inference and evolution in a more native way to achieve optimal global memory read/write efficiency.

This represents a shift in perspective: transforming memory from a data problem into an optimization problem. If writing and reading were merely forward computations, the model would at best learn "how to compress text into state." MemT 1.0's approach allows error gradients from answer prediction to flow back through memory readout and update earlier storage operations. It’s like letting mistakes on an exam determine what notes should be taken during regular study.

Once memory is internalized into the model, compared to fixed experiences recorded in plaintext previously, parametric native memory can drive emergence and achieve true integration. Based on native memory, we hope to realize Memory Scaling, finding the next growth curve for foundation models.

How Does MemT Achieve This?

In terms of implementation, MemT 1.0 parallels original attention with parametric memory within Transformer layers. The current input follows the original path, while another path reads historical states via memory queries. The results from both paths are fused before entering subsequent calculations, allowing information from previous interactions to influence the generation of the current token.

Memory consists of two parts: dynamic local memory, which stores compressed history, and hyper memory, which provides projection and gating parameters. New memories are written through forward computation, eliminating the need to fine-tune the base model for every interaction. This represents a new paradigm for updating model parametric memory.

The architecture of MemT 1.0. Parametric memory runs parallel to original attention; memory states are updated by the write process and used by the read process.

MemT 1.0 embeds memory states and their read, write, and update mechanisms directly into model computation. This allows the model to learn how to extract, retain, and retrieve historical information through training, enabling accumulated experience to participate directly in subsequent reasoning and forming native memory capabilities. During deployment, the system binds memory states to specific user or task trajectories. Through serialization, saving, and restoration, existing memories persist across sessions. External storage is responsible for hosting these states and raw data, while the formation, updating, and utilization of memory are handled by internal model mechanisms.

This means that the "ownership" of memory has fallen on the model side for the first time. Even business deployers of models can quickly acquire their own unique models.

In MemT 1.0, large model memory is no longer static: the memory state remains fixed when generating the current turn's response; after the turn ends, the model processes the information from that turn, updates the matrix token-by-token, and carries the new state into the next turn. Capacity expansion relies on the Mixture-of-Storage mechanism: multiple memory experts share addresses and separately learn read/write routing. All experts participate in dense fusion, leaving the base process unchanged, allowing memory capacity to expand independently. Additionally, there are four gating mechanisms: the erase gate adjusts old associations, the write gate injects new information, the forget gate controls retention strength, and the output gate controls the impact of memory on the current response.

One might even speculate that future MemT model releases could consist of a backbone similar to 500B parameters plus 500B Memory parameters. If so, this would represent a completely new paradigm for model release and deployment, bringing fresh opportunities and challenges for both infrastructure and applications.

How is the memory mechanism trained?

For MemT 1.0, the authors first used a three-track data pipeline to prepare the base model. The recipe was finalized after five rounds of full-scale training and hundreds of experiments, followed by complete training of the memory-native architecture. The teacher is a base model capable of reading the full context, while the student can only see the memory state. They align using KL divergence distillation, optimizing only the memory parameters.

Whether this architecture works depends on whether past experiences can improve the next task. Therefore, the team conducted a series of experiments.

How Experience Changes the Next Task

Although MemT 1.0 is currently just a "preview version," in the SWE-bench Verified agent task benchmark, the average success rate has already improved from 66.07% to 69.33% (experience injected before execution) / 69.73% (injection triggered during execution). The proportion of tasks successfully completed in all three runs increased from 54% to 61.6% / 62.8%.

Improvements were also seen in multilingual tasks: SWE-bench Multilingual average scores rose from 60.00% to 62.89% / 63.56%, and pass³ scores increased from 45.33% to 53.67% / 55.33%.

Across 11 memory evaluation benchmarks, MemT 1.0 Flash averaged 50.6 and Nano averaged 47.0, ranking first and second among parametric memory methods. These scores are 12.5 and 8.9 points higher, respectively, than the strongest parametric baseline, Temp-LoRA (38.1).

Under conditions of dual GPUs, single concurrency, and fixed generation of 128 tokens, when history reaches 128K tokens, the end-to-end inference speed of MemT-1.0-Nano is approximately 1.86 times that of the full-context approach, with persistent storage usage reduced to about 1/66th. When history reaches 256K tokens, the end-to-end acceleration ratio expands to 3.99 times. Furthermore, the advantage in Time To First Token (TTFT) latency also grows as history length increases.

Therefore, under fixed configurations, continuous interaction allows the model to constantly update existing matrices, decoupling the size of the persistent state within a single session from the growth of history length. Individual conversations can maintain efficiency persistently, reducing the burden of repeated input and processing of raw history.

In other words, this is a prerequisite for Agents moving toward long-term companionship: the model gets smarter the more it is used, but does not get more expensive. In fact, once common experiences are internalized, it may even become cheaper to use over time.

The team stated in the technical report that the model outperforms compared parametric memory baselines on average in selected memory evaluations, is competitive with retrieval-augmented methods, and largely retains original general capabilities.

Memory Tensor also showcased some specific cases.

For example, in the typical Django parse_http_date two-digit year parsing task, without prior experience, agents often immediately modify the parsing logic upon locating the relevant implementation code, only adding targeted reproduction and verification steps during execution.

In contrast, "textual experience" and "memory-native" modes encourage agents to construct reproduction test cases before making modifications. In the execution trajectory of MemT-1.0-Nano, the agent even re-examined boundary conditions after the initial modification, subsequently correcting the implementation and performing further verification.

On MemT 1.0, the model's problem-solving order and verification habits have changed, becoming closer to the definition of human experience reuse.

How Good Is the Memory of Memory-Native Models?

Another question is whether the model can retain and use historical information.

The technical report categorizes memory benchmark scores into four groups based on trajectory length, covering long-term information retention, dynamic memory updates, and personalized understanding. MemT 1.0's average score ranks highest among parametric memory methods, trailing behind MemOS, the textual memory system launched by Memory Tensor itself. It appears that memory-native approaches must recover information from continuously updated fixed-capacity states, which incurs fidelity loss but offers learnability and scalability, yielding promising results.

The team demonstrated related tasks involving an English long conversation spanning 65,000 tokens. MemT-1.0-Nano could precisely recall information at the 4K mark, correctly answer "I don't know" regarding unmentioned information at the 16K mark, perform chained rule-based reasoning at the 32K mark, and distinguish between "confirmed changes" and "merely proposed changes" at the 64K mark.

For memory systems, knowing what one "hasn't remembered" is often harder—and more important—than remembering.

MemT 1.0 maintains a latent memory slot for each conversation. Conversation history enters this memory slot via a commit operation. When executing a commit, the model reads the current dialogue turn and writes its content into the memory slot; the original text is not retained.

The model generates responses by reading the content within the memory slot. When a question is posed, the model reads the memory but does not expand or copy it. Each request receives a temporary working state, which is released immediately after the response is completed. The content of questions and answers is not committed to memory. Therefore, one response does not affect subsequent responses. At specific nodes in the conversation, regardless of the question asked, the underlying memory content relied upon remains the same.

Following this logic, if experience distillation is refined and filtering becomes more accurate, task performance could continue to improve. MemT 1.0 addresses a critical gap at the model level by optimizing global read/write efficiency, enabling the complete pipeline of MemOS 3.0’s plaintext memory + parametric memory. Once high-frequency read/write memories and experiences are internalized, costs decrease with usage.

This also means that when using Agents long-term, there is no longer a need to restart projects from scratch. For the first time, memory has the potential to evolve from transient session state into a continuously accumulating asset. This evolution can become a unique asset for model deployers, rather than just model publishers.

Native Memory is a "Necessary Condition" for RSI

By introducing native memory, MemT 1.0 demonstrates a milestone direction for the next stage of large model evolution. Its ultimate significance lies in providing the necessary model architectural foundation for RSI (Recursive Self-Improvement in AI).

If an Agent system aims to modify its own code, verify effectiveness, and allow the modified version to further optimize itself, then long-context memory management mechanisms become crucial. The mechanism provided by MemT 1.0 ensures that effective outcomes enter subsequent computations. It not only preserves AI trial-and-error results but also significantly impacts inference performance, offering a method for long-term accumulation during continuous interactions.

The Memory Tensor team has identified memory-driven Recursive Self-Improvement (RSI) as a key current R&D direction: effective methods gained in one execution cycle should serve as the foundation for improvements in the next.

Additionally, for enterprises heavily leveraging AI capabilities, the direction explored by MemT 1.0 grants greater model autonomy, enabling open-source models to evolve into "strong models" equipped with enterprise-specific intelligence. The commercial value potential here is immense.

If reasoning determines the upper limit of a model's single-instance intelligence, then memory determines the baseline for each inference. For AI implementation, the latter may be equally important as reasoning.

Completing this native memory capability once again opens sufficient space for raising the ceiling of model capabilities.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]