DiffuSpace Open-Sources Clover 35B Diffusion Language Model for Agents

By 蔡芳芳Original title: 杀入Agent主场之后:两年内部分替代自回归,扩散智能的底气从何而来?| 专访
78AI Score

DiffuSpace has open-sourced Clover-35B-A3B-Preview, a 35B-parameter diffusion language model with roughly 3B active parameters that uses block-wise diffusion for parallel token generation, reaching 5-10x the throughput of comparable autoregressive models. Under a unified evaluation protocol it tops all seven agent benchmarks reported, scoring 73.6 on SWE-bench Verified and 51.7 on Terminal-Bench 2.1, with the technical report and SFT code released alongside. The team expects diffusion LMs to replace same-scale autoregressive models in some scenarios within one to two years.

Full Report

Contents4 sections

AI has already begun writing code, operating software, and executing complex tasks on behalf of humans, but the mainstream large language models behind them still generate answers one token at a time.

Diffusion language models are trying another path. They can generate multiple tokens in parallel and repeatedly adjust content during generation, without strictly following a left-to-right order. In the past, the biggest appeal of this route was speed; today, as agents become the focus of competition among large models, diffusion models are also trying to prove that they can complete more complex tasks.

Not long ago, DiffuSpace open-sourced Clover-35B-A3B-Preview (hereinafter referred to as Clover), with 35 billion total parameters and about 3 billion activated parameters. On November 19, the team officially released a technical report and simultaneously opened the SFT code, further disclosing the model's architecture design, training methods, and evaluation results.

Technical report link: https://arxiv.org/abs/2611.24636 Open code link: https://github.com/DiffuSpace/Clover-35B-A3B-Preview/tree/main/sft

Under a unified evaluation protocol, Clover achieved the highest scores on all seven agent metrics among the comparison models listed in the technical report, covering tool use, professional work, and agentic coding. Among them, SWE-bench Verified reached 73.6, and Terminal-Bench 2.1 reached 51.7. In terms of speed, under some test configurations, Clover's token generation speed reached 5–10 times that of autoregressive models of the same level.

But going from generating tokens to completing tasks also involves tool calls, environment feedback, testing, and code repair. Faster generation can shorten part of the waiting time, but how much of it can ultimately be converted into real productivity also depends on factors such as the task itself and the execution workflow.

"The team is still continuously verifying the path from faster generation to improved efficiency on complete tasks," Gong Shansan, co-founder and chief scientist of DiffuSpace, admitted frankly in an exclusive interview with InfoQ. To enable diffusion models to efficiently handle long contexts and complex tasks, the team made major adjustments to the underlying architecture, adopting block diffusion to reduce redundant computation and achieve bidirectional information interaction within the current generation block. On this basis, how to further expand the scope of bidirectional modeling and leverage the global planning potential of diffusion models remains the team's next research focus.

Gong Shansan maintains firm confidence in this technical route. She still stands by her previous judgment: within the next one to two years, diffusion language models have the opportunity to replace autoregressive models of the same scale in some specific scenarios. From the early exploration by a few researchers in 2022 to today's competition in agentic coding and tool use, which thresholds have diffusion models already crossed, and which directions are worth further exploration?

Diffusion models want to run faster, so why cut text into small blocks instead?

In 2022, when Gong Shansan began exploring diffusion language models, autoregressive generation was already a popular technical route in the large language model field. DiffuSeq, which she participated in, was one of the earlier studies applying diffusion mechanisms to text conditional generation. The team realized at the time that this generation method could bring higher parallel efficiency and was also expected to improve reasoning and global planning capabilities through bidirectional information interaction. Looking back four years later, Gong Shansan feels that most of the subsequent development of this technical route exceeded initial expectations.

The first was scaling. In 2023, the team began trying to expand model scale, but there was almost no mature experience to draw on at the time. There were too many technical paths to explore, and which one would work and which would not could only be tested one by one. The team initially tried diffusion modeling in continuous space, but found that the learning complexity was very high and the model was difficult to train successfully; later they shifted to discrete space and only gradually found a feasible route after multiple rounds of experiments.

Harder than trial and error was not knowing whether this path could work at all. At the time, very few people were researching diffusion language models, and it was unclear whether others were doing similar research or how far this direction could go. This uncertainty was itself a challenge for researchers. From the initial exploration by only a few people, to the later formation of a research group, and to today when more and more teams such as Google, Inception, and Ant have joined, a niche direction once explored by only a few people gradually gained more fellow travelers, which made Gong Shansan more convinced that diffusion language models have development potential worth long-term investment.

As model scale expanded, one problem became increasingly prominent: how can diffusion models maintain efficient generation when handling long contexts? Early approaches required repeatedly processing relatively large ranges of sequences, and as context grew, the cost of redundant computation increased accordingly.

Clover chose block diffusion to alleviate this contradiction. The model divides the generation process into multiple blocks. Within each block, tokens are still generated in parallel through diffusion, while completed content can be reused through KV Cache to avoid repeatedly computing historical context. This preserves the advantage of parallel generation within a block while reducing the overhead of long-sequence generation.

But after blocking, the model must handle two types of information simultaneously: the historical content already generated previously, and the tokens within the current block that need to reference one another. To this end, Clover made bidirectional modifications to Gated DeltaNet (GDN), forming BD-GDN. Compared with the original design mainly oriented toward left-to-right generation, BD-GDN must both efficiently store and read historical information and support bidirectional information interaction within the current block.

According to DiffuSpace's technical report, Clover adopts a hybrid architecture, with every four layers as a group, including three layers of BD-GDN–dMoE and one layer of Full Attention–dMoE, repeated ten times. In terms of implementation, the team processes the information read from historical states and the information within the current block in the same Softmax, and introduces a learnable block size calibration mechanism; at the same time, it adds an independent right-side context convolution branch so that when processing the current position, the model can also use information from subsequent positions within the current block. These modifications are all intended to balance historical information reuse and bidirectional modeling within blocks.

Clover adopts block diffusion, reducing the overhead of long-sequence generation while using bidirectional information interaction within blocks to achieve competitive performance in tasks such as agentic coding and tool use. And this is not yet the full extent of diffusion models' bidirectional modeling capability: the current model still generates progressively backward block by block, and bidirectional generation and modification mainly occur within blocks.

"True global planning means planning globally from start to finish, and that hasn't actually been fully achieved yet." In Gong Shansan's view, Clover, which has yet to fully unleash its global modeling potential, is already showing competitive task performance and giving the team more confidence to keep exploring.

Scaling up the generation block is one direction. The technical report shows that in a full replay test of 64 Agent history trajectory prefixes, after Clover increased the block size from 8 to 16, both the number of tokens submitted per round and the actual throughput improved. Building on this, the team will further study the impact of larger generation blocks on reasoning quality and task performance.

The model runs fast—can Agents complete tasks proportionally faster?

For diffusion language models hoping to enter real-world applications, generation speed is only the first threshold. Today, Agents need to directly read files, call tools, run programs, and continuously adjust their actions based on execution results. Whether a model can complete these complex tasks has become just as important as generation efficiency. Clover 35B therefore focuses on Agent coding and tool use.

In evaluations published by Diffusion Intelligence, Clover 35B scored 73.6, 51.7, and 65.3 on SWE-bench Verified, Terminal-Bench 2.1, and τ³ Overall, respectively, leading among the listed autoregressive models of the same scale and other diffusion models. Among these, SWE-bench Verified contains 500 human-verified real GitHub software engineering issues, requiring the model to understand the code repository and complete fixes; Terminal-Bench 2.1 requires the model to actually execute tasks in a terminal environment.

However, an Agent's final performance depends not only on the model but also on factors such as the task execution framework (Harness), tool configuration, and context management. In public evaluations, Clover used the same Harness as the comparison models to eliminate as much as possible the influence of differences in execution systems. But in practical applications, Diffusion Intelligence is also developing a Harness specifically adapted to diffusion models, hoping to further leverage the advantages of parallel generation.

Most existing Agent frameworks are designed around autoregressive models, typically working in the manner of "generate content—call tools—wait for feedback—continue generating." Diffusion models can generate multiple tokens at once, opening up new space for coordination between model generation and tool execution. For example, for tool calls with no dependencies between them, the system can attempt to execute them in parallel rather than waiting strictly in sequence. Gong Shansan believes that through the co-design of the model and the Harness, there is still room to further improve the overall execution efficiency of Agents.

Some real-world tasks have already demonstrated this potential. In a test involving creating a private project through a browser and inviting members, Clover completed the task in 13 steps and 169.9 seconds, while the comparison Qwen3.6 Flash took 21 steps and 392.3 seconds. Another set of WindTunnel tests covered 49 tasks, where Clover's median execution time was 1.52 seconds, compared with 3.21 seconds for the Jev and Mercury 2.5 combination. Beyond model response speed, the time overhead in task execution steps and tool-calling processes is also beginning to become an area where diffusion models demonstrate efficiency advantages.

But in more complex AI coding tasks, generating code is only part of the entire workflow. The model also needs to run tests, handle errors, modify code, and sometimes even make multiple attempts. If a single generation is fast but requires repeated error fixes, the final task time may not be proportionally reduced.

If the model generates tokens several times faster, how much faster can an Agent complete an entire task? Gong Shansan believes this is a complex engineering problem. The team has not yet conducted a rigorous systematic comparison of end-to-end task time, but based on current usage experience, she estimates that the improvement in model speed can roughly translate into task efficiency gains at a ratio of about 1. In the future, if model generation can work better in coordination with links such as tool calls in the Harness, there is still an opportunity to achieve a "1+1 greater than 2" effect, which is also a direction the team is exploring.

Beyond speed, diffusion models also have potential advantages in consistency for code and structured data generation. For example, variables referenced before and after in code need to remain consistent, and the brackets and fields of JSON must comply with strict formatting constraints. Diffusion models can simultaneously leverage information from preceding and following positions within a single generation block, which Gong Shansan believes helps improve the consistency of generated content.

However, Clover does not lead in all evaluations. In highly difficult tests such as Humanity's Last Exam (HLE), Clover's performance is relatively weaker, and its scores on some traditional tasks are also slightly lower than Qwen's. Gong Shansan mainly attributes this to insufficient relevant training data.

It is worth noting that diffusion models do not require a completely different set of training data. The data currently used by the team is basically the same as that for autoregressive models, with the main difference lying in data processing and learning methods: during training, some tokens are randomly masked, and the model then learns to recover this content. The same data, through different processing and learning processes, may lead the model to ultimately master different capabilities.

Why AI is starting to relearn "making decisions by intuition"

Over the past few years, the large model industry has continuously pursued stronger reasoning capabilities, trying to trade longer chains of thought and more test-time compute for more accurate answers. But recently, another line of thinking has begun to attract attention: does AI really need to think for that long every time?

The emergence of decision models such as Jev has refocused the industry on System 1 fast decision-making capabilities similar to human intuition. In Gong Shansan's view, although Jev itself has some technical controversies, it has reawakened attention to the value of fast decision-making. For diffusion language models, which take parallel generation as an important advantage, this is also a favorable direction. When launching Clover, Diffusion Intelligence also showcased research progress on a joint decision model, a scenario that can further leverage the advantages of diffusion models in speed and bidirectional information interaction.

Gong Shansan uses playing badminton as an analogy: when a player receives the shuttlecock, they do not first calculate the trajectory of the ball, analyze all possible movements, and then decide how to swing. Many judgments rely on quick reactions formed through experience, completed in an extremely short time. After AI enters real environments, it will encounter similar situations, such as a robot judging whether a road is safe, whether an object can be grasped, or which direction to move in next. If every time requires a lengthy Thinking process, there may simply be no time to react.

However, making a single judgment quickly is only the first step. When multiple decisions are interrelated, can the model account for the constraints among different judgments while maintaining response speed?

In its official technical blog, Diffusion Intelligence demonstrated a contract judgment task: the model needs to answer five interrelated questions based on the same contract, such as whether the notice was timely and whether the deposit should be retained. These questions have logical constraints among them—for example, the deposit may be retained only if the notice was timely. If each question is judged independently, the model might conclude on one hand that the notice was overdue while on the other hand that the deposit can be retained, leading to self-contradictory conclusions. Diffusion Intelligence's joint decision model, by contrast, uses a bidirectional generation mechanism to let multiple answers reference one another within the same generation process. In this test, the joint mode answered all five questions correctly, the independent mode answered four correctly, and Jev answered only two correctly.

Similar joint decision-making approaches also apply to tasks such as chess and Sudoku: choices at different positions constrain one another, and the model needs to consider the constraints among multiple judgments simultaneously rather than focusing only on whether a single-step decision is correct.

Larger-scale internal testing shows similar differences. According to the latest technical report, in a validation set containing 500 groups and a total of 2,420 interrelated questions, when the same decision model used joint prediction, the per-question accuracy was 93.88% and the all-correct rate for entire groups was 77.0%; with independent question-by-question prediction, the two scores were 90.37% and 65.4%, respectively. Joint prediction raised the all-correct rate for entire groups by 11.6 percentage points, supporting the value of this approach in interrelated judgment tasks.

Compared with Agent tasks that require generating very long text, these kinds of decisions usually have shorter, relatively fixed output ranges, and diffusion models can leverage bidirectional information interaction to consider the relationships among multiple judgments in a single generation. This also contrasts with the global planning limitations Clover currently faces: long-sequence generation is still constrained by Block Diffusion, whereas the joint decision model can perform complete bidirectional modeling within a shorter output range.

Beyond fast decision-making, Diffusion Intelligence is also exploring more efficient reasoning methods, hoping to further leverage the parallel advantages of the diffusion architecture to improve response speed while maintaining reasoning quality. For users, the internal reasoning mechanism a model adopts does not matter; what really matters is whether it can complete thinking and tasks faster while ensuring result quality.

For Diffusion Intelligence, this exploration is also beginning to extend to broader scenarios such as embodied intelligence. Gong Sansan revealed that the team has already begun attempting robot deployment and conducting related research, and that multimodality will also become an important direction in the next stage. From Agent programming to joint decision-making, and further to robots that require real-time perception and action, Diffusion Intelligence is seeking more application space for the diffusion generation paradigm.

Committed to Being a Foundation Model Company, but Diffusion Intelligence Still Has Several Hurdles to Cross

As Clover begins to target Agent programming and tool use, Diffusion Intelligence is also developing its own Agent Harness and programming client. This seems to imply that a company focused on diffusion language model research is extending toward the application layer. But Gong Sansan's attitude is clear: Diffusion Intelligence is still a foundation model company.

"A company that purely does products is actually very easily swallowed by foundation models." In her view, AI products such as personal assistants and programming assistants can differentiate themselves through interactive experience and specific features, but as foundation model capabilities continue to strengthen, these features may also gradually be absorbed by model vendors. Rather than competing directly with mature AI Coding products for users, Diffusion Intelligence prefers to invest resources in underlying models. Developing the Harness and client is mainly to adapt to the parallel generation characteristics of diffusion models and demonstrate a user experience different from that of autoregressive models, rather than shifting toward C-end product competition.

In foundation model R&D, how to use training compute more effectively is likewise an important topic. Clover-35B-A3B-Preview chooses to start from an existing autoregressive model, reusing the capabilities Qwen3.6-35B-A3B accumulated during pretraining, and then performing diffusion adaptation. This route reduces the investment required for training from scratch, allowing the team to focus more on exploring the diffusion architecture and parallel generation capabilities.

This approach has already greatly reduced training costs, but diffusion models may still require more training rounds. Gong Sansan believes that trading additional investment in the training stage for higher efficiency in the inference stage is a worthwhile calculation: after a model is trained, it will be called repeatedly, and the time and computational costs saved on the inference side may accumulate continuously over long-term use.

A more realistic challenge is that today's large-model software and hardware ecosystem has mainly developed around the autoregressive architecture, and existing inference operators, serving frameworks, and even GPU compute capabilities and memory configurations are not necessarily best suited to diffusion models. Much of the infrastructure still needs to be developed and optimized by the team itself. For a startup planning to continue scaling up its models, beyond algorithmic breakthroughs there is still a large amount of infrastructure work to complete.

This is also why Diffusion Intelligence continues to invest in the open-source ecosystem. Clover has released model weights and related code, and supports inference frameworks such as SGLang and llama.cpp. The team hopes to attract more developers to participate in model deployment, operator optimization, and application exploration, gradually improving the still-immature engineering ecosystem for diffusion language models.

Despite the many challenges, Gong Sansan still stands by an earlier rather bold judgment: within the next one to two years, diffusion models have a chance to replace autoregressive models. However, the "replacement" she refers to will first occur in certain specific application scenarios, rather than comprehensively replacing the frontier models of companies such as OpenAI and Anthropic in the short term. The latter's advantages in model scale, compute, and engineering ecosystem still require time to catch up with.

More realistic opportunities may emerge in scenarios sensitive to speed and cost, such as on-device deployment. Diffusion Intelligence is exploring applications such as AI PCs and in-vehicle systems. Compared with continuously calling cloud APIs, when a device already has the corresponding compute capability, local deployment can reduce long-term invocation costs and provide more timely responses. In the technical report, the team used an RTX 5070 Ti with 16GB of VRAM, an AMD Ryzen 7 9700X CPU, and 64GB of host memory to perform GPU–CPU hybrid expert inference via SGLang and KTransformers. Clover's routed expert weights use INT4 quantization, while the main non-expert modules remain in BF16. In the full MATH-500 test, after optimizing expert placement, the top-8 validation configuration reached a decoding throughput of 97.52 Tokens/s, with a manually reviewed accuracy of 97.6%; when further adopting top-6 validation, throughput increased to 106.25 Tokens/s, with an accuracy of 96.0%.

For diffusion intelligence, this means it does not need to compete head-on from the start with frontier models that have the largest parameter scale and the highest compute investment. Instead, it can seek replacement opportunities in specific scenarios by leveraging advantages in capability, speed, and cost at the same scale.

In 2027, Diffusion Intelligence still plans to devote the vast majority of its resources to foundation model training, with scaling and multimodality as the two main directions, while Agent products and the developer ecosystem continue to be built around model capabilities. Four years ago, diffusion language models were still a niche path explored by only a handful of researchers; today, Clover has already begun competing with autoregressive models in complex Agent tasks. However, algorithmic breakthroughs still require the combined support of compute, the engineering ecosystem, and real-world applications to truly translate into market competitiveness.

For diffusion models, the truly important breakthrough may not be proving that they can generate tokens in a different way, but rather making that difference ultimately become the reason users are willing to choose them.