Clover-35B Open-Sourced: Diffusion LLM Uses MoE to Challenge Agentic Coding

Diffusion Large Language Models (dLLMs) break the serial constraints of autoregressive generation through iterative denoising and parallel token updates. Clover-35B-A3B-Preview is the first large-scale dLLM to introduce a Mixture-of-Experts (MoE) architecture, featuring 35 billion total parameters with approximately 3 billion active parameters per inference step.

By decoupling the generation order, dLLMs support higher levels of parallelism. Clover-35B leverages MoE to balance parameter scale with inference efficiency. The open-source preview version provides the first empirical baseline for validating the capability boundaries of dLLMs at large model scales.

This article is suitable for NLP algorithm engineers, LLM inference optimization engineers, and AI architects.

Diffusion Language Models Begin Scaling Up

Must large language models always generate tokens sequentially from left to right?

For the past few years, autoregressive language models represented by GPT, Claude, and Qwen have dominated the field. As model capabilities continue to improve, the computational overhead caused by token-by-token decoding has become a significant factor affecting inference speed.

Diffusion Large Language Models (dLLMs) are exploring an alternative technical path. Instead of strictly generating text from left to right, they complete generation through iterative denoising and parallel updates of multiple tokens, potentially achieving higher decoding efficiency in specific scenarios.

However, this approach faces a critical question: as model scale expands and task complexity increases, can it maintain its speed advantage while reaching the capability level of mainstream autoregressive models?

On October 28, Shenzhen-based AI startup DiffuSpace released and open-sourced Clover-35B-A3B-Preview, attempting to answer this question.

DiffuSpace was founded by Professor Kong Lingpeng, co-director of the HKU NLP Lab, along with his PhD students Gong Shansan and Ye Jiacheng. The team previously published research on diffusion language models such as DiffuSeq and Dream 7B. Recently, the company completed two consecutive funding rounds totaling hundreds of millions of RMB.

The newly released Clover-35B-A3B-Preview adopts a Mixture-of-Experts (MoE) architecture with 35 billion total parameters, activating approximately 3 billion parameters per inference step. Compared to the previous generation Clover model, its total parameter scale has expanded roughly fivefold.

According to internal test results published by DiffuSpace, the new model achieves an overall generation speed of 680 tokens/s. In certain Agent tasks and general capability benchmarks, its performance matches or exceeds that of autoregressive models of comparable size.

For diffusion language models, this update is noteworthy because it attempts to solve two problems simultaneously: maintaining parallel decoding efficiency at larger parameter scales, and translating speed advantages into actual execution capabilities in real-world tasks.

680 Tokens/s: Can Speed Advantages Translate to Task Efficiency?

The test data released by DiffuSpace primarily covers model generation speed, Agent task capabilities, and end-to-end execution time.

In an official demo showcasing drone airspace management, the model needed to read 20 air traffic control messages, extract information such as locations and radii, and set up corresponding no-fly zones on a 3D map. Clover completed this task in 7.32 seconds, whereas the comparison model Qwen3.5 took 62.3 seconds—a difference of approximately 8.5 times.

This case demonstrates the potential advantages of diffusion models in structured information processing tasks. However, execution time in demos is also influenced by factors such as model configuration, hardware environment, output length, and tool invocation workflows.

In a WindTunnel test comprising 49 tasks, Clover-35B-A3B-Preview had a median execution time of 1.52 seconds. The combination of Jev and Mercury 2.5 took 3.21 seconds, GPT-5.6 Luna took 5.74 seconds, and Gemini 3.6 Flash took 7.22 seconds. Based on these results, Clover's median task duration is approximately 47% of that of the Jev and Mercury 2.5 combination.

However, different model combinations may vary in inference deployment methods, task processing workflows, and billing rules. Median execution time alone is insufficient to determine comprehensive efficiency in actual production environments.

This point is particularly important for Agent applications. A model's token generation speed does not equate to the speed at which an Agent completes tasks. A real-world task may involve multiple rounds of planning, tool invocations, waiting for external services, and result validation. Even if model generation speed improves several-fold, if the majority of time is spent on external tools or task execution steps, the final benefits may be limited.

Therefore, rather than simply comparing tokens/s, Clover's performance within complete Agent workflows deserves closer attention.

What Level of Agent Programming Capability Has Been Achieved?

A major challenge for diffusion language models has historically been maintaining complex reasoning, code generation, and instruction-following capabilities while improving generation efficiency. Evaluation results released by DiffuSpace indicate that Clover-35B-A3B-Preview achieved high scores across multiple Agent benchmarks. Specifically, it scored 73.6 on SWE-bench Verified, 51.7 on Terminal-Bench 2.1, and 65.3 on τ³ Overall.

According to comparative data provided by the team, Clover outperformed Mercury 2.5 and the similarly sized Qwen3.6-35B-A3B in these evaluations. These tests respectively cover fixing real-world software issues, executing tasks in terminal environments, and handling complex tasks involving tool interactions. Compared to traditional knowledge Q&A or code completion, these benchmarks more closely resemble the multi-step work required by Agents.

In terms of general capabilities, Clover was also tested on knowledge, mathematics, coding, logical reasoning, instruction following, and long-context understanding. Based on the data released by the team, Clover surpassed same-size autoregressive models on some metrics but did not dominate all categories. For instance, other models maintained a lead in certain math and code evaluations.

This also indicates that capability improvements in diffusion language models do not occur uniformly across the board. Different tasks have varying requirements for generation order, context modeling, and reasoning processes, causing the advantages of model architectures to shift accordingly.

Reconstructing GDN: How Do Diffusion Models Balance Long Contexts with Parallel Decoding?

Compared to parameter scale and test scores, the more noteworthy technical change in Clover is the adjustment to its underlying network architecture. Clover was not trained from scratch but built upon Qwen3.6-35B-A3B. While retaining the original hybrid GDN/Attention MoE backbone, it introduces a block-level diffusion generation mechanism and modifies the Gated Delta Network (GDN) to be bidirectional, adapting it to the parallel denoising process.

Traditional autoregressive models typically rely on causal attention mechanisms, where the current token can only access historical information that has already been generated. In contrast, diffusion language models can process multiple tokens simultaneously within a text block, gradually forming the final output through iterative updates.

This generation approach imposes different architectural requirements. On one hand, the model needs to retain sufficient historical context to support long-text understanding and multi-turn tasks; on the other hand, it must allow tokens within the same generation block to fully exchange information, preventing semantic inconsistencies during parallel generation. Clover’s design attempts to combine these two needs.

According to the team, the model compresses historical context using recursive states, reducing the computational overhead of repeatedly processing past information. Simultaneously, it introduces bidirectional attention within the current generation block, enabling finer-grained information interaction between tokens at different positions.

Building on this foundation, the model also incorporates a learnable block-size calibration mechanism and bidirectional local feature extraction to adapt to various generation needs. The core of this design is employing different mechanisms for handling historical information versus the current generation block: historical information is preserved via compressed states, while the current block allows for bidirectional interaction and parallel updates.

For diffusion language models, this design is highly targeted. Relying entirely on standard attention mechanisms could lead to continuously rising computation and storage costs as context length and model scale increase; however, over-compressing historical information might result in the loss of contextual details required for complex tasks. Clover attempts to strike a balance between these two extremes through a hybrid structure.

Currently, Clover demonstrates an architectural design approach suitable for large-scale diffusion language models, but whether its efficiency advantages can remain stable across different model scales and task scenarios requires further validation.

From Text Generation to Agents: How Far Are Diffusion Models?

The release of Clover-35B-A3B-Preview reflects the effort by diffusion language models to expand from high-speed text generation to more complex task execution scenarios. In the past, the most easily demonstrable advantage of dLLMs was generation speed. By updating multiple tokens in parallel, they have the potential to reduce the serial decoding steps required for certain text generation tasks.

However, in Agent scenarios, the work required of the model goes far beyond generating text. It needs to understand task objectives, plan execution steps, call external tools, and continuously adjust subsequent behaviors based on execution results. Some tasks also require the model to output strictly formatted responses or maintain logical consistency across multiple related judgments. These demands imply that generation speed is only one component of Agent efficiency.

DiffuSpace revealed plans to launch decision-making models and multimodal models based on Clover-35B-A3B-Preview in the future. Among them, the decision-making model will target fast, structured judgment tasks, planning to jointly process up to 8 related questions in a single generation pass. According to the team’s vision, this approach allows the model to share context across multiple judgments, reducing potential logical conflicts caused by making independent decisions item by item. However, this capability remains part of future product planning, and its actual effectiveness is yet to be determined.

Looking at broader industry trends, diffusion language models are no longer just research directions confined to laboratories. Google previously launched Gemini Diffusion, Inception continues to iterate on its Mercury series models, and open-source research efforts like LLaDA are exploring the boundaries of diffusion-based text generation capabilities. While different teams follow slightly divergent technical paths, they are all attempting to break through the efficiency limitations of traditional autoregressive decoding.

Meanwhile, autoregressive models are also continuously improving generation efficiency through speculative decoding, model structure optimization, and inference engine enhancements. Therefore, the speed advantage of diffusion models does not necessarily mean they will replace autoregressive architectures. In the future, these two technical paths may establish their respective advantages under different tasks and deployment conditions.

For developers, the more valuable question is: Under identical hardware, comparable model capabilities, and acceptable quality constraints, which solution can complete tasks at a lower cost? This is also the critical question that diffusion language models must answer as they transition from research achievements to production environments.

Currently, Clover-35B-A3B-Preview has been officially open-sourced, with model weights and related code released on Hugging Face, GitHub, and ModelScope. The model uses the Apache-2.0 license and provides support for inference backends such as SGLang and llama.cpp.

As more large-scale diffusion language models become open-source, the industry will have the opportunity to further compare the actual efficiency of the two generation paradigms under unified hardware and task conditions. For Clover, the 35B parameter scale and 680 tokens/s speed data are merely starting points. Whether this speed advantage can be stably translated into cost and efficiency gains in complex Agent tasks is the more significant challenge for the next phase.