Prefrontal AI and ETH Zurich Unveil LSA to Speed Ultra-Long Context Attention

Original title: 千万token下,比DeepSeek式稀疏注意力再快22倍
78AI Score

London startup Prefrontal AI and ETH Zurich jointly propose Latent Sparse Attention (LSA), which targets the indexer overhead that dominates sparse attention at extreme context lengths. At 10M tokens it speeds up single-layer prefill by 22.7x on GLM-5.3-Flash and 72.4x on Qwen3.8-27B while keeping accuracy close to the original models, offering a subquadratic path toward 10M-token context.

Full Report

A New Approach to Sparse Attention: Up to 72× Speedup in Attention Layers at Tens of Millions of Tokens

Editor|Panda

Today, million-token context has gone from a selling point to a standard feature. In April, DeepSeek-V4 made 1M context the default specification for all official services; in August, Zhipu's GLM-5.3-Flash also supports 1M context and, for the first time in the GLM series, introduced a hybrid architecture of sparse attention and linear attention. Behind these models lies a common choice: each query no longer looks at all historical tokens, but selects only a small number of the most relevant ones to "look at."

This approach has been repeatedly validated at the 1M scale. But as context continues to push toward 10 million tokens, a previously negligible cost begins to spiral out of control. Taking a single global attention layer of GLM-5.3-Flash as an example, when doing prefill at 10M tokens, 98% of the time is not spent on attention computation, but on the indexer that decides "which tokens to look at."

The nature of the problem has therefore changed. Sparse attention has long proven that "looking at only a small subset" is enough to preserve accuracy; what truly bottlenecks ultra-long context is that "finding that small subset" still grows quadratically with context length.

Now, London-based startup Prefrontal AI and ETH Zurich have jointly published a paper proposing a new method called Latent Sparse Attention (LSA).

At 10M tokens, it speeds up prefill of a single attention layer of GLM-5.3-Flash by 22.7× and Qwen3.8-27B by 72.4×, while maintaining accuracy comparable to the original models across a range of tasks.

The "Hidden Bill" of Sparse Attention

Let's first review the cost structure of attention. In standard full attention, each query computes relevance against all previous keys. When the context length is T, the prefill computation is O(T²), and decoding each token is O(T). As context grows from 1 million to 10 million, the prefill attention computation grows by 100×.

The idea of sparse attention is division of labor: first, a lightweight "indexer" selects the k most relevant keys for each query, and then the actual attention is computed only over these k keys. This reduces the cost of attention itself to O(Tk).

This idea rests on a premise: the model really only needs a small number of keys. The paper's appendix specifically validates this: at 128k tokens, letting each query of Qwen3.8-27B look at only its "truly" most relevant 512 keys (about 0.8% of all keys), the model still retains 98% of full-attention accuracy on the RULER long-context benchmark.

But there is a paradox here. For the indexer to know which 512 keys are most relevant, it must first score every key. DeepSeek's DSA, MiniMax's sparse attention, and the block-sparse schemes adopted by models such as GLM-5.3-Flash and Qwen each use methods like reducing precision, reducing dimensions, or compressing 4 keys into 1 to "lighten the load" of the indexer. But these improvements only divide the constant term by a few times; the indexer's workload still grows linearly with context, and prefill remains quadratic.

An analogy: you want to borrow 4,096 books from a library with tens of millions of volumes. Borrowing itself is fast; the trouble is that to determine which 4,096 books to borrow, you have to look at the spine of every single book. The more books there are, the more time spent looking at spines dominates. The paper estimates that at 10M tokens, if the indexer's cost could be reduced to zero, sparse attention prefill could be another 58× faster.

Time breakdown of traditional sparse attention under ultra-long context (a single global attention layer of GLM-5.3-Flash, single H200). At 10M tokens, the indexer accounts for 98% of total prefill time.

Not Sorted by Time, but by Semantics

LSA's solution is to make the indexer stop checking keys one by one and instead check "groups of keys."

D-LSA and N-LSA overview. Top: In an attention layer, keys are first pooled and clustered into buckets; queries score only the bucket centers, then attend only to the selected remote keys (plus sinks and the sliding window). Bottom: Accuracy and speedup of the two variants.

The key lies in how to group. Most existing block-sparse methods group by position, packing adjacent keys into a block. This is like a library boxing books by acquisition date, with a summary label on each box. The problem is that as long as one critical book is in a box, it can be "drowned out" by the summaries of the other irrelevant books.

LSA chooses to group by content. It first projects keys into a learned low-dimensional latent space, then uses hierarchical clustering (Ward's method) in that space to group semantically similar keys into the same "bucket," with each bucket retaining only a center vector and an index table recording the positions of the keys in the bucket. At query time, the query only needs to score against the bucket center vectors, select the most relevant buckets, then "expand" the buckets into concrete keys, gathering k of them to feed into attention.

Returning to the library analogy: books are no longer boxed by acquisition date but shelved by topic. To find books on quantum error correction, you walk straight to that row of shelves; books on the same topic may have been acquired over a decade, but they sit together.

The benefit of this step is constant-order, which the paper calls Flat search. What truly changes the complexity is the Tree variant: buckets are further clustered into larger "node buckets," forming a hierarchical tree layer by layer. The query starts from the top of the tree, and at each layer only the highest-scoring branches are kept to descend (beam search). It is like first choosing a floor, then a section, then a shelf; each query only needs to examine O(log T) center vectors, and the indexer's prefill drops from quadratic to O(T log T), which is the origin of "subquadratic" in the paper's title.

D-LSA design. Left is Flat search: keys are projected, pooled, and clustered, then stored in a bucket cache; the query scores bucket centers and expands them into top-k indices. Right is Tree search: buckets are organized into a multi-level hierarchy, and the query descends layer by layer via beam search.

Two ways to use it: graft onto an existing model, or grow from scratch

LSA offers two deployment approaches.

The first is D-LSA (Distilled-LSA), aimed at already-trained models. It freezes all parameters of the original model, adding only two small projection matrices per layer to project keys and queries into a 64-dimensional index space, then uses distillation to make LSA's scores approximate the original model's attention distribution. The entire distillation used only 352 million tokens; for reference, DeepSeek DSA used 2.1 billion tokens for just the dense warm-up phase of its indexer. The paper also provides a fully training-free version that directly uses the principal component directions of the keys as the projection, achieving accuracy close to the trained version on Qwen3.8-27B.

The second is N-LSA (Native-LSA), which uses LSA natively during pretraining, trained solely with the standard next-token prediction loss. It directly clusters the attention's own keys, requiring no additional index projection, no teacher model, and no auxiliary loss, with zero increase in model parameters.

N-LSA design. The attention layer directly clusters its own keys and can be trained from scratch using only the next-token prediction loss.

On the engineering side, the team wrote GPU kernels targeting H200 and B300 for clustering, scoring, selection, and expansion (plus a Triton version that runs on any GPU), and integrated LSA into SGLang, supporting tensor parallelism, expert parallelism, pipeline parallelism, and context parallelism during both prefill and decode phases.

Report card at 10 million tokens

First, accuracy. The team tested D-LSA on two models, Qwen3.8-27B (whose attention layers are full attention) and GLM-5.3-Flash (which has native sparse attention), pushing the context to 10M tokens, far beyond the 1M limit officially stated for both. On knowledge, reasoning, and long-context tasks such as MMLU-Pro, GPQA-Diamond, AIME25, MATH500, and LongBench v2, D-LSA falls within the baseline's 95% confidence interval and does not significantly change the model's reasoning length.

On RULER from 4k to 10M tokens, Qwen3.8-27B with D-LSA stays within 5 points of full attention up to 1M, and at 10M it actually overtakes, 63.9 versus 59.9. GLM-5.3-Flash with D-LSA is within 3 points of native sparse attention up to 1M, lags by at most 8 points in the 2M to 5M range, and the gap returns to within 4 points at 10M. It should be noted that GLM's baseline sparse attention was trained together with the model, whereas D-LSA only performs distillation on a frozen model; this comparison itself favors the baseline.

To test ultra-long-context capability in a more realistic setting, the team also built a new benchmark, PaperTrail.

It embeds fictional "research threads" into a large number of real 2026 arXiv papers: a fictional paper proposes a result, paragraphs citing it are inserted into several real papers, a few weeks later another fictional paper corrects the value and renames the method, while some subsequent papers still cite the old value—just like real academic literature. The model needs to find and connect these facts as if doing a literature review; a 10M-scale sample contains nearly 400 to 465 papers. On this benchmark, Qwen3.8-27B with D-LSA scores 29.1 at 10M tokens, versus 19.8 for full attention.

On knowledge, reasoning, and long-context tasks (up to 10M tokens), D-LSA achieves accuracy comparable to full attention and sparse-attention baselines.

Now for speed, which is the most striking part of this work. At 10M tokens, for the prefill of a single attention layer, D-LSA is 22.7× faster than GLM-5.3-Flash's native sparse attention and 72.4× faster than Qwen3.8-27B's full attention; in the decode phase, it is 4.6× and 42.5× faster, respectively. This speedup ratio grows with context length: at 1M tokens, the prefill speedup over GLM-5.3-Flash is 3.5×, and at 512k it is only 2.1×, because block-by-block scanning is inherently inexpensive at shorter contexts. Extrapolating the trend, the Tree variant is projected to be more than 90× faster than the baseline sparse attention at 100M tokens.

When the single-layer speedup propagates to the entire model, it gets diluted, because the model still contains many other layers that are unaffected. In an actual SGLang deployment at 10M tokens, D-LSA reduces GLM-5.3-Flash's time to first token (TTFT) by 2.2× (context parallelism), from about 7.5 minutes to about 3.4 minutes; for Qwen3.8-27B, it reduces TTFT by 9.2×, from about 76 minutes to about 8 minutes.

The index itself also uses less GPU memory. At 10M tokens, D-LSA's per-layer index occupies 246 MiB, compared with 331 MiB for GLM-5.3-Flash's native scheme and 642 MiB for Qwen-style block-sparse schemes.

Speedup of a single global attention layer with D-LSA (log scale), index memory at 10M tokens, and prefill latency extrapolated to 100M tokens.

The experimental scale for N-LSA is much smaller. The team trained a hybrid model of about 500M parameters from scratch, in which 1 out of every 4 of the 24 layers is an attention layer and the rest are Gated DeltaNet linear attention layers, with annealing training at a context of 128k tokens. The results show that N-LSA's validation loss is exactly on par with full attention (both 1.085), while PISA, NSA, and sliding-window attention under the same sparse budget achieve 1.100, 1.117, and 1.154, respectively. On RULER at 128k, N-LSA scores 32.2, full attention scores 34.7, and NSA and PISA drop to 19.2 and 14.6. In terms of training speed, the forward plus backward pass of a single attention layer at 128k is 1.6× faster than full attention implemented with FlashAttention-3.

N-LSA training speedup. Left: speedup of the forward plus backward pass of a single attention layer; right: validation loss of the entire model during the annealing phase as a function of training time.

Conclusion

Over the past few years, the main thread of competition in long context has been making attention "look at less." From fixed sliding windows to DeepSeek's DSA and then to various block-sparse schemes, the industry has largely proven that accuracy can be maintained by looking at less than 1% of tokens. What LSA points out is the next bottleneck: once "looking" becomes cheap, "finding" becomes the bulk of the cost.

Turning the attention layer into a retrieval system with a hierarchical index is not a new direction; early works such as Reformer and Routing Transformer all attempted to organize keys by content, but few methods have previously validated both accuracy and speed at scales above 1M. As Agents begin to take on long-horizon tasks such as scientific research and programming that last for days, context windows will continue to grow, and by then what determines the cost curve may well be the complexity of the indexer.

Image

© THE END

For reprinting, please contact this official account for authorization.

Submissions or report requests: [email protected]

Open in WeChat