Latent Lattice Language Models Cut KV Cache 45% with 22 Layers

Original title: Latent Lattice Language Models
74AI Score

The paper introduces Latent Lattice Language Models (LLLM), which add a horizontal latent width alongside the vertical residual-stream width. At 20B-token pretraining scale, LLLM matches a 40-layer Transformer's validation perplexity using only 22 layers, cutting KV cache by 45%, and can be combined with a Jacobi refiner for further gains.

Full Report

Contents5 sections

Language modeling after 20B-token pretraining

LLLM (dmodel, dr) has a vertical residual-stream width dmodel and horizontal-latent width dr.

Eval set View Transformer LLLM (single pass) LLLM + Jacobi refiner (J3)

GFLOP/token: floating-point operations per token, excluding attention scores (which add only ≈ 0.1 GFLOP/token) and the embedding lookup. J3 runs three sweeps; its sequential rollout uses a single pass at inference (click Show full table for results).

Show full table

Scaling pretraining tokens: 10B → 20B → 30B

FineWeb-Edu validation perplexity as training continues. Circle area is proportional to parameter count.

Both LLLMs use only 22 layers vs. 40 for the Transformer 40L, so with the same attention width (1024) they keep a 45% smaller KV cache.

Lattice of Latents vs. prior residual-stream enhancements

400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. The Transformer and its single enhancements appear first, then the Lattice and its single enhancements, then two-enhancement combinations.

Transformer variants Lattice variants (ours)

HC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the single +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual.

Deconstructing the Lattice

A hard regime: the vertical residual stream is squeezed to 256 dimensions. Press play to add the horizontal-update enhancements one at a time and watch them recover most of the gap to a full-width Transformer. 10B-token pretraining.

  • adds a component; ↻ replaces one. Parameter counts are shown under each bar.

Recurrent backbones

Pure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) across native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart. Bars appear from highest to lowest perplexity, each at its state-size position.

Pure backbone + Lattice (ours)

Native-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.