Termsslug: hyper-connections

Hyper-Connections

超连接

Residual-stream enhancement that widens residual connections into multiple parallel pathways.

Curated Intelligence

1 items · Page 1/1
Hacker News AIT2·Models
74 pts
Latent Lattice Language Models Cut KV Cache 45% with 22 Layers

The paper introduces Latent Lattice Language Models (LLLM), which add a horizontal latent width alongside the vertical residual-stream width. At 20B-token pretraining scale, LLLM matches a 40-layer Transformer's validation perplexity using only 22 layers, cutting KV cache by 45%, and can be combined with a Jacobi refiner for further gains.