Hacker News AIT2·Models
74 pts
The paper introduces Latent Lattice Language Models (LLLM), which add a horizontal latent width alongside the vertical residual-stream width. At 20B-token pretraining scale, LLLM matches a 40-layer Transformer's validation perplexity using only 22 layers, cutting KV cache by 45%, and can be combined with a Jacobi refiner for further gains.