In-Weights Memorization

Through theory and experiments, this work shows that the gating mechanism common in modern recurrent networks explains why State Space Models lag behind Transformers on in-context learning and precise retrieval: gating drives SSMs to first learn an in-weights memorization solution, delaying or even preventing convergence to a correct in-context learning solution, even absent architectural or memory-capacity limits. Conversely, gating often benefits generalization to long sequence lengths, revealing its dual role in training dynamics.