Gating Mechanism Found to Bias State Space Models Toward Memorization

Through theory and experiments, this work shows that the gating mechanism common in modern recurrent networks explains why State Space Models lag behind Transformers on in-context learning and precise retrieval: gating drives SSMs to first learn an in-weights memorization solution, delaying or even preventing convergence to a correct in-context learning solution, even absent architectural or memory-capacity limits. Conversely, gating often benefits generalization to long sequence lengths, revealing its dual role in training dynamics.