ALL INTELLIGENCE
Hugging Face BlogT1·Models

Nyströmformer: Linear-Time Self-Attention Approximation

Original title: Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method
45AI Score

Nyströmformer approximates self-attention in Transformers using the Nyström method, reducing time and memory complexity from quadratic to linear. The study employs low-rank matrix decomposition to significantly improve efficiency for long sequences while maintaining model performance.

Full Report

Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method