Nyströmformer: Linear-Time Self-Attention Approximation
Original title: Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method
45AI Score
Nyströmformer approximates self-attention in Transformers using the Nyström method, reducing time and memory complexity from quadratic to linear. The study employs low-rank matrix decomposition to significantly improve efficiency for long sequences while maintaining model performance.
Related Entities
Full Report
Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method