Termsslug: swiglu

SwiGLU

An efficient feed-forward network structure combining Swish activation with Gated Linear Units.

Curated Intelligence

1 items · Page 1/1
Hugging Face BlogT1·Models
85 pts
ModernBERT Released: A Modern Replacement for BERT

Hugging Face and partners released ModernBERT, designed to replace the decade-old BERT architecture. By integrating modern techniques like RoPE, SwiGLU, and extended context windows (up to 8k), it significantly outperforms baselines in retrieval, classification, and entity recognition while maintaining inference efficiency.