Hugging Face BlogT1·Models
68 pts
Microsoft releases Differential Transformer V2, optimizing the differential attention mechanism by subtracting two heads within the same GQA group that share keys and values. This update aims to improve computational performance and simplify implementation, with code now open-sourced.