A developer released an open-source project that reimplements the DeepSeek V3 architecture from scratch in roughly 190 lines of pure JAX, covering Multi-head Latent Attention, decoupled RoPE, DeepSeekMoE and auxiliary-loss-free load balancing, and pretrains it on a MacBook Air. With no Flax, Equinox or PyTorch dependency and parameters as plain dicts of arrays, the code is readable top to bottom, making it a high-value reference for understanding frontier MoE architectures.
MLA
多潜头注意力
Attention mechanism compressing keys and values into a low-dimensional latent to cut KV cache cost.
Curated Intelligence
vLLM announces Day-0 support for Mistral Large 4, a frontier MoE native multimodal model. The integration features a DeepSeek-style sparse attention stack, cross-layer KV sharing, NoPE layers, and built-in speculative decoding, driving generic improvements in context parallelism and host-memory KV offload within the vLLM framework.
vLLM introduces Decode Context Parallelism (DCP) to address KV cache memory duplication bottlenecks in GQA and MLA architectures during long-context inference. By optimizing context distribution at the decode stage, DCP significantly enhances concurrency and memory efficiency for agent workloads ranging from 64K to 1M tokens on NVIDIA B200 GPUs.