Today for AI
HOT RADAR
opensourceHEAT 11.9°

AI CURATED HOT STORY · 11/13/2026

llama.cpp Adds Arm NEON Kernel for 6.5x Faster Prompt Processing

1 reports archived1 independent sourcesupdated 11/13/2026, 18:28:36
Synthesis & Latest Updates
1 Sources Cross-Validated

llama.cpp's tiled K-quant matmul previously shipped only AVX-512, AVX2 and AVX inner kernels, forcing Q3_K, Q2_K and IQ-family formats on Arm into a slow path that re-unpacked weights for every dot product. This contribution adds the missing Arm inner kernel built on the NEON SDOT instruction, lifting IQ3_M prompt processing on Neoverse N1 from 11.5 to 74.5 tokens/s (about 6.5x), with 3.3x to 4.9x gains on Neoverse V1 and V2.

LATEST/llama.cpp's tiled K-quant matmul previously shipped only AVX-512, AVX2 and AVX inner kernels, forcing Q3_K, Q2_K and IQ-family formats on Arm into a slow path that re-unpacked weights for every dot product. This contribution adds the missing Arm inner kernel built on the NEON SDOT instruction, lifting IQ3_M prompt processing on Neoverse N1 from 11.5 to 74.5 tokens/s (about 6.5x), with 3.3x to 4.9x gains on Neoverse V1 and V2.

Hourly heat curve

3 fully observed hours
Now
11.9
Peak
12.721:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first
  1. Hacker News AIT2·72 pts

    llama.cpp Adds Arm NEON Kernel for 6.5x Faster Prompt Processing

    • Arm lacked a dedicated inner kernel for Q3_K, Q2_K and IQ-family quantization, forcing weight re-unpacking on every dot product and dominating prompt-processing latency
    • The new kernel uses the NEON SDOT dot-product instruction, delivering 6.5x on IQ3_M for Neoverse N1 and roughly 4.9x and 4.3x on V1 and V2
    • Benchmarks cover only 4 threads and 512-token prompts; the author expects gains to shrink or reverse above 16 threads, and Apple Silicon and Windows on Arm remain untested