Hacker News AIT2·Open Source
72 pts
llama.cpp's tiled K-quant matmul previously shipped only AVX-512, AVX2 and AVX inner kernels, forcing Q3_K, Q2_K and IQ-family formats on Arm into a slow path that re-unpacked weights for every dot product. This contribution adds the missing Arm inner kernel built on the NEON SDOT instruction, lifting IQ3_M prompt processing on Neoverse N1 from 11.5 to 74.5 tokens/s (about 6.5x), with 3.3x to 4.9x gains on Neoverse V1 and V2.