Termsslug: tiled-k-quant-matmul

tiled K-quant matmul

K-quant 分块矩阵乘

Tiled matmul kernel in llama.cpp for K-quant formats, implemented per CPU instruction set.

Curated Intelligence

1 items · Page 1/1
Hacker News AIT2·Open Source
72 pts
llama.cpp Adds Arm NEON Kernel for 6.5x Faster Prompt Processing

llama.cpp's tiled K-quant matmul previously shipped only AVX-512, AVX2 and AVX inner kernels, forcing Q3_K, Q2_K and IQ-family formats on Arm into a slow path that re-unpacked weights for every dot product. This contribution adds the missing Arm inner kernel built on the NEON SDOT instruction, lifting IQ3_M prompt processing on Neoverse N1 from 11.5 to 74.5 tokens/s (about 6.5x), with 3.3x to 4.9x gains on Neoverse V1 and V2.