llama.cpp Adds Arm NEON Kernel for 6.5x Faster Prompt Processing

Arm NEON kernel for llama.cpp's tiled K-quant matmul

llama.cpp reads prompts several times slower on Arm CPUs than on x86 for low-bit quantized models. The reason is one routine: the "tiled K-quant matmul" added in September 2026 (llama.cpp PR #27851, by jbooth with bartowski) has inner kernels for AVX-512 VNNI, AVX2 and AVX only. On Arm, the formats without another kernel (Q3_K, Q2_K and the IQ family) fall back to a path that unpacks the weights again for every dot product.

This repository adds the missing Arm inner kernel (NEON with the dot-product instruction, SDOT) and measures it on three generations of Arm server cores.

Prompt-processing speed, tokens per second, before -> after (speedup). Qwen2.5 1.5B, 4 threads, 512-token prompt.

Largest measured gain: 6.5x (IQ3_M on Neoverse N1). Generation speed is unchanged. Formats that already had an Arm kernel (Q4_K_M, Q5_K_M, Q6_K, Q8_0) do not change.

Full tables, including generation speed and the x86 reference: RESULTS.md. Raw llama-bench output: results/bench/.

What changed

The patch is 164 added lines in two files of llama.cpp v0.9.0 (ggml/src/ggml-cpu/tiled/):

• tiled-kernel.cpp: a tiled_run_microtile_neon kernel, its dispatch rung, and a tiled_repack_src1 variant for Arm.
• tiled.cpp: one line, so the routine is enabled when the compiler targets an Arm CPU with the dot-product extension (__aarch64__ and __ARM_FEATURE_DOTPROD).

Two design points:

1. Debias on load. For every supported format, weight code minus bias fits in a signed byte. The kernel subtracts the bias as it loads the codes, so one signed dot product gives the bias-corrected sum directly, in exact integer arithmetic.
2. Interleave the prompt side. The 16 rows of a band are interleaved in place (the same group-local layout the VNNI kernel uses), so one SDOT-by-lane instruction adds four partial sums for four different rows and nothing is reduced afterwards. Four weight rows by sixteen prompt rows are accumulated in sixteen registers per pass.

The portable driver, the tile format, the unpackers and the test are upstream's and are unchanged.

Apply, build, test

git clone --depth 1 --branch v0.9.0 https://github.com/ggml-org/llama.cpp
cd llama.cpp
git apply ../patch/arm-tiled.diff
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=ON
cmake --build build --target llama-bench test-tiled-mulmat -j 4
./build/bin/test-tiled-mulmat          # 15 cases
./build/bin/test-tiled-mulmat --fuzz   # 88 cases

The same change is also provided as whole files in src/patched/, and as the kernel alone in src/neon_kernel.inc with the script that splices it into pristine v0.9.0 files (tools/apply_neon.py).

Benchmark

Before and after come from the same binary: GGML_CPU_TILED_MM=0 switches the routine off, which is the path stock llama.cpp v0.9.0 takes on Arm.

M=Qwen2.5-1.5B-Instruct-IQ3_XS.gguf   # from bartowski/Qwen2.5-1.5B-Instruct-GGUF
./build/bin/llama-bench -m $M -p 512 -n 32 -t 4 -r 3                       # after
GGML_CPU_TILED_MM=0 ./build/bin/llama-bench -m $M -p 512 -n 32 -t 4 -r 3   # before

scripts/ab.sh runs this for all ten formats and tools/make_results.py turns the output into the tables.

What is checked

• llama.cpp's own test of the routine (test-tiled-mulmat, new kernel against the reference path): 15 of 15 cases pass on Neoverse N1, V1 and V2. Logs: results/bench/test-*.log.
• Its randomised mode (--fuzz): 88 of 88 cases pass on all three chips. Logs: results/bench/fuzz-*.log.
• Generation speed is unchanged with the routine on against off, on every format and chip.
• A line-by-line reading review found no correctness bug: docs/REVIEW.md.

Not done: review by a human kernel engineer, test-backend-ops, sanitizer builds, and comparing model output on real prompts.

Limits

• Measured on rented Arm servers (AWS Graviton2, 3 and 4), not on phones. The instruction used exists on phone cores from Cortex-A55 and A75 onward, but phones have smaller caches, slower memory, mixed core sizes and heat limits. No phone was measured.
• One small model (Qwen2.5 1.5B), 4 threads, a 512-token prompt.
• Only prompt processing gets faster. Generation does not use the routine.
• Formats that already had an Arm kernel in llama.cpp (Q4_K, Q5_K, Q6_K, Q8_0, through the older "repack" path) do not change.
• The Arm figures are still well below the x86 reference. What bounds them now is not profiled.
• Tested at 4 and 8 threads only. The reviewer expects the gain to shrink, and possibly reverse, above about 16 threads. Not measured.
• Not tested: Apple Silicon, Windows on Arm, batches between 8 and 32 tokens, small-cache cores.
• 128-bit NEON only; SVE and I8MM are not used.

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）