llama.cpp fixes MoE cache prefetch ordering: verify correctness before speed
PR #32100 merged on November 12 and shipped in b12328. It prevents stale expert reads in a cross-backend scheduling path and bypasses unsupported cache operations. This is a targeted correctness fix, not a universal MoE speedup.
Source: llama.cpp GitHub PR / release
On November 12, llama.cpp merged PR #32100 and included it in release b12328. The issue sits between MoE caching and cross-backend scheduling: an input for the next compute split could be prefetched before a copy callback uploaded the new experts. The following split could therefore read stale weights. Correctness comes before throughput in this case.
-
The trigger is specific. If the cache device cannot execute the cached MUL_MAT_ID operation, the scheduler may place the matrix multiplication on the CPU. The device bank then becomes an input to a CPU split. Prefetching that bank before the upload callback reads it too early. The author identifies some Vulkan cases, not every Vulkan device.
-
The fix changes ordering and capability checks. Prefetch now targets inputs in compute buffers, meaning results produced by earlier splits. Cache-slot construction also checks whether the selected device supports the cached operation, using the uncached path otherwise. This avoids transferring the whole bank to another backend for each micro-batch. Bypassing this cache is not model pruning or reduced weight precision.
-
Preserve the test boundary. The PR supplies scheduler regression and architecture tests and states that the usual CUDA cache path is unchanged because its prefetched inputs are computed activations. These are developer tests, not independent multi-device reproduction by AraoAI. They do not establish an end-to-end speedup percentage.
For deployment debugging, preserve the actual build, backend, cache settings, weight identity and failing input. Compare outputs under a fixed configuration before comparing first-token latency, prefill, decode or transfers. Fluent text is not proof that the correct expert data was read. Faster execution cannot replace a correctness check.
Source: https://github.com/ggml-org/llama.cpp/pull/32100 Release: https://github.com/ggml-org/llama.cpp/releases/tag/b12328