llama.cpp adds CPU sharing for MoE cache misses, not a universal speed multiplier
Merged November 7 and included in b12191, #30966 can compute a share of missed experts on the CPU and adds a local tuning script. Calibration measurements and predicted rates are separate evidence.
Source: llama.cpp GitHub
llama.cpp merged #30966 on November 7 and included the MoE cache update in b12191, released the same day. The change concerns how missed experts are computed, not the model's parameter count. The event within this reporting window is the merge and release of an older proposal.
-
A cache miss need not always trigger an upload. The new --moe-cache-cpu-share setting computes a share of missed experts on the CPU using host weights. The remaining work uses device cache slots, and the results are combined. This makes upload-versus-CPU work a tuning decision. It does not mean the entire model resides in VRAM, and the setting is not a fixed fraction of VRAM usage.
-
Tune on the actual machine. scripts/moe-cache-tune.py wraps a llama-server command, calibrates two cache sizes and three CPU shares, then replays routing to predict other combinations. Its calibration runs and Predicted table are different types of evidence. A recommended rate is not a measured result suitable for a benchmark record. CPU capability, host memory bandwidth and the upload link can all change the useful split.
-
The upstream example is not a buying guide. The PR includes one developer's tuning output, not independent, protocol-matched evidence across hardware. We do not turn its speed table into a GPU ranking or claim that a higher CPU share is always better. Cache size and CPU share need to be tested together.
Keep the previous build and complete launch arguments. Hold model quantization, context and workload constant while comparing generation speed, output correctness and memory state. llama.cpp uses the MIT license; model weights have separate terms. The verified news is a new configurable execution path, not a universal speed improvement. Calibration also costs time. Short and long requests and different cache states need their own checks; one calibration does not establish performance for every workload.
Sources: https://github.com/ggml-org/llama.cpp/pull/30966 and https://github.com/ggml-org/llama.cpp/releases/tag/b12191.