llama.cpp fixes tied Top-K selection on CUDA and Metal, not cross-backend equivalence
Merged November 8 and shipped in b12208, #31084 addresses scheduling-dependent selection at tied thresholds. Its scope does not guarantee identical whole-model output across backends.
Source: llama.cpp GitHub PR 与发布页
llama.cpp merged #31084 on November 8 and included it in b12208 that day. The new event is the merge and release; earlier discussion and measurements are not presented as new tests.
-
Run-to-run variation is not always sampling. When more equal-valued candidates sit at a Top-K threshold than there are slots, thread scheduling can decide which positions survive. This concerns an internal selection operator, including attention-indexing use cases, rather than simply a random seed.
-
The guarantee is path-specific. The change handles tied candidates in column order, but the PR explicitly does not make every ordering above the threshold deterministic. With CCCL 3.4.3 or newer, one or two input rows may still use DeviceTopK, whose tied selection remains nondeterministic; that combination is skipped by the test. This is not a promise of byte-identical whole-model output for every backend and shape.
-
Repeatability within one backend is not equivalence across backends. Near ties can still be resolved differently because of rounding. Contributor checks and operator measurements have their own configurations. We do not turn historical operator timings into current model throughput, or describe contributor results as AraoAI replication.
For an upgrade, repeat the same workload with fixed weights, template, context and backend, retaining the exact build and outputs before comparing versions. Separate sampling, tied selection and numerical backend differences. The runtime's MIT license does not replace the model license. This release supplies a bounded repeatability fix, not a universal speed or quality guarantee.
Source: https://github.com/ggml-org/llama.cpp/pull/31084 Release: https://github.com/ggml-org/llama.cpp/releases/tag/b12208