A missing reasoning field can distort TTFT and decode rates in community benchmarks
club-3090 PR #2132 reports a reasoning-stream compatibility gap affecting TTFT and derived rates on vLLM. The proposal remains open; successive benches on one boot are not an engine speedup study.
Source: club-3090社区测量工具修补提案
club-3090 PR #2132, opened November 21, describes a benchmark compatibility gap: a reader accepting reasoning_content but not reasoning can miss the first streamed reasoning event. Reasoning time is then charged to prefill, while a short remaining answer window can produce an implausibly high derived decode rate. This changes the timing boundary, not the GPU. The proposal remains open and AraoAI has not independently reproduced it.
-
Stream fields belong to the measurement protocol. The report says vLLM 0.34 uses reasoning for output and identifies readers that recognized only reasoning_content. Proposed changes cover the benchmark, context probe and stress checks. An OpenAI-compatible endpoint does not guarantee identical extension fields; preserve the adapter version as well as the engine build.
-
Reasoning preceding an answer must not disappear from timing. On Inkling-2-Mini with two RTX 3090s and vLLM, the author sequentially ran the original and patched benchmark on the same boot. Old timings showed multi-second TTFT and anomalous decode estimates; patched timings agreed more closely with wall-clock rates. The author explicitly disclaims a speedup comparison. These figures are not being imported as hardware-selection records.
-
An accepted off-switch is not an honored off-switch. A request using enable_thinking:false may return OK only after reasoning first. Checking HTTP success or final content alone can misclassify the model as non-reasoning, impose an insufficient output budget and blame the resulting failures on the model. Validate reasoning fields, answer fields and the actual request together.
Mock-server tests in the proposal exercise tool behavior; they are not hardware benchmarks. The author also explicitly did not run live SGLang or llama.cpp regressions, so this is not cross-framework reproduction. Preserve raw streams and timing-script versions. If a suspicious result's timing boundaries cannot be reconstructed, downgrade it rather than infer correctness from plausibility.
Source: https://github.com/noonghunna/club-3090/pull/2132