Community benchmark fix: a buffered tool call has no observable decode window
Two club-3090 fixes merged on October 30 remove decode rates for whole-reply output chunks and align client/server comparisons by request shape and measurement window. The extreme rates were timing artifacts, not hardware gains.
Source: club-3090 community PRs
An apparent rate of hundreds of thousands of tokens per second calls for a timing audit, not celebration. On October 30, club-3090 merged PR #1864 so bench-agentic no longer reports decode throughput when an entire tool call arrives in one output-bearing chunk. The associated community report used QEngine and Qwen3.8-27B. This changes the measurement harness, not model or GPU performance.
A direct probe in the PR shows an empty role chunk, a complete tool_calls chunk, then a finish chunk and usage. The old guard counted choices chunks without distinguishing actual output. It therefore treated the tiny interval from receiving the complete call to receiving the finish marker as a generation window, and divided the full token count by that interval. Those tokens were not necessarily generated inside that window.
The fix counts output-bearing chunks. A whole reply delivered in one chunk has no observable decode window, so its decode rate is marked unavailable. The warning also explains that its reported TTFT includes generation of the whole reply. Wall-clock throughput can still be reported separately, but is not pure decode throughput. The maintainer reports reproducing the same measurement shape on another rig using the recipe. This is community reporting and maintainer checking, not an AraoAI hardware reproduction or a conclusion about every engine.
PR #1865, merged the same day, addresses a different false warning. The client combined prose and code means while the engine comparison used a pooled median including warm-ups and deep-context probes. The requests did not match. The new comparison separates workload shapes, uses their measured-run log windows and compares medians on both sides.
Before using a number for selection, check whether output is incrementally visible, what the TTFT endpoint represents and whether both measurements include the same requests. The source rig is explicitly a power-capped RTX 3090 plus RTX 3090 Ti with different PCIe widths, not a standard pair of RTX 3090s.
Licenses are separate too: club-3090 is Apache-2.0, whereas the QEngine LICENSE retrieved for this report is PolyForm Noncommercial 1.0.0. Do not treat the latter as unrestricted commercial open-source software.
Sources: https://github.com/noonghunna/club-3090/pull/1864 ; https://github.com/noonghunna/club-3090/pull/1865 ; https://github.com/noonghunna/club-3090/issues/1862 ; https://github.com/bpx03/QEngine/blob/main/LICENSE . AI-assisted editorial summary.