DeepSeek Maya-S follow-up: higher thinking score comes with a contamination flag
A November 12 dual-3090-Ti report adds a 150-scenario test: pass@1 is 119 with thinking off and 131 with it on. The harness flags the thinking arm CONTAMINATED. It is not a clean thinking benefit or a model ranking; AraoAI has not reproduced it.
On November 12, club-3090 issue #2009 reported DeepSeek-V4.1-Flash-1017 Maya-S on two RTX 3090 Ti cards, followed by a quality-test comment at 20:32 UTC. The important detail is not simply a higher total: the harness itself marked the thinking-on arm CONTAMINATED.
-
Preserve the configuration. The author used Project Maya v1.1.57, tensor parallelism of two, NVLink and the DSpark drafter, with each GPU capped at 325 W. The environment was a KVM guest with GPU passthrough and substantial host-RAM expert offload. These are not stock-power dual-3090 measurements or a GPU-only deployment, and they do not establish a result for arbitrary dual-card systems.
-
A higher total is not a clean causal result. Each arm ran once over 150 scenarios. Pass@1 was 119 with thinking off and 131 with it on, while pass@3 was 134 and 136. Only five of fifteen tool-calling responses in the thinking-on arm contained reasoning, triggering the contamination flag. The twelve-point difference is not a demonstrated stable thinking benefit, and retry-aware scores are not first-answer scores.
-
The protocol matters. Sampling inherited server temperature 1.0 and top_p 1.0 rather than the harness temperature-zero default; the author explicitly warns against comparison with that baseline. Two thinking-on cases hit token limits, and sandbox images predated the latest harness update. The follow-up adds quality testing, not completed verify-full, verify-stress or continuous soak, which the comment still lists as unrun.
The report also distinguishes repeated short prompts from fresh-text speed and warns about comparing different engines, power caps and interconnects. For selection decisions, request raw logs, exact weights, engine identity, sampling protocol and failures. A single self-report is a reproduction lead, not cross-machine validation. AraoAI has checked the source but has not independently measured these results.
Report: https://github.com/noonghunna/club-3090/issues/2009 Follow-up: https://github.com/noonghunna/club-3090/issues/2009#issuecomment-6835193065