← News
Arena

DeepSeek Maya-S follow-up: higher thinking score comes with a contamination flag

A November 12 dual-3090-Ti report adds a 150-scenario test: pass@1 is 119 with thinking off and 131 with it on. The harness flags the thinking arm CONTAMINATED. It is not a clean thinking benefit or a model ranking; AraoAI has not reproduced it.

Source: club-3090 community report / follow-up

On November 12, club-3090 issue #2009 reported DeepSeek-V4.1-Flash-1017 Maya-S on two RTX 3090 Ti cards, followed by a quality-test comment at 20:32 UTC. The important detail is not simply a higher total: the harness itself marked the thinking-on arm CONTAMINATED.

  1. Preserve the configuration. The author used Project Maya v1.1.57, tensor parallelism of two, NVLink and the DSpark drafter, with each GPU capped at 325 W. The environment was a KVM guest with GPU passthrough and substantial host-RAM expert offload. These are not stock-power dual-3090 measurements or a GPU-only deployment, and they do not establish a result for arbitrary dual-card systems.

  2. A higher total is not a clean causal result. Each arm ran once over 150 scenarios. Pass@1 was 119 with thinking off and 131 with it on, while pass@3 was 134 and 136. Only five of fifteen tool-calling responses in the thinking-on arm contained reasoning, triggering the contamination flag. The twelve-point difference is not a demonstrated stable thinking benefit, and retry-aware scores are not first-answer scores.

  3. The protocol matters. Sampling inherited server temperature 1.0 and top_p 1.0 rather than the harness temperature-zero default; the author explicitly warns against comparison with that baseline. Two thinking-on cases hit token limits, and sandbox images predated the latest harness update. The follow-up adds quality testing, not completed verify-full, verify-stress or continuous soak, which the comment still lists as unrun.

The report also distinguishes repeated short prompts from fresh-text speed and warns about comparing different engines, power caps and interconnects. For selection decisions, request raw logs, exact weights, engine identity, sampling protocol and failures. A single self-report is a reproduction lead, not cross-machine validation. AraoAI has checked the source but has not independently measured these results.

Report: https://github.com/noonghunna/club-3090/issues/2009 Follow-up: https://github.com/noonghunna/club-3090/issues/2009#issuecomment-6835193065