Strata author Niko: 6GB VRAM + three-tier offload = RTX 5090 performance

Strata framework author Niko Veit (@coldniko) on 2026-10-06:

"Strata will support Qwen4 fully... I can bring same performance as you now have on 5090 or 5070 with just 6GB VRAM. I'm only constrained with time and compute but will get better"

Why this might be real

Strata is not a plugin — it's a redesigned inference stack:

1. Three-tier offload: GPU / RAM / SSD auto-scheduled by per-layer latency (not default sharding)
2. MTP speculation: Multi-Token Prediction emits multiple tokens, 1.68× speedup (measured: 47→79 tok/s)
3. AVX-512 VNNI: CPU-side INT8 matrix via AVX-512 — no new hardware required

Stacked: 6GB VRAM can host Qwen 3.5 27B Q4_K_M (~16GB model):

Component	Role
6GB GPU	attention layers (hottest)
64GB DDR5	FFN layers
2TB SSD	remaining low-frequency layers

6GB vs 12GB measurements

Our DB has Strata records:

• RTX 3060 12GB + IQ2_XS = 70 tok/s
• RTX 5070 12GB + IQ2_XS = 79 tok/s
• Intel Arc B580 12GB + IQ2_XS = 45 tok/s

Back-of-envelope from Niko's claim:

• 6GB + SSD hosting 2GB model + MTP 1.68× → 79 tok/s
• i.e. used GTX 1080 6GB could run 27B Q4 near 5090 performance

Three judgment points

1️⃣ This is forward-looking, not reproducible today. Niko wrote "I'm only constrained with time and compute" 2️⃣ "6GB = 5090" applies only to decode (generation), not prefill. Prefill is GPU-bound + capped 3️⃣ Used 1080 bottleneck isn't just VRAM, it's old CUDA cores (Pascal). Strata works around this via MTP

Evidence

• 6GB = 5090: author's own claim (forward-looking, not measured)
• 47 → 79 tok/s: MTP 1.68×, real Strata benchmark
• Three-tier offload + AVX-512 VNNI: from Strata DETAILS.md

Sources

• Niko tweet: https://x.com/coldniko/status/2107406312682258511
• Strata GitHub: https://github.com/Niko1221/Strata
• Local DB Strata records (id 51/52/53)