News
What is changing in local AI — model releases, hardware news, runtime updates, and deployment guides.
News
Edge AI measurements: Jetson AGX Orin 110 tok/s @ 37W, 3 tok/s per watt record
10-16 batch 108 edge / mobile device measurements: Jetson AGX Orin 64GB avg 110.96 tok/s (37.5W), Snapdragon X Elite 80.82 (22.5W), Snapdragon 7 Gen 4 78.59 (21.2W). **Per-watt tok/s: Jetson 2.96 — DB record**.
Small model discovery: Qwen3.5-0.8B + Q8_0 + RTX 5090 = 384 tok/s
10-10 batch measurement: Qwen3.5-0.8B (0.8B params) Q8_0 quantization on RTX 5090 hits 384.44 tok/s, 4.03GB VRAM. Read 6000 chars/sec. 0.8B is a build sweet spot — single card, low VRAM, desktop inference peak.
Low-cost local LLM inference: not just VRAM, but whole-machine data movement and scheduling
Local inference bottleneck has shifted from VRAM to whole-machine data movement + scheduling. Same card + model: llama.cpp ~50 tok/s vs Strata 70 tok/s (+40%). 3060 12GB whole-machine at ¥2500 = 77% performance, 38% price of 4090.
Strata install in 5 steps: 30 minutes to Qwen 3.5 27B
Five steps to install Strata and run Qwen 3.5 27B IQ2_XS at 70 tok/s. Rust toolchain → clone → release build → GGUF model → run + monitor.
Used GPU keep or sell: VRAM 6GB is the dividing line
Strata gives used GPUs three more years. Decision rule: VRAM ≥ 6GB keep, < 6GB sell. RTX 3050 4GB cannot fit 27B quantizations.
Manual setup with llama.cpp
Download GGUF weights, pick your own quantization and flags, and understand every step — more control, higher performance ceiling.
Quickstart with Ollama
The fastest path from install to your first chat: run an open model on your own machine with one command.
7 articles