News

What is changing in local AI — model releases, hardware news, runtime updates, and deployment guides.

News

Guides

Edge AI measurements: Jetson AGX Orin 110 tok/s @ 37W, 3 tok/s per watt record

10-16 batch 108 edge / mobile device measurements: Jetson AGX Orin 64GB avg 110.96 tok/s (37.5W), Snapdragon X Elite 80.82 (22.5W), Snapdragon 7 Gen 4 78.59 (21.2W). **Per-watt tok/s: Jetson 2.96 — DB record**.

Qwen3.5-4BQwen3.5-9BQwen3.5-27BQwen3.5-0.8BQwen3-1.7BQwen2.5-Omni-3BQwen3-Embedding-0.6Bgemma-4-E2B-itNVIDIA RTX 5090Apple M5 MaxNVIDIA Jetson AGX Orin 64GBQualcomm Snapdragon X EliteQualcomm Snapdragon 7 Gen 4
Guides

Small model discovery: Qwen3.5-0.8B + Q8_0 + RTX 5090 = 384 tok/s

10-10 batch measurement: Qwen3.5-0.8B (0.8B params) Q8_0 quantization on RTX 5090 hits 384.44 tok/s, 4.03GB VRAM. Read 6000 chars/sec. 0.8B is a build sweet spot — single card, low VRAM, desktop inference peak.

Qwen3.5-0.8BNVIDIA RTX 4090NVIDIA RTX 3090Apple M3 MaxNVIDIA RTX 5090NVIDIA RTX 5080NVIDIA RTX 3060 12GBNVIDIA RTX 5070Intel Arc B580
Guides

Low-cost local LLM inference: not just VRAM, but whole-machine data movement and scheduling

Local inference bottleneck has shifted from VRAM to whole-machine data movement + scheduling. Same card + model: llama.cpp ~50 tok/s vs Strata 70 tok/s (+40%). 3060 12GB whole-machine at ¥2500 = 77% performance, 38% price of 4090.

Qwen3.8-27BQwen3.6-27BQwen3.5-27BNVIDIA RTX 4090NVIDIA RTX 5090NVIDIA RTX 3060 12GBNVIDIA RTX 5070NVIDIA GTX 1080 TiIntel Arc B580
Guides

Strata install in 5 steps: 30 minutes to Qwen 3.5 27B

Five steps to install Strata and run Qwen 3.5 27B IQ2_XS at 70 tok/s. Rust toolchain → clone → release build → GGUF model → run + monitor.

Qwen3.5-27BNVIDIA RTX 3060 12GB
Guides

Used GPU keep or sell: VRAM 6GB is the dividing line

Strata gives used GPUs three more years. Decision rule: VRAM ≥ 6GB keep, < 6GB sell. RTX 3050 4GB cannot fit 27B quantizations.

NVIDIA RTX 3060 12GB
Guides

Manual setup with llama.cpp

Download GGUF weights, pick your own quantization and flags, and understand every step — more control, higher performance ceiling.

Guides

Quickstart with Ollama

The fastest path from install to your first chat: run an open model on your own machine with one command.

7 articles