Models
The machine has ~110 GiB of realistically usable unified memory (128 GiB minus OS and
KV-cache headroom). Every pick below was verified live against HuggingFace and the
llama.cpp project — not from prior knowledge. Full research:
research/models-deep-dive.md in the build docs repo.
Selected for this machine
| Model | Size on disk | Why |
|---|---|---|
| Qwen3.8-27B (UD-Q4_K_XL + MTP draft + mmproj vision) | ~19.2 GB | Daily driver. Vision + tool use + 262K context + draft-MTP speedup. Community: 26–32 t/s on this hardware. Downloading now (hotspot). |
| DeepSeek-V4-Flash (Strix-Halo-verified GGUF, 2.90bpw) | ~103 GB | The big frontier MoE. 1M context, MIT license, 90.8% API-parity verified, 20–27 t/s short context on this hardware. USB transfer pending. |
| Qwen3.8-Flash-Next (ROCmFP4 STRIX build, PLE streamed from NVMe) | ~113.5 GB | Purpose-built Strix Halo quant — only 63–75 GiB memory-resident (51B PLE table streams from disk). 40–47 t/s on code with the HIP top-k fix that today's build includes. USB transfer pending. |
| GLM-5.3-Flash (6block IQ2_XS) | ~85 GB | Staged for when llama.cpp support merges (3 PRs open, none merged as of 2026-09-09 — stock builds can't load it yet). 18B active → speed tolerates the aggressive quant. USB transfer pending. |
| Llama 3.3 70B (Q4_K_M) | ~40 GB | Requested dense-70B baseline. Bandwidth-bound on this iGPU (~5–6 t/s est.) — benchmarked for comparison, not a serving model. USB transfer pending. |
Considered and excluded — with reasons
- kimi-k3 (Moonshot) — smallest GGUF quant is ~330 GB; does not fit
128 GiB unified memory. The C disk-streaming runners exist
(
kimi-k3-in-c~10.7 s/token, K3Flight ~0.8 tok/s, warp ~0.6 tok/s) — all non-interactive. Not viable on this machine. - GLM-5.2 (753B) and DeepSeek-V4-Pro (1.6T) — even expert-pruned quants exceed the memory budget.
- FreeToken — NVIDIA-only upstream (CUDA 13); AMD/ROCm is a roadmap
item (issue #79). A genuine community Strix Halo port exists
(
liangshen68/FreeTokenHalo, Aug 2026, published gfx1151 benchmarks) — but FreeToken's expert-offload engine is CUDA-only, so the port degrades to a resident engine at llama.cpp-parity speeds (~20 t/s on the 8060S), loads BF16 only (no GGUF), and targets ROCm 7.2 vs this machine's 7.1.1. Watch-list, not build-now — revisit if upstream lands a ROCm offload lane. Full notes:research/freetoken-halo-eval.md. - Qwen3.8-Flash-Next UD-IQ1_S (82 GB) — fits fully in memory but is a 1-bit quant with real quality loss; the ROCmFP4 build supersedes it. Second copy staged on ashborn-prime as a fallback.
Capabilities matrix (selected models)
| Capability | Qwen3.8-27B | DSV4-Flash | Flash-Next | GLM-5.3-Flash |
|---|---|---|---|---|
| Tool / function calling | Yes (jinja template) | Yes | Yes | Yes |
| Vision | Yes (mmproj-F16) | Verify at load | Yes (mmproj-BF16) | Unverified for community quants |
| Embeddings | Separate embedding model (served on its own endpoint — see Architecture) | |||
| Draft-MTP speedup | Yes (Q4_0 draft, 1.3 GB) | No | Yes (Q4_K_M draft) | PR-dependent |
| Context | 262K | 1M | 262K+ | 1M |
Why these quants
On a ~256 GB/s unified-memory iGPU, decode speed is memory-bandwidth-bound: bytes moved per token is the whole game, so the quality/size trade-off of the quant dominates throughput. Dense models (like the 70B) move every parameter every token and therefore crawl; MoE models (DSV4-Flash, Flash-Next, GLM) move only their active experts, which is why 100GB+ MoE models outperform a 40GB dense model here. The 27B dense at Q4_K_XL sits at a sweet spot: entire model + KV cache comfortably in memory, 26–32 t/s, full vision and tool support.