Models
The machine has ~110 GiB of realistically usable unified memory (128 GiB minus OS and
KV-cache headroom). Every pick below was verified live against HuggingFace and the
llama.cpp project — not from prior knowledge. Full research:
research/models-deep-dive.md in the build docs repo.
Selected for this machine
| Model | Size on disk | Why |
|---|---|---|
| Qwen3.8-27B (UD-Q4_K_XL + MTP draft + mmproj vision) | ~19.2 GB | Daily driver. Vision + tool use + 262K context + draft-MTP speedup. Community: 26–32 t/s claims did NOT reproduce bare — measured 12.3 t/s dense floor, 19.3 t/s served with MTP. Serving live. |
| DeepSeek-V4-Flash (Strix-Halo-verified GGUF, 2.90bpw) | ~103 GB | The big frontier MoE. 1M context (metadata-verified), MIT license, 90.8% API-parity. On box, on the endpoint (dsv4-flash): measured 21.0 t/s decode, loads at full 1M context. Tools ✓. |
| Qwen3.8-Flash-Next (ROCmFP4 STRIX build, PLE streamed from NVMe) | ~113.5 GB | Purpose-built Strix Halo quant — only 63–75 GiB memory-resident (51B PLE table streams from disk). UNLOCKED: pinned-branch build (d3ca537 + qwen4exp patches); benched 28.6 t/s decode; on the endpoint as flashnext-fp4. Tools ✓ verified. |
| GLM-5.3-Flash (6block IQ2_XS) | ~85 GB | 17B active → speed tolerates the aggressive quant. UNLOCKED: loads via the PR-27752 build, serving live with the tool-template fix; 128.5 t/s prefill, 14.1 t/s decode. Tools work after the 2026-09-10 template fix. |
| Llama 3.3 70B (Q4_K_M) | ~40 GB | Requested dense-70B baseline. Bandwidth-bound on this iGPU (~5–6 t/s est.) — benchmarked for comparison, not a serving model. On box; benched at 5.2 t/s decode — estimate confirmed. |
| Qwen3-Embedding-0.6B (Q8_0) | ~0.9 GB | Embeddings service (embed), always resident (persistent llama-swap group, ~20 ms latency). Backs the gateway's memory layer (mem0 → Qdrant, 1024-dim). |
Tool calling: every chat model serves
structured tool calls; GLM-5.3-Flash needed a 2-line chat-template fix (shipped 2026-09-10,
overlay at /srv/llm/templates/glm53-toolfix.jinja). Auth: two
API keys are live (builder + owner) — see the change log,
2026-09-12 entry. Known perf defect: the CPU governor is still
powersave — one command + optional systemd unit to fix it, documented on the
Runbook.
Considered and excluded — with reasons
- kimi-k3 (Moonshot) — smallest GGUF quant is ~330 GB; does not fit
128 GiB unified memory. The C disk-streaming runners exist
(
kimi-k3-in-c~10.7 s/token, K3Flight ~0.8 tok/s, warp ~0.6 tok/s) — all non-interactive. Not viable on this machine. - GLM-5.2 (753B) and DeepSeek-V4-Pro (1.6T) — even expert-pruned quants exceed the memory budget.
- FreeToken — NVIDIA-only upstream (CUDA 13); AMD/ROCm is a roadmap
item (issue #79). A genuine community Strix Halo port exists
(
liangshen68/FreeTokenHalo, Aug 2026, published gfx1151 benchmarks) — but FreeToken's expert-offload engine is CUDA-only, so the port degrades to a resident engine at llama.cpp-parity speeds (~20 t/s on the 8060S), loads BF16 only (no GGUF), and targets ROCm 7.2 vs this machine's 7.1.1. Watch-list, not build-now — revisit if upstream lands a ROCm offload lane. Full notes:research/freetoken-halo-eval.md. - Qwen3.8-Flash-Next UD-IQ1_S (82 GB) — fits fully in memory but is a 1-bit quant with real quality loss; the ROCmFP4 build supersedes it. Second copy staged on ashborn-prime as a fallback.
Capabilities matrix (selected models)
| Capability | Qwen3.8-27B | DSV4-Flash | Flash-Next | GLM-5.3-Flash |
|---|---|---|---|---|
| Tool / function calling | Yes (jinja template) | Yes | Yes | Yes |
| Vision | Yes (mmproj-F16) | Verify at load | Yes (mmproj-BF16) | Unverified for community quants |
| Embeddings | Separate embedding model (served on its own endpoint — see Architecture) | |||
| Draft-MTP speedup | Yes (Q4_0 draft, 1.3 GB) | No | Yes (Q4_K_M draft) | PR-dependent |
| Context | 262K | 1M | 262K+ | 1M |
Why these quants
On a ~256 GB/s unified-memory iGPU, decode speed is memory-bandwidth-bound: bytes moved per token is the whole game, so the quality/size trade-off of the quant dominates throughput. Dense models (like the 70B) move every parameter every token and therefore crawl; MoE models (DSV4-Flash, Flash-Next, GLM) move only their active experts, which is why 100GB+ MoE models outperform a 40GB dense model here. The 27B dense at Q4_K_XL sits at a sweet spot: entire model + KV cache comfortably in memory, 26–32 t/s, full vision and tool support.