Benchmarks
Qwen3.8-27B UD-Q4_K_M — Vulkan vs HIP (llama-bench, 3 runs each)
| Test | Vulkan (RADV) | HIP (ROCm) | Winner |
|---|---|---|---|
| pp512 — prefill | 359.4 ± 1.4 t/s | 347.5 ± 2.8 t/s | Vulkan +3.4% |
| pp1024/2048 — prefill | 347.1 ± 0.8 (pp2048) | 346.8 ± 0.3 (pp1024) | tie |
| tg128 — decode | 12.31 ± 0.00 t/s | 11.97 ± 0.01 t/s | Vulkan +2.8% |
| tg256 — decode | 12.27 ± 0.00 t/s | 11.98 ± 0.01 t/s | Vulkan +2.4% |
Lane decision: Vulkan for serving (wins both prefill and decode). HIP build retained for A/B re-runs.
Reproduce: /srv/llm/bench/run-qwen-bench.sh · raw logs: /srv/llm/logs/qwen27b-bench-*.log on the machine.
Serving decode with MTP speculative decoding (the real-world number)
| Config | Decode speed | Notes |
|---|---|---|
| Bare dense decode (llama-bench) | 12.3 t/s | bandwidth-bound floor |
| Serving: Vulkan + draft-MTP + KV q4_0 | 19.25 t/s | 256-token completion, wall-clock measured · +57% over bare decode |
--spec-type draft-mtp uses the GGUF's embedded nextn draft head (n=1 — this model has exactly one MTP layer; higher draft counts silently clamp, verified on ashborn-prime 2026-08-15).
Sanity math (why 12 t/s bare is correct, not a defect)
16.34 GiB model × 12.3 t/s ≈ 201 GB/s effective bandwidth — a healthy fraction of the LPDDR5X-8000 theoretical ~256 GB/s. The "26–32 t/s" figures in community reports for this model class were MoE-active-expert math or MTP-accelerated serving, not bare dense decode; treat them as UNVERIFIED for this box.
Functional verification (all pass)
| Capability | Test | Result |
|---|---|---|
| Chat | "Reply with exactly: ARISE" | exact match |
| Tool use | get_weather("Tokyo") via OpenAI tools API | correct call + args |
| Embeddings | /v1/embeddings via Qwen3-Embedding-0.6B | 1024-dim vector |
| Rerank | bge-reranker-v2-m3, relevant doc ranks first | −0.75 vs −11.0 |
| Vision | mmproj-F16 wired into server cmd | describes chart images accurately |
| Llama-3.3-70B tool use | get_weather("Tokyo") via OpenAI tools API | correct call + args |
| Llama-3.3-70B vision | repo ships no mmproj — text-only model | N/A (architecture) |
| DeepSeek-V4-Flash tool use | get_weather("Tokyo") via OpenAI tools API — re-verified 2026-09-09 16:58 | correct call + args |
| DeepSeek-V4-Flash vision | repo ships no mmproj — text-only model | N/A (architecture) |
| GLM-5.3-Flash tool use | same test, PR-27752 build | FIXED 2026-09-10 — 2-line template guard (string-subscript crash in template threw during llama.cpp's parser derivation); now emits structured tool_calls. See changelog. |
| GLM-5.3-Flash vision | repo ships no mmproj — text-only model | N/A (architecture) |
GLM-5.3-Flash unlock story (2026-09-09)
Stock master cannot load the GGUF (failed to load model). Resolution:
PR-27752 was proven to be the matching implementation by parsing the GGUF's KV header
directly — architecture glm5next plus indexer.block_size and
hyper_connection.* keys appear in exactly that PR's converter (ruling out
#27754's kda_* keys and #27773's glm5-next arch name). Build
shipped as a 36 MB PR tarball over the tailnet (the box's own GitHub route crawls at
60 KB/s), built with Vulkan, and the 85.8 GB model loads and benches. Build tree:
/srv/llm/src/llama.cpp-glm.
FP4 unlock story (2026-09-09)
The ROCmFP4 GGUF uses the qwen4exp packed tensor format — unparseable by
stock llama.cpp master. Resolution: a separate build tree at
/srv/llm/src/llama.cpp-fp4 from the exact upstream commit the patches target
(d3ca537, fetched as a tarball — GitHub won't serve a raw SHA as a git ref),
both patches applied cleanly, Vulkan build. An independent cross-check exists via the
author's ready-made fork kingjones30/ROCmFPX at
llama.cpp-fp4b. Full writeup:
research/fp4-unlock-attempt.md.
Serving warnings from the model card (verified source): ngram-based
spec decode collapses to 0 acceptance on Vulkan — do not use; ROCm at ≥64K context can
wedge the GPU (QSA desync) — workaround flags -ctxcp 0 -cpent -1.
Flash-Next serving notes (2026-09-09)
- Tool use: verified — correct
get_weather("Tokyo")call via the OpenAI tools API. - Vision: verified — accurate chart description via
mmproj-BF16. - Draft-MTP does NOT work on Flash-Next: unlike the 27B (whose MTP head is
embedded in the GGUF), Flash-Next's draft head ships as a separate sidecar file and
--spec-type draft-mtpfails withfailed to create MTP context. Served bare at 31 t/s — already the fastest model on the box. Sidecar-draft serving (-md mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf) is a future tuning experiment.
Model comparison — all loadable models benched 2026-09-09 (Vulkan lane)
| Model | Params | Size | pp512 | tg256 (decode) | Verdict |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next UD-IQ1_S (MoE, 1.5625 bpw) | 176.9 B | 67.6 GB | 417.5 ± 0.7 | 31.0 ± 1.6 t/s | best capability-per-t/s · tools ✓ vision ✓ (verified) |
| Qwen3.8-27B UD-Q4_K_M (dense, +MTP serving) | 27.3 B | 16.3 GB | 359.4 | 12.3 bare / 19.3 served t/s | daily driver — tools ✓ vision ✓ (verified) |
| Llama-3.3-70B Q4_K_M (dense baseline) | 70.6 B | 39.6 GB | 109.1 | 5.2 t/s | baseline proof only |
| Qwen3.8-Flash-Next ROCmFP4 (MoE, 4-bit packed, PLE streams from NVMe) | 176.9 B | 113.5 GB | 169.5 / 211.7 | 28.59 ± 0.02 t/s | unlocked — pinned-branch build; best-quality Flash-Next quant · tools ✓ (verified) · vision N/A (no mmproj in repo) |
| GLM-5.3-Flash IQ2_XS (312B MoE, 17B active, 2.31 bpw) | 313.3 B | 85.8 GB | 128.5 ± 0.3 | 14.12 ± 0.03 t/s | unlocked — PR-27752 build (KV-header-verified match) |
| DeepSeek-V4-Flash IQ3_XXS-q6kattn (MoE, 37B active) | 284.3 B | 96.1 GB | 161.6 ± 1.1 | 21.0 ± 0.03 t/s | 1M context — verified loadable, full pool |
The MoE thesis, now proven on this box: the 176B-param MoE decodes at 31 t/s — 2.5× faster than the 27B dense and 6× faster than the 70B dense, while being the largest model of the three. Bytes-moved-per-token is the law of this hardware, and MoE architectures move fewer bytes.
Flash-Next variance note: tg128 showed 21.8 ± 14.9 and
pp2048 342 ± 100 on cold runs — warm-up / PLE-table streaming effects; the tg256
figure (31.0 ± 1.6) is the stable steady-state. Raw logs:
/srv/llm/logs/flashnext-iq1s-bench-fix.log, llama70b-bench-*.log,
glm53-bench-*.log.
Gateway overhead measurement (2026-09-11)
Measured through vs direct to llama-swap, 5 request pairs, post groups-fix:
gateway mean 929 ms vs direct 1408 ms — overhead ≈ 0
(the gateway was faster, its memory-cache hits included). Embed search component:
21 ms resident (was 2.4 s under the pre-fix eviction ping-pong — a ~120×
win). Streaming: byte-for-byte SSE passthrough verified. This is why clients point at
the gateway :8081, not raw :8080.
Media-gen timings (2026-09-11–12)
Video + image + audio generation benched separately — full tables live on the Media Gen page. Headlines: LTX-2.3 dev Q8_0 headline at 1920×1088/33f in ~31 min; Wan 2.2 A14B + Lightning LoRA (4 steps) sampling 287.6 s — the daily-driver video pick; SDXL photoreal images ~55 s each; 121-frame clips = hard upstream memory wall.
Full logs: research/gfx1151-sdcpp-ltx-debug.md,
research/wan-lightning-ab.md, research/uncensored-battery.md.