Architecture

llama.cpp built from current source with two backends — Vulkan (RADV) and HIP (ROCm gfx1151) — benchmarked head-to-head, winner per workload serves. Model weights live on the dedicated 2TB NVMe (/srv/llm/models). Multiple models serve side-by-side where memory allows (Qwen3.8-27B at 19 GB + DeepSeek-V4-Flash at 103 GB ≈ 121 GB co-resident; the 27B + a smaller endpoint otherwise), coordinated by llama-swap — the same tier/eviction design proven on our ashborn-prime node. One OpenAI-compatible endpoint (:8080) fronts everything; a small embeddings model gets a dedicated endpoint; the CPU reranker pattern carries over verbatim. Everything runs as user wg via systemd user units so it survives the dan account removal.

• HIP/ROCm — full-speed prefill (prompt processing) and the path with the most Strix Halo community tuning (draft-MTP, the top-k radix fix for Qwen3.8 — both included in today's source build).
• Vulkan/RADV — 2026 consensus: faster decode on Strix Halo than HIP for MoE models (~40% gap in llama.cpp issue #24438), with MTP spec-decode support merged. Built from the same source tree.
• Benchmarks (see Benchmarks) decide which one each model runs on; both binaries stay installed.

/srv/llm/                     (owned by wg — survives dan-account removal)
├── src/llama.cpp/            source + build-vulkan/ + build-hip/
├── venv/                     Python 3.14 + huggingface_hub CLI (model pulls)
└── models/                   weights, one directory per model (2TB NVMe)
    └── Qwen3.8-27B-GGUF/     UD-Q4_K_XL + mmproj-F16 (vision) + MTP draft

The OS keeps its own 1TB NVMe; wiping/replacing models never touches it.

• llama-swap tier + evict_cost + startup-preload design — with 128G unified memory the VRAM-eviction pressure that shaped the prime config mostly vanishes; co-residency becomes the default.
• DRY sampling block (agent-safe): --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2, repeat-penalty 1.0, no presence penalty, --jinja, -n 8192 cap.
• Expert offload + big-ubatch ingest: -ot ".ffn_...=CPU" pattern, -ncmoe, -ub 2048 — re-validated on gfx1151 during benchmarking (CPU here is 16C/32T, strong for expert compute).
• CPU rerank service: bge-reranker-v2-m3 Q8_0, -ngl 0 -t 4 --rerank --pooling rank — unit file copied pattern.
• MTP draft via embedded nextn head (≤2 draft caveat, ~1.7 GB draft context).

Bound to the tailnet interface only — the LAN-facing firewalld zone stays closed beyond ssh/mdns.

• /srv/llm tree owned by wg (owner's account) — dan is a temporary setup account and will be removed at handback.
• All services will be systemd user units under wg (lingering enabled), same pattern as ashborn-prime — no root-owned services.
• firewalld untouched except by the owner; no inbound exposure beyond ssh.