Runbook / How to Use

1. Start here — this page. It tells you what exists and what to type.
2. System Audit — the machine's as-found state, before any of this work. If something surprises you, compare against the audit.
3. Change Log — everything that was done, newest first, each entry with why and a revert procedure. This is the receipt page: if you want to undo or tweak anything, start there.
4. Architecture / Models / Benchmarks — the design, the picks and their trade-offs, and the measured numbers with reproduction commands.
5. Anything the setup work does from now on lands in the change log the same day. The site is rebuilt and redeployed on every update.

ssh wg@frame-serv-2          # from any device on the tailnet (MagicDNS)
ssh [email protected]        # same node, by IP

An admin SSH key (from the operator's machine, dxverm@ashborn) is installed for wg. Your own password still works. wg has full sudo (wheel); the inference tree /srv/llm is owned by wg.

/srv/llm/src/llama.cpp/          inference engine source + builds (vulkan/, hip/)
/srv/llm/models/                 model weights (2TB NVMe, ext4, "LLM-MODELS" label)
/srv/llm/venv/                   python env with the hf CLI (model downloads)
/etc/sudoers.d/dan.bak-20260909-orig   original sudoers fragment (pre-fix)

Being built now; the exact unit names and commands will be recorded here as they ship, and each new service gets its own change-log entry.

• Start/stop serving: systemctl --user start|stop llama-swap (as wg)
• Hit the API: curl http://localhost:8080/v1/chat/completions …
• Swap/preview models: llama-swap routes by model name in the request
• Monitor: btop, nvtop, rocm-smi (installed)

export HF_HUB_ENABLE_HF_TRANSFER=1
/srv/llm/venv/bin/hf download / [file.gguf ...] \
  --local-dir /srv/llm/models/

Then register it in the llama-swap config (Architecture page) and reload.

• sudo prints a parse error? It shouldn't — the broken sudoers fragment was fixed 2026-09-09 (see change log). The original is backed up.
• Model disk missing after reboot? It auto-mounts via fstab (UUID 4d7f50e0-…, nofail). If it doesn't: sudo mount /srv/llm/models
• GPU gone from llama-server? Check lsmod | grep amdgpu and re-run llama-server --version — it prints the detected device.
• Want everything back to as-found? The audit page is the baseline; every deviation from it is a change-log entry with its revert command.