Runbook / How to Use

1. Start here — this page. It tells you what exists and what to type.
2. System Audit — the machine's as-found state, before any of this work. If something surprises you, compare against the audit.
3. Change Log — everything that was done, newest first, each entry with why and a revert procedure. This is the receipt page: if you want to undo or tweak anything, start there.
4. Architecture / Models / Benchmarks — the design, the picks and their trade-offs, and the measured numbers with reproduction commands.
5. Anything the setup work does from now on lands in the change log the same day. The site is rebuilt and redeployed on every update.

ssh wg@frame-serv-2          # from any device on the tailnet (MagicDNS)
ssh [email protected]        # same node, by IP

An admin SSH key (from the operator's machine, dxverm@ashborn) is installed for wg. Your own password still works. wg has full sudo (wheel); the inference tree /srv/llm is owned by wg.

/srv/llm/src/llama.cpp/          production llama.cpp (vulkan/ + hip/ builds)
/srv/llm/src/llama.cpp-fp4/      pinned branch for Flash-Next FP4 (experimental)
/srv/llm/src/llama.cpp-glm/      GLM PR-branch build (experimental)
/srv/llm/models/                 model weights (2TB NVMe, ext4, "LLM-MODELS" label)
/srv/llm/bench/  ·  /srv/llm/logs/   benchmark scripts + raw run logs
/srv/llm/venv/                   python env with the hf CLI (model downloads)
/etc/sudoers.d/dan.bak-20260909-orig   original sudoers fragment (pre-fix)

Agent integration: Hermes profile template + model ids + reachability matrix at Hermes Integration. Endpoint of record: http://frame-serv-2:8081/v1 (gateway — memory + tracing + persona; tailnet for the builder, LAN for the owner post-handback). Raw admin/benchmark lane: :8080/v1.

• Start/stop serving: systemctl --user start|stop llama-swap llama-rerank (as wg)
• Hit the API: curl http://localhost:8081/v1/chat/completions -H "Authorization: Bearer $(cat /srv/llm/secrets/api-key)" -H "Content-Type: application/json" -d '{"model":"qwen38-27b","messages":[…]}' (all endpoints require a key — raw admin lane: :8080)
• Available model ids: qwen38-27b (27B dense + MTP, vision, tools, MCP), flashnext-iq1s (Flash-Next MoE, vision, tools), flashnext-fp4 (ROCmFP4), glm53-flash (GLM-5.3-Flash, tool-calls fixed via template overlay), dsv4-flash (DeepSeek-V4-Flash, 1M ctx), llama-3.3-70b (70B dense, q8 KV), embed (embeddings, always-resident)
• Rerank: POST http://localhost:18190/v1/rerank with {"query":…,"documents":[…]}
• llama-swap loads models on demand and keeps them resident (ttl 0) — first request per model pays the load
• Monitor: btop, nvtop, rocm-smi; live graphs at the Live Status page
• Config: /srv/llm/llama-swap-config.yaml — edit, then systemctl --user restart llama-swap

export HF_HUB_ENABLE_HF_TRANSFER=1
/srv/llm/venv/bin/hf download / [file.gguf ...] \
  --local-dir /srv/llm/models/

Then register it in the llama-swap config (Architecture page) and reload.

The CPU ships in powersave (cores pinned near 1.5 GHz), which measurably drags prefill and MTP-draft speed on the served models. One-shot fix (as wg, with your sudo password):

echo <sudo-password> | sudo -S bash -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > $g; done'

To make it permanent across reboots, install a systemd unit once:

echo <sudo-password> | sudo -S tee /etc/systemd/system/cpu-performance.service >/dev/null <<'UNIT'
[Unit]
Description=Set CPU governor to performance
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > $g; done'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
UNIT
echo <sudo-password> | sudo -S systemctl daemon-reload
echo <sudo-password> | sudo -S systemctl enable --now cpu-performance.service
# verify:  cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor   → performance

Rollback: sudo systemctl disable --now cpu-performance.service (governor returns to the kernel default). Verify the gain with any chat completion and reading timings.predicted_per_second in the response.

• sudo prints a parse error? It shouldn't — the broken sudoers fragment was fixed 2026-09-09 (see change log). The original is backed up.
• Model disk missing after reboot? It auto-mounts via fstab (UUID 4d7f50e0-…, nofail). If it doesn't: sudo mount /srv/llm/models
• GPU gone from llama-server? Check lsmod | grep amdgpu and re-run llama-server --version — it prints the detected device.
• Network down / node unreachable? The uplink is the phone's ethernet tether — reconnecting the phone restores tailnet access. No local action needed.
• Want everything back to as-found? The audit page is the baseline; every deviation from it is a change-log entry with its revert command.