START HERE

A Framework Desktop (AMD Ryzen AI MAX+ 395, Radeon 8060S iGPU, 128 GiB unified memory) running Fedora 44. It serves seven model ids through llama.cpp — from a 27B daily driver up to 313B MoE reasoning models — behind one OpenAI-compatible gateway (:8081) that adds persistent memory, tracing, and a server-side persona. A mediagen stack (image + video generation via ComfyUI and stable-diffusion.cpp), a reranker, and document RAG (LightRAG) run alongside. Full design and service layout: Architecture.

SSH as the wg user — the box's single admin account (wheel sudo, your password). <BOX-IP> below is wherever the box currently sits; step 3 makes it a fixed address.

ssh wg@<BOX-IP>          # password: your password (wheel sudo)
sudo -v                  # confirm sudo works

The setup contractor retains tailnet-only access (ssh [email protected], key auth) and has no root on the box.

Give the box a static IP in your scheme — either a static lease in your router, or on the box itself:

sudo nmtui                       # edit the wired connection → manual IPv4
ip -br addr show enp191s0        # confirm the new address
# from your workstation:
ssh wg@<BOX-IP>                  # confirm SSH works over your LAN

All steps below use <BOX-IP> — replace it with the address you just assigned.

The shipped firewall rules scope the AI ports to the build network's tailnet range only. Open them to your subnet with one rich rule per port:

for PORT in 8080 8081 18190 8188 9621; do
  sudo firewall-cmd --permanent --zone=public \
    --add-rich-rule="rule family=\"ipv4\" source address=\"<YOUR-SUBNET>\" port port=\"$PORT\" protocol=\"tcp\" accept"
done
sudo firewall-cmd --reload
# <YOUR-SUBNET> = your LAN, e.g. 192.168.1.0/24

Bind state today (verify with ss -tlnp | grep -E ':(8080|8081|18190|8188|9621)'): the gateway (:8081) and llama-swap (:8080) listen on all interfaces — the firewall rule is all they need. The reranker (:18190), ComfyUI (:8188), and LightRAG (:9621) bind 127.0.0.1 and are tailnet-exposed via tailscale serve; to reach those three from your LAN, change each service's bind to 0.0.0.0 (config lives in the systemd user units — systemctl --user edit) and reload. Public internet stays blocked. Background: Roadmap 5.3.

The CPU ships in powersave, which measurably drags prefill and MTP-draft speed. One-shot fix:

echo <sudo-password> | sudo -S bash -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > $g; done'

Make it permanent with a systemd unit (once):

echo <sudo-password> | sudo -S tee /etc/systemd/system/cpu-performance.service >/dev/null <<'UNIT'
[Unit]
Description=Set CPU governor to performance
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > $g; done'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
UNIT
echo <sudo-password> | sudo -S systemctl daemon-reload
echo <sudo-password> | sudo -S systemctl enable --now cpu-performance.service
# verify:  cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor   → performance

Full context: Runbook.

Two independent keys are live — both work on every endpoint, both were emailed to the builder on 2026-09-12:

To rotate a key: write the new value into the file (plain text, one line), then restart the services that read it:

nano /srv/llm/secrets/api-key-2       # your key
systemctl --user restart llama-swap gateway
# verify:  curl -s http://localhost:8081/v1/models -H "Authorization: Bearer $(cat /srv/llm/secrets/api-key-2)" | head -c 200

Rotating the builder key cuts the contractor's access — coordinate first if both keys are in use.

Point any OpenAI-compatible client (OpenWebUI, or anything else) at the gateway:

URL:  http://<BOX-IP>:8081/v1
Key:  contents of /srv/llm/secrets/api-key-2

OpenWebUI is not installed on this box — you run it on your own machine (Docker or pip, 5 minutes). The full walk-through, including RAG, image generation, and web-search wiring: OpenWebUI guide →

• Chat — send one prompt to each model id: qwen38-27b (daily driver) · flashnext-iq1s / flashnext-fp4 (176B MoE) · glm53-flash (313B reasoning) · dsv4-flash (1M context) · llama-3.3-70b. Or use the Playground to run one prompt through all of them sequentially (tailnet required).
• Web search — in OpenWebUI, toggle the ＋ → Web Search switch in a chat (DuckDuckGo engine, pre-wired).
• Image gen — OpenWebUI image button via ComfyUI (:8188); settings in the OpenWebUI guide §4, timings on Media Gen (~55 s per SDXL image).
• Rerank — POST http://<BOX-IP>:18190/v1/rerank with {"query":"…","documents":["…","…"]}.

• Do not re-add amdgpu.lockup_timeout to the kernel command line. It was removed 2026-09-11 13:05 (see change log) after it was root-caused to hang every large video-VAE decode on RADV/gfx1151. Re-adding it breaks video generation, not fixes it.
• The 121-frame clip limit is upstream, not a misconfiguration. LTX 121-frame clips hit two hard memory walls (RADV maxBufferSize at 1080p-class, ~76 GB OOM at 720p). The reliable video ceiling is 1920×1088/33f; longer clips need an upstream tiled-VAE decoder. Details: Media Gen.
• Don't treat raw :8080 as the client endpoint. It is the admin/benchmark lane and skips memory/tracing/persona — clients point at :8081.