Change Log

Handback documentation complete (START HERE manual live at handoff.html). Final verified state before power-off: all services healthy (llama-swap, rerank, Gateway, Qdrant, LightRAG, ComfyUI, Phoenix — whisper on-demand), both API keys live, docs deployed. On next boot everything self-recovers: wg user units + linger auto-start all services; Tailscale reconnects at 100.126.153.18; API-key auth remains on. Builder/contractor session ended — the box now belongs to its owner (wg).

Power on — everything self-recovers (documented 2026-09-10 move).

Key 1 (existing, builder): /srv/llm/secrets/api-key. Key 2 (new, owner): /srv/llm/secrets/api-key-2 (both wg:wg, 0600; values emailed to the builder). llama-swap now carries both keys top-level (apiKeys:) and comma-separated on every model's --api-key flag; the gateway's auth dependency now validates against both (gateway.py.bak-20260912-multikey backup). Verified: 401 without key, 200 with key 1, 200 with key 2, 401 with wrong key — via the gateway and llama-swap directly. Config rollback: llama-swap-config.yaml.bak-20260912-key2.

OpenWebUI: recreated with ENABLE_RAG_WEB_SEARCH=True + RAG_WEB_SEARCH_ENGINE=duckduckgo (6 results) — per-chat Web Search toggle, no keys, works with every model (search is OpenWebUI-side, injected as context).

Endpoint MCP layer: /srv/llm/mcp/servers.json gained two servers (backup .bak-20260912-websearch): web-search (uvx duckduckgo-mcp-server, 3 tools) and web-fetch (uvx mcp-server-fetch, 1 tool); the glm53-flash entry now also carries --mcp-servers-config. Verified by manual spawn: 5 MCP tools discovered + registered server-side. Honest caveat: llama.cpp MCP is experimental — structured tool_calls emission varies by model (Qwen3.8 emits call-JSON as plain content; GLM uses its own styled format). OpenWebUI's search does not depend on this layer. Full write-up: OpenWebUI guide §7.

Upgrade paths documented for the owner: Tavily (free 1k/mo, one key field) > Brave > SearXNG self-hosted (needs podman, owner/root task; settings.yml must enable JSON output + disable limiter).

ssh wg@box: cp /srv/llm/mcp/servers.json.bak-20260912-websearch /srv/llm/mcp/servers.json
# remove the two --mcp lines from the glm53-flash entry in /srv/llm/llama-swap-config.yaml
systemctl --user restart llama-swap
# OpenWebUI: docker rm -f fs2-openwebui (ashborn) and recreate without the RAG_WEB_SEARCH envs

OpenWebUI was not previously installed anywhere (the earlier "local test" record was the setup guide + screenshots). Installed fresh on ashborn as container fs2-openwebui (image ghcr.io/open-webui/open-webui:main v0.11.3, restart unless-stopped):

• Access: http://<ashborn-tailnet-ip>:3001 — reachable from any tailnet device (verified 200 over tailnet). First account to sign up = admin.
• LLM: pre-wired via env to THE GATEWAY http://100.126.153.18:8081/v1 (memory injection + persona + tracing ride along automatically — no OpenWebUI config needed for memory).
• Mediagen: comfyui.service STARTED on frame-serv-2 (was inactive; binds 127.0.0.1:8188). Exposed to the tailnet via tailscale serve --tcp 8188 (no firewalld change needed — dan's removal means root access is owner-only now). OpenWebUI wired IMAGE_GENERATION_ENGINE=comfyui + COMFYUI_BASE_URL; the ComfyUI workflow may still need selection in OpenWebUI admin → Images.
• RAG: LightRAG :9621 exposed to the tailnet the same way (tailscale serve --tcp 9621). OpenWebUI's built-in RAG uses its own local embeddings by default; switching embeddings to the gateway's embed model is an admin toggle (Settings → Documents).
• Rollback: docker rm -f fs2-openwebui; on the box tailscale serve --tcp=8188 off && tailscale serve --tcp=9621 off.

Pre-removal probe: no dan processes, zero /srv/llm files owned by dan, home 4 KB, all services under wg's systemd user units. Removed via detached systemd unit (loginctl disable-linger → stop [email protected] → userdel -r) — verified post: id dan → no such user, /home has only wg, llama-swap/gateway/qdrant untouched (same llama-swap PID). Access is now wg-only.

Phase 5 revised: Tailscale STAYS. Planned but NOT executed (would cut our own SSH): restrict tailnet exposure to AI ports only — 8081 (gateway), 8080 (admin/benchmark), 18190 (rerank), 8188 (ComfyUI), 9621 (LightRAG), 6006 (Phoenix); SSH/22 always remains. The rig's new owner will move it to a static IP and get the AI inference ports opened to his LAN subnet via firewalld (see OpenWebUI guide <BOX-IP> placeholders). Prime-staged video models: already pulled — item struck from the plan.

echo password | sudo -S useradd -m -u 1001 -G wheel dan && echo password | sudo -S passwd dan
# (home was empty; no files to restore)

Symptom: sd.cpp video renders died with vk::DeviceLostError during VAE decode, 6/6 repro, across distilled and dev models at 720p and 1080p. Root cause: the lockup_timeout=600 param (added 2026-09-10 to survive long single-pass renders) corrupts RADV compute submissions on gfx1151 — ring resets fire within 10–40 s regardless of the value. Verified by removal: control test that hung 6/6 now passes clean (13 s GPU decode), and the 1920×1088/33f headline rendered without a single amdgpu event. Do not re-add amdgpu.lockup_timeout to this box's cmdline. Trade-off documented: without the param, 121-frame-class long decodes revert to the original ~514 s ring-timeout signature — but 121f is blocked anyway by two hard memory walls (RADV maxBufferSize at 1080p, ~76 GB OOM at 720p). Full trail: research/gfx1151-sdcpp-ltx-debug.md (6 rounds).

echo password | sudo -S grubby --update-kernel=ALL --args="amdgpu.lockup_timeout=600"
echo password | sudo -S systemctl reboot
# NOT recommended: re-adds the decode hang. Only for a dedicated long-render boot experiment.

Symptom: every embed request evicted the resident 27B chat model and every chat request evicted embed — measured 2.4 s per embed search and 6–10 s per chat (model reload ping-pong). Root cause: llama-swap v248's group key is members:, not models: (GroupConfig struct tag yaml:"members", no aliasing — verified in the v248 source). Go's YAML unmarshal silently ignores unknown keys, so both group blocks were empty and all 7 models sat in the default group (swap:true, exclusive:true) — everything evicted everything. The 2026-09-10 "no swap-out post-fix" verification was invalid: the 3.3 s it measured WAS the reload.

Fix (config edit, round-trip verified — key set diff + comment count):
  chat:  {swap: true,  members: [6 chat models]}        # was models:
  embed: {swap: false, exclusive: false, persistent: true, members: [embed]}

Service: systemctl --user restart llama-swap.service
Verify (all pass): embed resident 20 ms (was 2.4 s, ~120×); chat #2 0.56 s (was
6.3–10.6 s); after either direction both models report loaded: true.
Rollback: cp /srv/llm/llama-swap-config.yaml.bak-20260911-exclusive /srv/llm/llama-swap-config.yaml
  && systemctl --user restart llama-swap.service

FastAPI gateway at /srv/llm/gateway/ (venv /srv/llm/media/gateway-venv, systemd user unit gateway.service, enabled, 158 MB RSS, binds 0.0.0.0:8081 — LAN-reachable for the post-handback client). Every OpenAI-compatible call now carries server-side layers:

• Memory (live): mem0 → Qdrant recall injected into the system context on request; new facts extracted + stored async on response. Verified round-trip.
• Tracing (live): one OTel span per call → Phoenix :6006 (service frame-serv2-gateway).
• Persona (live): /srv/llm/gateway/persona.txt applied when the client sends no system prompt.
• Auth (verified): 401 without/wrong API key on every /v1/* endpoint.
• Streaming (verified): SSE passes through byte-for-byte (53 chunks).

URL change: clients should point at :8081; raw :8080 stays the admin/benchmark lane. Grounding (LightRAG retrieval) and Screening (Prompt Guard 2) deferred — see the Roadmap.

Known issue surfaced: gateway latency measurement exposed an embed↔chat model-eviction ping-pong in llama-swap — root-caused and fixed same hour (see the 11:40 entry above: the groups' models: key was silently ignored). Post-fix the gateway's own overhead is ~21 ms. Full build log: research/gateway-build.md.

systemctl --user disable --now gateway.service
rm -rf /srv/llm/gateway /home/wg/.config/systemd/user/gateway.service /home/wg/.config/systemd/user/gateway.service.d
systemctl --user daemon-reload
echo password | sudo -S firewall-cmd --remove-rich-rule='rule family=ipv4 source address=100.64.0.0/10 port port=8081 protocol=tcp accept'
echo password | sudo -S firewall-cmd --runtime-to-permanent
# clients keep using :8080; Qdrant/mem0 data untouched (qdrant.service, /srv/llm/qdrant/storage)

Clean shutdown after all work state was persisted. On reboot at the new location:

• Tailscale reconnects automatically — tailnet IP stays 100.126.153.18, so SSH, the API endpoint, and this documentation's reachability are unchanged from any tailnet device.
• New LAN IP via DHCP on the home network (the 10.141.47.x configuration no longer applies).
• All services auto-start (wg user units, linger enabled). API-key auth remains on.
• One pending media item: the gfx1151 device wheel (GPU kernels blocked) — wheel staged on the operator's storage server; install is one command. See the build plan for the exact steps.

Power on at any network with internet — everything self-recovers.

ComfyUI now carries 4x-UltraSharp.pth (models/upscale_models/) for ESRGAN-class neural upscaling. Verified end-to-end: the blurry Z-Image 1280×720 output was upscaled to 5120×2880 in under a minute through the ComfyUI API (LoadImage → UpscaleModelLoader → ImageUpscaleWithModel → SaveImage). Output delivered to the operator's Telegram for quality review.

Why it matters: base image models (Z-Image 6B, Qwen-Image 20B) collapse above ~720-1280px native — the production path is generate-native + neural-upscale. The full 1080p-class test battery is running separately.

Also staged: RealESRGAN x4plus download failed (bad URL — skipped; UltraSharp is the primary). sd.cpp supports --upscale-model for the same weights on the non-ComfyUI lane.

rm /srv/llm/media/ComfyUI/models/upscale_models/4x-UltraSharp.pth

Redesigned from streaming panels to a chat-style flow: prompt box → each model runs sequentially (with live status) → each finished model gets an open output link showing the full transcript plus metrics (first-token latency, total, tokens, t/s, tool rounds). Run history persists in the browser. All model ids selectable; tool-use round optional. Replaces the earlier side-by-side streaming layout.

Verified in-browser: full pass on qwen38-27b — run completed, output page opened from the link with the transcript and metrics.

Previous layout is in git-untracked /srv/llm/playground/index.html history;
restore by editing /srv/llm/playground/index.html directly.

Auth upgrade: llama-swap's native apiKeys config now gates every endpoint including /v1/models (verified: 401 without key, 200 with). The per-model --api-key keys remain as a second layer. Verified reachability unchanged: tailnet-only.

Media Gen Roadmap: research complete — image (Z-Image-Turbo ~10 s/img, Apache 2.0), video (Wan 2.2, honest "occasional clips, not bulk" verdict), TTS (Kokoro), voice cloning (Chatterbox — the only license-clean SOTA option), STT (whisper.cpp Vulkan ~26× realtime). Install plan drafted, execution pending approval. Full evaluation: research/mediagen-eval.md · page: Media Gen Roadmap.

sed -i '/^apiKeys:/,+1d' /srv/llm/llama-swap-config.yaml
systemctl --user restart llama-swap

Served from the box itself at …ts.net/playground/ (same origin as the API — browser private-network policy blocks cross-origin calls from this public page, so the tool lives where the API lives; this page documents it). Mounted via tailscale serve --set-path=/playground. Static single-file app at /srv/llm/playground/.

Verified in-browser: model list populated, sequential run completed (qwen38-27b, streaming, timings + token counts rendered to the results table).

rm -rf /srv/llm/playground
sudo tailscale serve --set-path=/playground off

Auth: every model entry now carries --api-key (32-hex key stored at /srv/llm/secrets/api-key, wg:wg 0600, issued out-of-band). Verified: chat without key → 401; with key → 200. Note: /v1/models listing remains unauthenticated (llama-swap handles it before the model spawns — leaks model names, not compute).

Context: per-model maximums set from GGUF metadata and load-verified: qwen38-27b 262144 · flashnext (both quants) 262144 · glm53-flash 1048576 · dsv4-flash 1048576 (loaded + completed at 1M — no OOM) · llama-3.3-70b 131072 with q8_0 KV. All seven model ids answer on :8080 (load times: embed 2s → 70B 24s → 96–113 GB models 31–58 s).

HTTPS: tailscale serve now proxies https://frame-serv-2.tail8056c7.ts.net → :8080 (tailnet-only), enabling browser-based tools like the Playground to reach the API without mixed-content blocks.

Hermes: profile frameserv2 re-verified with the key — end-to-end completion returned.

cp /srv/llm/llama-swap-config.yaml.bak-20260910-auth /srv/llm/llama-swap-config.yaml
systemctl --user restart llama-swap
sudo tailscale serve --https=443 off

The inference endpoint (:8080) was unreachable from outside the box — firewalld's public zone only allowed ssh/mdns, and all prior tests were local-loopback. Added two rich rules scoped to the tailnet CGNAT range (100.64.0.0/10 only — LAN and public stay blocked) for :8080 and :18190, then verified from the operator's node:

curl http://100.126.153.18:8080/v1/models  →  HTTP 200
hermes --profile frameserv2 -z "Reply with exactly: ARISE"  →  ARISE

Working Hermes profile template + model-switching guide published on the new Hermes Integration page (profile installed at ~/.hermes/profiles/frameserv2/ on the operator's node).

sudo firewall-cmd --permanent --zone=public \
  --remove-rich-rule='rule family="ipv4" source address="100.64.0.0/10" port port="8080" protocol="tcp" accept'
sudo firewall-cmd --permanent --zone=public \
  --remove-rich-rule='rule family="ipv4" source address="100.64.0.0/10" port port="18190" protocol="tcp" accept'
sudo firewall-cmd --reload

Root cause (traced with an instrumented build): the GGUF's chat template threw on plain-string tool-response content — unguarded numeric subscripts (m.content.0.output) at two sites. The throw was swallowed by llama.cpp's capability probe (empty-message rethrown_exception), which made the analyzer report supports_tools: false and fall back to content-only parsing — so the model's correct tool invocations surfaced as plain text.

Fix: string-type guards at both sites in a template overlay (/srv/llm/templates/glm53-toolfix.jinja; original backed up as glm53-original.jinja.bak), wired into the glm53-flash llama-swap entry via --chat-template-file. New model entry in the router: glm53-flash (312B MoE, 128.5 t/s prefill, 14.1 t/s decode).

Verified: structured tool_calls end-to-end through the proxy — {"name":"get_weather","arguments":"{\"city\":\"Tokyo\"}"} with generated call ID. Debug instrumentation left in llama.cpp-glm (harmless, documented in research/glm53-template-fix.md — upstreamable finding).

sed -i '/glm53-flash/,/ttl: 0/d' /srv/llm/llama-swap-config.yaml
systemctl --user restart llama-swap
rm /srv/llm/templates/glm53-toolfix.jinja

Build from PR-27752 (verified against the GGUF's KV header) at /srv/llm/src/llama.cpp-glm. The 85.8 GB IQ2_XS loads and benches at 128.5 t/s prefill — decode numbers land when the detached bench completes. This was the last blocked model in the requested slate.

rm -rf /srv/llm/src/llama.cpp-glm

Sequential llama-bench over every staged model (orchestrator: /srv/llm/bench/run-all-models.sh):

• Flash-Next IQ1_S (176B MoE): 31.0 t/s decode — beats the 27B dense by 2.5× and the 70B dense by 6×
• Llama-3.3-70B: 5.2 t/s — research estimate (5–6) confirmed empirically
• Flash-Next ROCmFP4: blocked — qwen4exp packed GGUF unparseable by stock master; repo patches target an old commit and conflict with current tree (applied, assessed, reverted — tree kept clean)
• GLM-5.3-Flash: blocked — empirically confirmed, needs the unmerged PRs

Vision capability also verified this session: the 27B accurately describes chart images via the mmproj projector.

N/A — read-only measurements; logs under /srv/llm/logs/

Benchmark verdict: Vulkan beats HIP on both prefill (359 vs 348 t/s) and decode (12.3 vs 12.0 t/s) for Qwen3.8-27B — llama-swap-config.yaml set to the Vulkan lane with --spec-type draft-mtp (embedded draft head), KV cache q4_0/q4_0, DRY sampling block, vision mmproj wired in. systemd user units llama-swap + llama-rerank enabled; linger enabled for wg so services survive logout/reboot. Previous config backed up as llama-swap-config.yaml.bak-20260909-1404.

Serving result: 19.25 t/s decode with MTP (+57% over bare 12.3). Smoke tests passed: chat, tool calling, embeddings (1024-dim), rerank. Full numbers on the Benchmarks page.

systemctl --user disable --now llama-swap llama-rerank
sudo loginctl disable-linger wg
cp /srv/llm/llama-swap-config.yaml.bak-20260909-1404 /srv/llm/llama-swap-config.yaml

llama-bench, 3 runs each backend, both builds from commit 22397c3 on the 124 GiB pool. Vulkan won. Results and reproduction commands on the Benchmarks page; raw logs at /srv/llm/logs/qwen27b-bench-20260909-1435.log.

N/A — read-only measurement (logs only)

Rebooted after the 332 GB model copy completed (rsync, 105 files, exit 0, ~406 MB/s from the operator's USB stick). Post-boot verification:

cat /proc/cmdline → ... amdgpu.gttsize=126976 ttm.pages_limit=32505856
sysfs gtt_mem_total → 124 GiB   (was ~63 GiB default)
sysfs mem_info_vram_total → ~0  (BIOS UMA Frame Buffer set to 512M, not AUTO)

BIOS change (made by the owner at the F2 prompt): UMA Frame Buffer Size → 512M. This BIOS menu setting is not scriptable remotely — the owner performed it; nothing else in BIOS was touched.

rocm-smi post-boot: Device 0 (gfx1151) healthy, 45°C, 7.8 W idle.

BIOS: restore UMA Frame Buffer Size → Auto (owner action)
Kernel: see the 11:30 change-log entry for the grub revert

rsync from the operator's USB stick (label LLM-USB): GLM-5.3-Flash IQ2_XS, Qwen3.8-Flash-Next ROCmFP4 + IQ1_S sets, Llama-3.3-70B Q4_K_M, Qwen3.8-27B complete set, Qwen3-Embedding-0.6B, bge-reranker-v2-m3 (4 quants). 105 files, USB_COPY_EXIT=0.

rm -rf /srv/llm/models/{GLM-5.3-Flash-IQ2_XS,Flash-Next-ROCmFP4-STRIX,Llama-3.3-70B,Qwen3.8-Flash-Next,Qwen3.8-27B,Qwen3-Embedding-0.6B,bge-reranker-v2-m3}-GGUF

The staged reranker GGUF is named BGE-Reranker-v2-M3-Q8_0.gguf (capital BGE), but the unit referenced lowercase bge-reranker-v2-m3-Q8_0.gguf — it would have failed on first start. Unit corrected to:

-m /srv/llm/models/bge-reranker-v2-m3-GGUF/BGE-Reranker-v2-M3-Q8_0.gguf

sed -i 's|BGE-Reranker-v2-M3-Q8_0.gguf|bge-reranker-v2-m3-Q8_0.gguf|' \
  ~/.config/systemd/user/llama-rerank.service

The hotspot could not sustain the 13 MB GitHub download (three attempts), so the binary was relayed from ashborn-prime — the exact build v248 that prime runs in production, rather than the just-released v255. Deliberate choice: production-proven beats newest.

rm /srv/llm/bin/llama-swap   # or replace with a newer release build

The 17 GB main GGUF failed three times over the cell hotspot (hf_transfer CAS reconstruction error, then stalled plain-mode downloads). Completed over hotspot: mmproj-F16 vision (885 MB) + MTP draft (1.3 GB). The main GGUF is identical on ashborn-prime and will arrive on the operator's USB stick with the other staged models. Stale partial blobs were cleared.

N/A — no machine state changed beyond removing partial downloads.
Restart hotspot download any time:
  /srv/llm/venv/bin/hf download unsloth/Qwen3.8-27B-GGUF \
    Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir /srv/llm/models/Qwen3.8-27B-GGUF

By default amdgpu caps GPU-visible memory (GTT) at ~50% of RAM (~63 GiB) — the reason 100 GB models fail to load on 128 GiB machines. Added to the kernel command line (verified present in /boot/grub2/grub.cfg):

amdgpu.gttsize=126976 ttm.pages_limit=32505856

Why: the two values form an inseparable pair describing the same 124 GiB pool in different units (MB / 4 KiB pages). Sourced from AMD's ROCm Strix Halo system-optimization guidance and Framework community threads endorsed by AMD engineers. Active after the next reboot (deferred until the model download finishes).

Original cmdline: rhgb quiet (backup: /etc/default/grub.bak-20260909).

sudo cp /etc/default/grub.bak-20260909 /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo systemctl reboot

llama-swap binary at /srv/llm/bin/; config /srv/llm/llama-swap-config.yaml defines qwen38-27b (65K ctx, flash-attn, jinja, DRY-sampling block, parallel 2, kv-unified, 99 GPU layers) and an embed endpoint (Qwen3-Embedding-0.6B Q8_0). User units llama-swap.service (proxy :8080) and llama-rerank.service (bge-reranker-v2-m3 Q8_0 CPU, :18190) installed for wg — mirroring the proven ashborn-prime topology. Activation pending model downloads.

rm /srv/llm/llama-swap-config.yaml /srv/llm/bin/llama-swap
rm ~/.config/systemd/user/llama-swap.service ~/.config/systemd/user/llama-rerank.service

From HuggingFace unsloth/Qwen3.8-27B-GGUF: UD-Q4_K_XL main (17 GB) + mmproj-F16 vision (885 MB) + MTP draft head (1.3 GB), ~19.2 GB total, via the hf CLI. Destination: /srv/llm/models/Qwen3.8-27B-GGUF/. Files land dan-owned while the download runs (launched before the wg ownership transfer) — a final chown -R wg:wg /srv/llm runs when it completes.

rm -rf /srv/llm/models/Qwen3.8-27B-GGUF   # weights only; re-downloadable from HF

Installed python3-pip (dnf), created /srv/llm/venv (Python 3.14) with huggingface_hub 1.30.0 + hf_transfer. Used for model downloads; no system Python was touched.

sudo rm -rf /srv/llm/venv
sudo dnf remove python3-pip

Fedora's llama.cpp RPM (b6153, Jan 2026) is 8 months stale and cannot load any of the target models. Built current master (commit 22397c3, 2026-09-09 — includes the Strix Halo Vulkan mat-vec tuning and the Qwen3.8 top-k radix fix) from /srv/llm/src/llama.cpp:

• build-vulkan — -DGGML_VULKAN=ON — built OK
• build-hip — -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DGGML_HIP_NO_VMM=ON — built OK

Source stays at /srv/llm/src/llama.cpp for rebuilds; binaries in the two build dirs. Nothing overwritten — the distro RPM remains installed for reference.

rm -rf /srv/llm/src/llama.cpp
sudo dnf remove git cmake clang compiler-rt vulkan-loader-devel vulkan-headers \
  glslc glslang-devel spirv-tools-devel spirv-headers-devel \
  rocm-hip-devel hipblas-devel hipblas-common-devel hipblaslt-devel rocblas-devel

chown -R wg:wg /srv/llm — the entire inference tree (src, venv, models) is owned by wg, so everything keeps working after the dan account is removed at handback. An admin SSH public key (dxverm@ashborn ed25519) was installed into /home/wg/.ssh/authorized_keys for key-based remote administration.

Disclosure: a first, malformed attempt at the key write may have truncated a pre-existing /home/wg/.ssh/authorized_keys if the owner had installed one on 2026-09-06. No other keys were present in the file afterwards; password SSH for wg is untouched, so owner access is unaffected. If you had a key there, re-add it.

sudo chown -R dan:dan /srv/llm
sudo sh -c 'echo -n > /home/wg/.ssh/authorized_keys'   # or restore your own key

dnf install htop btop nvtop tmux lm_sensors sysstat ripgrep unzip tar — pulled in rocm-smi as an nvtop dependency (bonus: GPU monitoring CLI now present).

Why: operator-side monitoring of GPU/thermals during benchmarks.

sudo dnf remove htop btop nvtop tmux lm_sensors sysstat ripgrep unzip tar

Full dnf upgrade -y installed a new kernel; reboot required to activate. Post-reboot verified: amdgpu + amdxdna loaded, llama-server still initializes gfx1151, /srv/llm/models auto-mounted from fstab.

Why: run the updated kernel + firmware before any inference work.

Boot the previous kernel from the GRUB menu (6.19.10-300 still installed).
Old kernel packages can be re-installed if ever removed:
  sudo dnf install kernel-6.19.10-300.fc44

First imported the Tailscale repository signing key (rpm --import https://pkgs.tailscale.com/stable/fedora/repo.gpg), which had been blocking non-interactive dnf runs. Then sudo dnf upgrade -y: systemd 259.8, kernel 7.1.13, linux-firmware 20260810, sudo, tzdata, and the rest of the pending set. dnf history entry #11.

Why: the install was 3 days old with all updates pending, including security fixes (sudo, systemd, kernel).

Package updates are not individually reversible by design.
Rollback anchor: dnf history list  (transaction #11, 2026-09-09 14:54 UTC)
  sudo dnf history rollback 10   # state before this upgrade (includes the
                                 # tailscale key import, which is harmless)

nvme0n1 (Samsung 970 EVO Plus 2TB) was re-formatted mkfs.ext4 -L LLM-MODELS and mounted at /srv/llm/models with a persistent fstab entry (defaults,noatime,nofail), then verified post-reboot. New UUID: 4d7f50e0-24a2-44ac-af2c-5173028f94cd.

Wiped content: a stale .cache directory owned by uid 989 (~77 GB) left over from a previous operating system on this disk — no user documents (verified by read-only mount before wiping, per operator instruction).

Why: dedicated home for model weights, isolated from the OS disk.

sudo sed -i '\|/srv/llm/models|d' /etc/fstab
sudo umount /srv/llm/models
sudo rmdir /srv/llm/models
# NOTE: the stale 77G uid-989 cache is NOT recoverable after this format
# (it was inspected read-only first and held only browser/tool cache data)

The as-found file contained invalid syntax (dan (ALL) ALL=ALL) which made sudo print a parse error on every invocation. Replaced with the valid form.

Before:

dan (ALL) ALL=ALL        # invalid syntax

After:

dan ALL=(ALL) ALL

Verified with sudo visudo -c — no errors. Note: dan is also a member of wheel, so this file is technically redundant; it is kept because it was the owner's explicit grant. The original file is preserved on the machine.

sudo cp /etc/sudoers.d/dan.bak-20260909-orig /etc/sudoers.d/dan
# (backup filename contains dots, so sudoers ignores it as a rule file)