Change Log
Anything the setup work did to this machine appears here. Backups of replaced
files live on the machine itself (names ending in .bak-…, which their
consumers ignore); raw command transcripts are archived with the build docs.
Kernel 7.1.13-200 (updated from 6.19.10-300, rebooted) · /srv/llm/models live on the 2TB NVMe · baseline ops tools installed · llama.cpp verified on gfx1151 after reboot · fwupd reports 10 updatable firmware devices (BIOS flash deferred — owner decision, see below).
From HuggingFace unsloth/Qwen3.8-27B-GGUF: UD-Q4_K_XL main
(17 GB) + mmproj-F16 vision (885 MB) + MTP draft head (1.3 GB),
~19.2 GB total, via the hf CLI. Destination:
/srv/llm/models/Qwen3.8-27B-GGUF/. Files land dan-owned while the
download runs (launched before the wg ownership transfer) — a final
chown -R wg:wg /srv/llm runs when it completes.
rm -rf /srv/llm/models/Qwen3.8-27B-GGUF # weights only; re-downloadable from HF
Installed python3-pip (dnf), created /srv/llm/venv
(Python 3.14) with huggingface_hub 1.30.0 + hf_transfer.
Used for model downloads; no system Python was touched.
sudo rm -rf /srv/llm/venv sudo dnf remove python3-pip
Fedora's llama.cpp RPM (b6153, Jan 2026) is 8 months stale and cannot load any of
the target models. Built current master (commit 22397c3, 2026-09-09 —
includes the Strix Halo Vulkan mat-vec tuning and the Qwen3.8 top-k radix fix)
from /srv/llm/src/llama.cpp:
build-vulkan—-DGGML_VULKAN=ON— built OKbuild-hip—-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DGGML_HIP_NO_VMM=ON— built OK
Source stays at /srv/llm/src/llama.cpp for rebuilds; binaries in the
two build dirs. Nothing overwritten — the distro RPM remains installed for reference.
rm -rf /srv/llm/src/llama.cpp sudo dnf remove git cmake clang compiler-rt vulkan-loader-devel vulkan-headers \ glslc glslang-devel spirv-tools-devel spirv-headers-devel \ rocm-hip-devel hipblas-devel hipblas-common-devel hipblaslt-devel rocblas-devel
chown -R wg:wg /srv/llm — the entire inference tree (src, venv,
models) is owned by wg, so everything keeps working after the
dan account is removed at handback. An admin SSH public key
(dxverm@ashborn ed25519) was installed into
/home/wg/.ssh/authorized_keys for key-based remote administration.
Disclosure: a first, malformed attempt at the key write may have
truncated a pre-existing /home/wg/.ssh/authorized_keys if the owner had
installed one on 2026-09-06. No other keys were present in the file afterwards;
password SSH for wg is untouched, so owner access is unaffected. If you
had a key there, re-add it.
sudo chown -R dan:dan /srv/llm sudo sh -c 'echo -n > /home/wg/.ssh/authorized_keys' # or restore your own key
dnf install htop btop nvtop tmux lm_sensors sysstat ripgrep unzip tar
— pulled in rocm-smi as an nvtop dependency (bonus: GPU monitoring CLI now present).
Why: operator-side monitoring of GPU/thermals during benchmarks.
sudo dnf remove htop btop nvtop tmux lm_sensors sysstat ripgrep unzip tar
Full dnf upgrade -y installed a new kernel; reboot required to activate.
Post-reboot verified: amdgpu + amdxdna loaded, llama-server still initializes gfx1151,
/srv/llm/models auto-mounted from fstab.
Why: run the updated kernel + firmware before any inference work.
Boot the previous kernel from the GRUB menu (6.19.10-300 still installed). Old kernel packages can be re-installed if ever removed: sudo dnf install kernel-6.19.10-300.fc44
First imported the Tailscale repository signing key
(rpm --import https://pkgs.tailscale.com/stable/fedora/repo.gpg), which had been
blocking non-interactive dnf runs. Then sudo dnf upgrade -y:
systemd 259.8, kernel 7.1.13, linux-firmware 20260810, sudo, tzdata, and the rest
of the pending set. dnf history entry #11.
Why: the install was 3 days old with all updates pending, including security fixes (sudo, systemd, kernel).
Package updates are not individually reversible by design.
Rollback anchor: dnf history list (transaction #11, 2026-09-09 14:54 UTC)
sudo dnf history rollback 10 # state before this upgrade (includes the
# tailscale key import, which is harmless)nvme0n1 (Samsung 970 EVO Plus 2TB) was re-formatted
mkfs.ext4 -L LLM-MODELS and mounted at /srv/llm/models with
a persistent fstab entry (defaults,noatime,nofail), then verified
post-reboot. New UUID: 4d7f50e0-24a2-44ac-af2c-5173028f94cd.
Wiped content: a stale .cache directory owned by
uid 989 (~77 GB) left over from a previous operating system on this disk — no
user documents (verified by read-only mount before wiping, per operator instruction).
Why: dedicated home for model weights, isolated from the OS disk.
sudo sed -i '\|/srv/llm/models|d' /etc/fstab sudo umount /srv/llm/models sudo rmdir /srv/llm/models # NOTE: the stale 77G uid-989 cache is NOT recoverable after this format # (it was inspected read-only first and held only browser/tool cache data)
The as-found file contained invalid syntax (dan (ALL) ALL=ALL) which made
sudo print a parse error on every invocation. Replaced with the valid form.
Before:
dan (ALL) ALL=ALL # invalid syntax
After:
dan ALL=(ALL) ALL
Verified with sudo visudo -c — no errors. Note: dan is also a
member of wheel, so this file is technically redundant; it is kept because it
was the owner's explicit grant. The original file is preserved on the machine.
sudo cp /etc/sudoers.d/dan.bak-20260909-orig /etc/sudoers.d/dan # (backup filename contains dots, so sudoers ignores it as a rule file)
- BIOS + device firmware updates (fwupd: 10 devices) — flashing firmware requires an unattended reboot cycle on hardware that belongs to the owner; deferred until the owner confirms. The UEFI dbx update is additionally blocked by low efivarfs space.
- SSH hardening (key-only auth) — deferred until the owner confirms he no longer needs password SSH for his own access.
- Disk encryption — none on either NVMe; an owner decision (changing it now is cheap, later is expensive).