# Troubleshooting & lessons from the first end-to-end install Every failure mode below was hit in practice during the first end-to-end install on a real Omarchy/NVIDIA/Wayland host. The fixes landed in this repo so future installs (e.g. on the Framework) shouldn't repeat them — this doc explains the *why* so you can recognize them if they come back with a twist. The "Real failures hit on first install" section is in roughly the order they surfaced. --- ## Real failures hit on first install ### 1. `sunshine-bin` breaks on rolling Arch (library drift) **Symptom** ``` sunshine: error while loading shared libraries: libicuuc.so.76: cannot open shared object file: No such file or directory ``` Service hits the systemd start-limit after five rapid failures and won't try again until `reset-failed`. **Cause** The AUR `sunshine-bin` package ships a binary built against whatever ICU was current at AUR-build time (ICU 76). Arch had moved to ICU 78. Library SONAME is hard-pinned at build time, so the binary fails to load any of its ICU calls. This recurs whenever Arch bumps a major library and the `sunshine-bin` maintainer hasn't rebuilt yet. **Fix in this repo** `lib/packages.sh` → `install_sunshine` now runs `ldd $(command -v sunshine)` after install. If anything is "not found", removes `sunshine-bin` (and `sunshine-bin-debug`) and rebuilds from source. The source-build linking pulls in the *current* ICU, so it works as long as Arch ships any version. `lib/service.sh` → `enable_sunshine_service` runs `systemctl --user reset-failed` before each restart, so a previous failure doesn't preempt the recovery. **Manual recovery if you hit it on a non-installer flow** ```bash sudo pacman -Rns --noconfirm sunshine-bin sunshine-bin-debug yay -S --needed --noconfirm sunshine sudo setcap cap_sys_admin+p "$(readlink -f "$(command -v sunshine)")" systemctl --user reset-failed sunshine.service ``` ### 2. Source build is slow (~10 min); cached pkg.tar.zst install collides on debug symbols **Symptom** ``` error: failed to commit transaction (conflicting files) sunshine-debug: /usr/lib/debug/usr/bin/sunshine.debug exists in filesystem (owned by sunshine-bin-debug) ``` After a successful `makepkg`, `pacman -U` of the resulting `.pkg.tar.zst` refuses to install. **Cause** `sunshine-bin` ships with a separate `sunshine-bin-debug` package for debug symbols, owned by `sunshine-bin-debug`. The source build's `sunshine-debug` package wants to put its symbols at the same path. Removing `sunshine-bin` doesn't auto-remove its `-debug` sibling. **Fix in this repo** `lib/packages.sh` removes both `sunshine-bin` AND `sunshine-bin-debug` before the source rebuild. `uninstall.sh` similarly drops all four variants (`sunshine`, `sunshine-debug`, `sunshine-bin`, `sunshine-bin-debug`). ### 3. Source `sunshine` doesn't ship `sunshine.service` **Symptom** ``` Failed to restart sunshine.service: Unit sunshine.service not found. ``` After replacing `sunshine-bin` with the source `sunshine` build, the service that worked before is gone. **Cause** The AUR source PKGBUILD installs its systemd unit under a Flatpak-style reverse-DNS filename: `app-dev.lizardbyte.app.Sunshine.service`. The `sunshine-bin` variant ships `sunshine.service` directly. **Fix in this repo** `lib/service.sh` → `ensure_sunshine_unit_present` resolves the actual unit name in three steps: 1. Look for `sunshine.service` in every systemd-user lookup path. 2. If not found, look for `app-dev.lizardbyte.app.Sunshine.service` and use that name directly (we no longer try to symlink it as `sunshine.service` — systemd refuses to `enable` linked unit files). 3. Last resort: install the repo's `files/sunshine.service` fallback unit. The drop-in path (for the headless prestart hook) is computed against the resolved unit name so `.service.d/` always lands in the right place. ### 4. The fatal-but-misleading "Unable to find display or encoder during startup" **Symptom** ``` [wlgrab] -------- Start of Wayland monitor list -------- [wlgrab] --------- End of Wayland monitor list --------- Error: Unable to initialize capture method ... Encoder [nvenc] failed Encoder [vulkan] failed Encoder [vaapi] failed Encoder [software] failed Fatal: Unable to find display or encoder during startup. Fatal: Please ensure your manually chosen GPU and monitor are connected ``` Visible in the web UI as red banner warnings. Streaming may still appear to work later because Sunshine re-probes at stream time, but the startup state is broken. **Cause** In headless mode, `sunshine.conf` pins `output_name = HEADLESS-1` and `capture = wlr`. At service start, Sunshine connects to Wayland, enumerates outputs, and finds *nothing* — because the Hyprland headless output is created by the per-stream prep-cmd, not at boot. With no output, every encoder probe fails on platform-init. **Fix in this repo** `bin/sunshine-prestart.sh` runs as `ExecStartPre` for the Sunshine service. It guarantees one (and exactly one) Hyprland headless output exists *before* Sunshine probes encoders. It also rewrites `sunshine.conf`'s `output_name` line to match whatever name Hyprland assigned the surviving headless (see issue 6). The drop-in lives at `~/.config/systemd/user/.service.d/headless-prestart.conf`, managed by `lib/headless.sh` → `install_headless_prestart_dropin`. The `bin/sunshine-stream-undo.sh` was also changed to *keep* the headless across stream sessions — create/destroy per stream raced with the startup encoder probe. ### 5. Web UI 0.0.0.0 binding was open to the LAN **Symptom** By default Sunshine binds the web UI on `0.0.0.0:47990`. Anyone on your LAN who can reach the host can hit the admin panel; only a username/password guards it. **Fix in this repo** `lib/config.sh` writes `origin_web_ui_allowed = pc`. Sunshine then rejects web-UI requests from any source other than localhost regardless of bind address. Streaming/pairing on port 47989 stays LAN-accessible — only the admin surface is locked down. **Recommended access pattern (also documented inline in `sunshine.conf`):** ```bash # From a client machine: ssh -L 47990:127.0.0.1:47990 user@host # Then browse to: https://localhost:47990 ``` The host cert SANs include `localhost` and `127.0.0.1` (see issue 11) so the tunneled URL doesn't trigger hostname-mismatch warnings. ### 6. Hyprland headless counter never decrements **Symptom** After several restarts, `hyprctl monitors all` shows `HEADLESS-4`, `HEADLESS-9`, `HEADLESS-10`, ... `Selected monitor [] for streaming` in the journal — Sunshine can't match `output_name = HEADLESS-1` against any existing output. **Cause** Hyprland's `hyprctl output create headless` ignores the optional name argument (at least through 0.54.x) and auto-numbers as `HEADLESS-N`. The counter is monotonic across the Hyprland session — removing `HEADLESS-1` and re-creating gives you `HEADLESS-2`, not `HEADLESS-1`. Combine that with an old version of the prep-cmd script that hardcoded `MON=HEADLESS-1` and created a new output every connect, and you accumulate a herd. **Fix in this repo** Two-layer: 1. `bin/sunshine-prestart.sh` actively *removes* extras, keeping only the lowest-numbered `HEADLESS-N`. Then it rewrites `sunshine.conf`'s `output_name` to that surviving name. Self-healing. 2. `bin/sunshine-stream-do.sh` and `sunshine-stream-undo.sh` now discover the existing headless name via `jq` instead of hardcoding `HEADLESS-1`. Resize and workspace-migration target the actual output. ### 7. ufw was active but our detection missed it **Symptom** `ARP` between Mac and the host worked fine (layer 2 visible). TCP to port 47989 hung from the Mac. `nmap -p 47989 192.168.x.y` said "Host seems down." **Cause** `lib/firewall.sh` → `_ufw_active` previously ran `ufw status | grep -q "Status: active"`. `ufw status` requires root to read `/etc/ufw` state, so an unprivileged check returned nothing → returned false → we silently skipped opening Sunshine's ports. ufw was happily dropping inbound packets. **Fix in this repo** `_ufw_active` now uses `systemctl is-active ufw.service`. ufw runs as a Type=oneshot unit with RemainAfterExit, so `is-active` reports `active` while the rules are loaded — and the check works without sudo. **One-shot LAN allow if you hit it** ```bash for port in 47984 47989 47990 48010; do sudo ufw allow from 192.168.x.0/24 to any port $port proto tcp done for port in 47998 47999 48000 48010; do sudo ufw allow from 192.168.x.0/24 to any port $port proto udp done sudo ufw status verbose ``` ### 8. Existing prep-cmd hooks were a stale version **Symptom** Headless output stayed at 1920x1080 regardless of what Moonlight requested. `hyprctl monitors all` showed extra `HEADLESS-N` appearing on each reconnect. Journal had no `[sunshine-do]` log lines. **Cause** `~/.local/share/omarchy-moonlight/bin/sunshine-stream-do.sh` was an early version that hardcoded `MON=HEADLESS-1`. Each connect: didn't find `HEADLESS-1`, ran `hyprctl output create headless` (spawned a new unnamed one), then tried to resize the non-existent `HEADLESS-1` — silent no-op. The repo's `bin/` had the discovery-based version, but it had never been re-installed after the first install. Sunshine's stderr capture also doesn't propagate prep-cmd output to the journal, so the silent failures were invisible. **Fix in this repo** `lib/headless.sh` → `install_headless_hooks` now installs all three scripts (do / undo / prestart) into `~/.local/share/omarchy-moonlight/bin/` and is re-run on every `install.sh` invocation — re-installing keeps them in lockstep with the repo. For ad-hoc debugging of the prep-cmd, add a file-logging shim: ```bash sed -i '/^set -euo pipefail/a exec > >(tee -a /tmp/sunshine-prepcmd.log) 2>&1' \ ~/.local/share/omarchy-moonlight/bin/sunshine-stream-do.sh ``` then `cat /tmp/sunshine-prepcmd.log` after a reconnect to see exactly what the script saw and what it tried to do. ### 9. omarchy-screensaver shows in the stream instead of your apps **Symptom** Stream connects, shows the omarchy screensaver. Workspace shows `hasfullscreen: 1`, `lastwindowtitle: omarchy-screensaver`. **Cause** Omarchy's hypridle config runs the screensaver after the host's idle threshold. Input arriving via Sunshine's `/dev/uinput` virtual devices doesn't always reset hypridle's input-detection (depends on whether hypridle treats uinput devices as activity sources). When you sit on the Mac for a while, the host thinks the host is idle, fires the screensaver. **Current workaround** Wiggle the mouse / tap a key in the stream window — hypridle will register the activity and dismiss the screensaver. **Outstanding fix (not yet implemented):** `sunshine-stream-do.sh` should issue `hyprctl dispatch exec hyprctl reload` or `systemctl --user stop hypridle.service`-style inhibition when a stream starts, and re-enable on stream stop via the undo hook. Alternatively, add a `hypridle.conf` rule that excludes the streaming session from idle counting. See follow-up #1 below. ### 10. Moonlight Mac defaults make it a roach motel **Symptom** After enabling "Capture system keyboard shortcuts: Always," all standard Mac escapes (Cmd+Tab, Cmd+Q, even Cmd+Option+Esc) get forwarded to the host. Mouse is also captured in game-mode. No clear way out. **Causes & fixes (Mac-side settings)** In Moonlight on the Mac → gear icon → Input section: | Setting | Value | Why | |---|---|---| | Optimize mouse for remote desktop | **ON** | Absolute-position cursor; can mouse out of Moonlight onto other Mac apps | | Capture system keyboard shortcuts | **Only in fullscreen** | Windowed-mode Mac shortcuts still work; capture kicks in only on explicit fullscreen | | Swap Cmd and Ctrl keys | **OFF** | Keep `Cmd → Super` mapping for Hyprland binds | **Emergency exit** that always works (macOS-level, can't be captured): - Three-finger swipe up (Mission Control) - `Cmd + Option + Esc` (Force Quit window) — usually - `Ctrl + Shift + Q` (system logout dialog — steals focus from Moonlight, then cancel) **Reliable custom exit shortcut** — see "Custom keybinding" section below. ### 11. Web UI cert hostname mismatch under SSH tunnel **Symptom** After bringing the SSH tunnel up (`ssh -L 47990:127.0.0.1:47990 ...`), visiting `https://localhost:47990` shows a browser cert warning even though the cert IS signed by the omarchy-stream CA. **Cause** The host cert's SAN list was originally only `.lan` and the LAN IP. `localhost` and `127.0.0.1` weren't valid for that cert, so the browser correctly rejected the hostname. **Fix in this repo** `lib/certs.sh` now mints host certs with SANs: ``` DNS.1 = ${host_lc}.lan DNS.2 = localhost IP.1 = ${lan_ip} IP.2 = 127.0.0.1 ``` And the idempotency check in `fetch_and_install_certs` requires those SANs — existing hosts re-mint on next install. ### 12. Headless host: service never auto-starts after boot (`graphical-session.target` never activates) **Symptom** Sunshine works right after install, and works again if you `systemctl --user start` it by hand, but never comes up on its own after a reboot. `systemctl --user status` shows the unit `enabled` yet `inactive (dead)` — *not* failed, just never started. Ports 47984/47989/47990 aren't listening and nothing is in the journal because the unit was never invoked. **Cause** The packaged unit (`sunshine.service`, or `app-dev.lizardbyte.app.Sunshine.service`) is wired `WantedBy=graphical-session.target` and `After=graphical-session.target`. On a desktop, logging in activates `graphical-session.target`, which pulls Sunshine up. A **headless host has no graphical login session**, so `graphical-session.target` never activates — even with `loginctl enable-linger` on and a headless display server (Xorg/sway) already running. The installer's one-time `systemctl --user start` is why it appears to work at install time; the wiring just never fires again at boot. This bites any headless deployment (KVM box, server with a dummy/virtual display, the X11/NVENC path in §13). It does *not* bite a laptop/desktop that actually logs into Hyprland. **Fix** Add a drop-in that wires the unit into a target the lingering user-manager actually reaches (`default.target`) and orders it after whatever provides the display: ```bash # adjust the unit name to whichever one is installed (see issue #3) UNIT=app-dev.lizardbyte.app.Sunshine.service mkdir -p ~/.config/systemd/user/$UNIT.d cat > ~/.config/systemd/user/$UNIT.d/headless-boot.conf <<'EOF' [Unit] # Order after the headless display unit (xorg-headless.service for the X11 # path, sway-headless.service for the wlr path). Requires=xorg-headless.service After=xorg-headless.service [Install] # graphical-session.target never activates on a headless host; default.target # is reached by the lingering user manager, so this is what makes boot work. WantedBy=default.target EOF systemctl --user daemon-reload systemctl --user reenable $UNIT # recreates default.target.wants symlink systemctl --user restart $UNIT ``` Confirm the load-bearing symlink exists: `ls ~/.config/systemd/user/default.target.wants/ | grep -i sunshine`. The real proof is a reboot — `enabled` alone isn't enough on a headless box. > Note: dependency keys (`Requires=`, `After=`) belong in the `[Unit]` section. > A drop-in that puts them in `[Service]` is silently ignored — systemd logs > `Unknown key name 'Requires' in section 'Service', ignoring` in > `systemctl status`, and the ordering never takes effect. ### 13. X11/NVENC headless path (Ubuntu / non-Hyprland hosts) **Symptom / context** This repo's headless mode assumes Hyprland + `capture = wlr`. On a host that isn't running Hyprland (e.g. an Ubuntu box, or one where you want NVIDIA NVENC with a guaranteed GL context), the wlr path has nothing to capture and every encoder probe fails at startup (same red banner as issue #4). **The X11/NVENC alternative (as deployed on at least one host)** Instead of a wlroots compositor, run a **headless Xorg on `:0`** and have Sunshine grab the X root window. This is the systemd-service equivalent of the upstream "Remote SSH Headless Setup" guide's TwinView trick — NVIDIA Xorg is told a monitor is connected so it produces a hardware-accelerated virtual display. - A `xorg-headless.service` (user unit) runs `Xorg :0 -config xorg-headless.conf …` with an NVIDIA `ConnectedMonitor` + `ModeValidation` block (see the upstream guide). It's the headless display the Sunshine unit orders against in issue #12. - `~/.config/sunshine/sunshine.conf` uses `capture = x11`, `output_name = 0`, `encoder = nvenc`. (Hand-edited — delete the `# managed-by:` marker line so the Arch installer leaves it alone.) - A service drop-in pins the X11 environment so Sunshine doesn't auto-probe Wayland: ```ini [Service] Environment= Environment=DISPLAY=:0 Environment=XDG_SESSION_TYPE=x11 UnsetEnvironment=WAYLAND_DISPLAY XDG_SESSION_TYPE ``` - Input injection works the normal way — user in the `input` group + `60-sunshine.rules` udev rule on `/dev/uinput`. The upstream guide's `chown`-via-passwordless-sudo `/dev/uinput` workaround is **not needed**; it only applies when you SSH in without group membership. **Gotcha: leftover wlr drop-ins poison the X11 env.** If a host was first set up for the wlr path and later migrated to X11, a stale `sunshine.service.d/sway-headless.conf` drop-in can linger. Because `sunshine.service` is an *alias* of `app-dev.lizardbyte.app.Sunshine.service`, systemd merges drop-ins from **both** name directories, so that leftover keeps injecting `XDG_SESSION_TYPE=wayland`, `WAYLAND_DISPLAY=wayland-1`, and a hard `Requires=sway-headless.service` (a dead unit). It "works" only because `capture = x11` is explicit in the conf, but at boot it tries to pull in the dead sway unit. Delete the stale drop-in and verify the effective env: ```bash systemctl --user show $UNIT -p Environment -p Requires -p After # should show DISPLAY=:0 + XDG_SESSION_TYPE=x11, xorg-headless.service, no wayland/sway ``` **Gotcha: black screen in Moonlight even though everything "works".** Pairing succeeds, NVENC loads, mouse/keyboard input reaches the host — but the client sees only black. Cause: the headless Xorg on `:0` is up, but **nothing is rendering on it**. Unlike the wlr path (where the compositor *is* the desktop), a bare Xorg server draws nothing on its own, so `capture = x11` grabs an empty black root window. Confirm with: ```bash DISPLAY=:0 xlsclients # only 'sunshine' = no WM/desktop DISPLAY=:0 xprop -root _NET_SUPPORTING_WM_CHECK # 'not found' = no window manager ``` Fix: run a desktop on `:0`. The installer ships a `headless-desktop.service` for exactly this and enables it whenever it detects `capture = x11`. It defaults to a full **GNOME** session (`gnome-session --session=ubuntu`, forced to the X11 path); set `HEADLESS_DESKTOP=openbox` for a lightweight bare WM instead (lower overhead, but no panel/launcher — right-click menu only). The unit forces `XDG_SESSION_TYPE=x11` and does **not** wrap GNOME in `dbus-run-session` — GNOME must share the systemd *user* bus the service already inherits. ```bash systemctl --user enable --now headless-desktop.service DISPLAY=:0 xprop -root _NET_SUPPORTING_WM_CHECK # now reports a 'window id' # read the running desktop's name off that window: DISPLAY=:0 xprop -id _NET_WM_NAME # e.g. "GNOME Shell" ``` `status.sh` checks for this directly: with `capture = x11` it now FAILs if no window manager is present on `:0`, instead of only verifying the X server answers. --- ## Custom keybinding to escape Moonlight (Mac) Three options. Option C is most reliable. ### A. macOS App Shortcut (5 min, no extra software) System Settings → Keyboard → Keyboard Shortcuts → App Shortcuts → `+`: - Application: *Moonlight Game Streaming* - Menu Title: `Quit Moonlight Game Streaming` (or `Disconnect`) - Keyboard Shortcut: e.g. `⌘⎋` (Cmd+Esc) Requires Moonlight's "Capture system keyboard shortcuts" to be **Only in fullscreen** so the shortcut reaches macOS in windowed mode. ### B. Karabiner-Elements (best for fullscreen) Karabiner intercepts at HID, below the Moonlight app layer. The setup pasted in the troubleshooting chat: bind `Right ⌘ + Esc` → `killall Moonlight`. ```json { "title": "Force-quit Moonlight", "rules": [{ "description": "Right-Cmd + Escape → killall Moonlight", "manipulators": [{ "type": "basic", "from": { "key_code": "escape", "modifiers": { "mandatory": ["right_command"] } }, "to": [{ "shell_command": "killall Moonlight" }] }] }] } ``` ### C. Stream Deck (since you have one) System → Multimedia → Hotkey → send `⌃⌥⇧Q` (Moonlight's built-in quit-stream combo) or System → Open with command `/usr/bin/killall Moonlight`. Stream Deck input is HID; Moonlight can't capture it. --- ## Outstanding follow-ups ### 1. Inhibit hypridle / screensaver during stream Currently the omarchy screensaver fires after the host's idle timeout even when a stream is active. Either: - Have `sunshine-stream-do.sh` send a `systemd-inhibit` lock or `hyprctl dispatch global hyprlock:lock-off` (does that exist?) when a stream starts, release in `sunshine-stream-undo.sh`. - Or add an `unbind` for the input watcher in `~/.config/hypr/hypridle.conf` when a `SUNSHINE_STREAMING=1` env marker is present. **Outcome wanted:** no screensaver while any Moonlight client is connected. ### 2. Auto-switch to the workspace with the most windows on connect Currently `sunshine-stream-do.sh` migrates the *active* workspace to the headless. If active was empty (workspace 4) and the user's apps are on workspace 1, they see nothing until pressing `Cmd+1`. Two-line patch in `sunshine-stream-do.sh`: ```bash TARGET_WS="$(hyprctl workspaces -j | jq -r 'sort_by(-.windows) | .[0].id')" PREV_WS="$TARGET_WS" # instead of activeworkspace ``` Pick the busiest workspace and migrate that. User lands on their work. ### 3. 1Password window streams as a black rectangle By design — 1Password masks its window from screen capture. Workarounds in order of preference: 1. Launch 1Password with `--disable-gpu --no-sandbox` — software rendering path may evade the anti-capture. 2. Launch with `--ozone-platform=x11` — XWayland surface captured differently than native-Wayland surface. 3. Use the 1Password browser extension during the streamed session — the extension's UI lives in Chromium, which captures normally. 4. Use `op` CLI for everything CLI-driven (cert pipeline already does this). 5. Unlock 1Password locally on the host before walking away; other apps that talk to the 1Password agent (browser extension, `op` CLI, SSH signing agent) work fine even though the desktop app's window can't be seen. We haven't picked one yet — see chat for next steps. ### 4. mDNS / `your-host.lan` doesn't resolve Even on the host, `getent hosts your-host.lan` returns nothing. To use the friendly hostname instead of the LAN IP from clients, set up the DNS entry in Unifi (the user's network gear). Until then, clients use `192.168.x.y` directly. ### 5. Periodic 1Password SSH-agent timeout breaks git signing During this install the 1P SSH agent went idle several times mid-session, causing `git commit` to fail with `1Password: failed to fill whole buffer`. Touching the 1Password desktop app revives it. Possible mitigations: - Set a longer SSH-agent timeout in 1Password app settings. - Add a watchdog that warns if `ssh-add -l` fails for >N minutes. - Disable commit signing for this repo specifically (not recommended; just noting the option). --- ## Quick reference — the install sequence as it actually runs end-to-end After `./install.sh` (with all the fixes above): 1. Preflight checks (GPU, modeset, audio, headless prereqs) 2. `yay -S sunshine-bin moonlight-qt pipewire-pulse vulkan-tools libva-utils jq` 3. If `sunshine-bin` has unresolved libs → rebuild from source (`sunshine`) 4. Add user to `input` group, install uinput udev rule, `setcap cap_sys_admin+p` 5. Write tuned `~/.config/sunshine/sunshine.conf`: - `capture = wlr`, `output_name = HEADLESS-1` (will be re-synced by prestart) - `encoder = nvenc` + low-latency NVENC params - `global_prep_cmd` → discovery-based do/undo scripts - `origin_web_ui_allowed = pc` 6. Install hook scripts (`do`, `undo`, `prestart`) into `~/.local/share/omarchy-moonlight/bin/` 7. Install prestart drop-in into `~/.config/systemd/user/.service.d/headless-prestart.conf` 8. Open Sunshine's LAN ports on whatever firewall is active 9. Enable the service, `loginctl enable-linger`, start 10. (1Password-CLI signed-in) Fetch the CA from 1Password, mint a host cert with SANs for `.lan`, LAN IP, `localhost`, `127.0.0.1`. Install CA into `/etc/ca-certificates`. 11. Restart service so it picks up the new cert 12. Verify: cap_sys_admin set, service active on :47990, encoder reachable, hooks present, CA in trust store Manual one-time setup on each Moonlight client: - Mac: `./client/install-macos.sh` (brew cask + CA add to System keychain) - Other Linux host: same `./install.sh` puts the CA into its system trust store - iOS/Android/Apple TV: install Moonlight from app store + import CA manually