Omarchy-Stream/docs/TROUBLESHOOTING.md

Troubleshooting & lessons from the first end-to-end install

Every failure mode below was hit in practice during the first end-to-end install on a real Omarchy/NVIDIA/Wayland host. The fixes landed in this repo so future installs (e.g. on the Framework) shouldn't repeat them — this doc explains the why so you can recognize them if they come back with a twist.

The "Real failures hit on first install" section is in roughly the order they surfaced.

---

Real failures hit on first install

1. sunshine-bin breaks on rolling Arch (library drift)

Symptom

sunshine: error while loading shared libraries: libicuuc.so.76:
cannot open shared object file: No such file or directory

Service hits the systemd start-limit after five rapid failures and won't try again until reset-failed.

Cause

The AUR sunshine-bin package ships a binary built against whatever ICU was current at AUR-build time (ICU 76). Arch had moved to ICU 78. Library SONAME is hard-pinned at build time, so the binary fails to load any of its ICU calls. This recurs whenever Arch bumps a major library and the sunshine-bin maintainer hasn't rebuilt yet.

Fix in this repo

lib/packages.sh → install_sunshine now runs ldd $(command -v sunshine) after install. If anything is "not found", removes sunshine-bin (and sunshine-bin-debug) and rebuilds from source. The source-build linking pulls in the current ICU, so it works as long as Arch ships any version.

lib/service.sh → enable_sunshine_service runs systemctl --user reset-failed before each restart, so a previous failure doesn't preempt the recovery.

Manual recovery if you hit it on a non-installer flow

sudo pacman -Rns --noconfirm sunshine-bin sunshine-bin-debug
yay -S --needed --noconfirm sunshine
sudo setcap cap_sys_admin+p "$(readlink -f "$(command -v sunshine)")"
systemctl --user reset-failed sunshine.service

2. Source build is slow (~10 min); cached pkg.tar.zst install collides on debug symbols

Symptom

error: failed to commit transaction (conflicting files)
sunshine-debug: /usr/lib/debug/usr/bin/sunshine.debug exists in filesystem
  (owned by sunshine-bin-debug)

After a successful makepkg, pacman -U of the resulting .pkg.tar.zst refuses to install.

Cause

sunshine-bin ships with a separate sunshine-bin-debug package for debug symbols, owned by sunshine-bin-debug. The source build's sunshine-debug package wants to put its symbols at the same path. Removing sunshine-bin doesn't auto-remove its -debug sibling.

Fix in this repo

lib/packages.sh removes both sunshine-bin AND sunshine-bin-debug before the source rebuild. uninstall.sh similarly drops all four variants (sunshine, sunshine-debug, sunshine-bin, sunshine-bin-debug).

3. Source sunshine doesn't ship sunshine.service

Symptom

Failed to restart sunshine.service: Unit sunshine.service not found.

After replacing sunshine-bin with the source sunshine build, the service that worked before is gone.

Cause

The AUR source PKGBUILD installs its systemd unit under a Flatpak-style reverse-DNS filename: app-dev.lizardbyte.app.Sunshine.service. The sunshine-bin variant ships sunshine.service directly.

Fix in this repo

lib/service.sh → ensure_sunshine_unit_present resolves the actual unit name in three steps:

1. Look for sunshine.service in every systemd-user lookup path.
2. If not found, look for app-dev.lizardbyte.app.Sunshine.service and use that name directly (we no longer try to symlink it as sunshine.service — systemd refuses to enable linked unit files).
3. Last resort: install the repo's files/sunshine.service fallback unit.

The drop-in path (for the headless prestart hook) is computed against the resolved unit name so <unit-name>.service.d/ always lands in the right place.

4. The fatal-but-misleading "Unable to find display or encoder during startup"

Symptom

[wlgrab] -------- Start of Wayland monitor list --------
[wlgrab] --------- End of Wayland monitor list ---------
Error: Unable to initialize capture method
...
Encoder [nvenc] failed
Encoder [vulkan] failed
Encoder [vaapi] failed
Encoder [software] failed
Fatal: Unable to find display or encoder during startup.
Fatal: Please ensure your manually chosen GPU and monitor are connected

Visible in the web UI as red banner warnings. Streaming may still appear to work later because Sunshine re-probes at stream time, but the startup state is broken.

Cause

In headless mode, sunshine.conf pins output_name = HEADLESS-1 and capture = wlr. At service start, Sunshine connects to Wayland, enumerates outputs, and finds nothing — because the Hyprland headless output is created by the per-stream prep-cmd, not at boot. With no output, every encoder probe fails on platform-init.

Fix in this repo

bin/sunshine-prestart.sh runs as ExecStartPre for the Sunshine service. It guarantees one (and exactly one) Hyprland headless output exists before Sunshine probes encoders. It also rewrites sunshine.conf's output_name line to match whatever name Hyprland assigned the surviving headless (see issue 6).

The drop-in lives at ~/.config/systemd/user/<unit-name>.service.d/headless-prestart.conf, managed by lib/headless.sh → install_headless_prestart_dropin.

The bin/sunshine-stream-undo.sh was also changed to keep the headless across stream sessions — create/destroy per stream raced with the startup encoder probe.

5. Web UI 0.0.0.0 binding was open to the LAN

Symptom

By default Sunshine binds the web UI on 0.0.0.0:47990. Anyone on your LAN who can reach the host can hit the admin panel; only a username/password guards it.

Fix in this repo

lib/config.sh writes origin_web_ui_allowed = pc. Sunshine then rejects web-UI requests from any source other than localhost regardless of bind address. Streaming/pairing on port 47989 stays LAN-accessible — only the admin surface is locked down.

Recommended access pattern (also documented inline in sunshine.conf):

# From a client machine:
ssh -L 47990:127.0.0.1:47990 user@host
# Then browse to:
https://localhost:47990

The host cert SANs include localhost and 127.0.0.1 (see issue 11) so the tunneled URL doesn't trigger hostname-mismatch warnings.

6. Hyprland headless counter never decrements

Symptom

After several restarts, hyprctl monitors all shows HEADLESS-4, HEADLESS-9, HEADLESS-10, ...

Selected monitor [] for streaming in the journal — Sunshine can't match output_name = HEADLESS-1 against any existing output.

Cause

Hyprland's hyprctl output create headless ignores the optional name argument (at least through 0.54.x) and auto-numbers as HEADLESS-N. The counter is monotonic across the Hyprland session — removing HEADLESS-1 and re-creating gives you HEADLESS-2, not HEADLESS-1. Combine that with an old version of the prep-cmd script that hardcoded MON=HEADLESS-1 and created a new output every connect, and you accumulate a herd.

Fix in this repo

Two-layer:

1. bin/sunshine-prestart.sh actively removes extras, keeping only the lowest-numbered HEADLESS-N. Then it rewrites sunshine.conf's output_name to that surviving name. Self-healing.
2. bin/sunshine-stream-do.sh and sunshine-stream-undo.sh now discover the existing headless name via jq instead of hardcoding HEADLESS-1. Resize and workspace-migration target the actual output.

7. ufw was active but our detection missed it

Symptom

ARP between Mac and the host worked fine (layer 2 visible). TCP to port 47989 hung from the Mac. nmap -p 47989 192.168.x.y said "Host seems down."

Cause

lib/firewall.sh → _ufw_active previously ran ufw status | grep -q "Status: active". ufw status requires root to read /etc/ufw state, so an unprivileged check returned nothing → returned false → we silently skipped opening Sunshine's ports. ufw was happily dropping inbound packets.

Fix in this repo

_ufw_active now uses systemctl is-active ufw.service. ufw runs as a Type=oneshot unit with RemainAfterExit, so is-active reports active while the rules are loaded — and the check works without sudo.

One-shot LAN allow if you hit it

for port in 47984 47989 47990 48010; do
  sudo ufw allow from 192.168.x.0/24 to any port $port proto tcp
done
for port in 47998 47999 48000 48010; do
  sudo ufw allow from 192.168.x.0/24 to any port $port proto udp
done
sudo ufw status verbose

8. Existing prep-cmd hooks were a stale version

Symptom

Headless output stayed at 1920x1080 regardless of what Moonlight requested. hyprctl monitors all showed extra HEADLESS-N appearing on each reconnect. Journal had no [sunshine-do] log lines.

Cause

~/.local/share/omarchy-moonlight/bin/sunshine-stream-do.sh was an early version that hardcoded MON=HEADLESS-1. Each connect: didn't find HEADLESS-1, ran hyprctl output create headless (spawned a new unnamed one), then tried to resize the non-existent HEADLESS-1 — silent no-op. The repo's bin/ had the discovery-based version, but it had never been re-installed after the first install.

Sunshine's stderr capture also doesn't propagate prep-cmd output to the journal, so the silent failures were invisible.

Fix in this repo

lib/headless.sh → install_headless_hooks now installs all three scripts (do / undo / prestart) into ~/.local/share/omarchy-moonlight/bin/ and is re-run on every install.sh invocation — re-installing keeps them in lockstep with the repo.

For ad-hoc debugging of the prep-cmd, add a file-logging shim:

sed -i '/^set -euo pipefail/a exec > >(tee -a /tmp/sunshine-prepcmd.log) 2>&1' \
  ~/.local/share/omarchy-moonlight/bin/sunshine-stream-do.sh

then cat /tmp/sunshine-prepcmd.log after a reconnect to see exactly what the script saw and what it tried to do.

9. omarchy-screensaver shows in the stream instead of your apps

Symptom

Stream connects, shows the omarchy screensaver. Workspace shows hasfullscreen: 1, lastwindowtitle: omarchy-screensaver.

Cause

Omarchy's hypridle config runs the screensaver after the host's idle threshold. Input arriving via Sunshine's /dev/uinput virtual devices doesn't always reset hypridle's input-detection (depends on whether hypridle treats uinput devices as activity sources). When you sit on the Mac for a while, the host thinks the host is idle, fires the screensaver.

Current workaround

Wiggle the mouse / tap a key in the stream window — hypridle will register the activity and dismiss the screensaver.

Outstanding fix (not yet implemented):

sunshine-stream-do.sh should issue hyprctl dispatch exec hyprctl reload or systemctl --user stop hypridle.service-style inhibition when a stream starts, and re-enable on stream stop via the undo hook. Alternatively, add a hypridle.conf rule that excludes the streaming session from idle counting. See follow-up #1 below.

10. Moonlight Mac defaults make it a roach motel

Symptom

After enabling "Capture system keyboard shortcuts: Always," all standard Mac escapes (Cmd+Tab, Cmd+Q, even Cmd+Option+Esc) get forwarded to the host. Mouse is also captured in game-mode. No clear way out.

Causes & fixes (Mac-side settings)

In Moonlight on the Mac → gear icon → Input section:

Setting	Value	Why
Optimize mouse for remote desktop	ON	Absolute-position cursor; can mouse out of Moonlight onto other Mac apps
Capture system keyboard shortcuts	Only in fullscreen	Windowed-mode Mac shortcuts still work; capture kicks in only on explicit fullscreen
Swap Cmd and Ctrl keys	OFF	Keep Cmd → Super mapping for Hyprland binds

Emergency exit that always works (macOS-level, can't be captured):

• Three-finger swipe up (Mission Control)
• Cmd + Option + Esc (Force Quit window) — usually
• Ctrl + Shift + Q (system logout dialog — steals focus from Moonlight, then cancel)

Reliable custom exit shortcut — see "Custom keybinding" section below.

11. Web UI cert hostname mismatch under SSH tunnel

Symptom

After bringing the SSH tunnel up (ssh -L 47990:127.0.0.1:47990 ...), visiting https://localhost:47990 shows a browser cert warning even though the cert IS signed by the omarchy-stream CA.

Cause

The host cert's SAN list was originally only <hostname>.lan and the LAN IP. localhost and 127.0.0.1 weren't valid for that cert, so the browser correctly rejected the hostname.

Fix in this repo

lib/certs.sh now mints host certs with SANs:

DNS.1 = ${host_lc}.lan
DNS.2 = localhost
IP.1 = ${lan_ip}
IP.2 = 127.0.0.1

And the idempotency check in fetch_and_install_certs requires those SANs — existing hosts re-mint on next install.

12. Headless host: service never auto-starts after boot (graphical-session.target never activates)

Symptom

Sunshine works right after install, and works again if you systemctl --user start it by hand, but never comes up on its own after a reboot. systemctl --user status shows the unit enabled yet inactive (dead) — not failed, just never started. Ports 47984/47989/47990 aren't listening and nothing is in the journal because the unit was never invoked.

Cause

The packaged unit (sunshine.service, or app-dev.lizardbyte.app.Sunshine.service) is wired WantedBy=graphical-session.target and After=graphical-session.target. On a desktop, logging in activates graphical-session.target, which pulls Sunshine up. A headless host has no graphical login session, so graphical-session.target never activates — even with loginctl enable-linger on and a headless display server (Xorg/sway) already running. The installer's one-time systemctl --user start is why it appears to work at install time; the wiring just never fires again at boot.

This bites any headless deployment (KVM box, server with a dummy/virtual display, the X11/NVENC path in §13). It does not bite a laptop/desktop that actually logs into Hyprland.

Fix

Add a drop-in that wires the unit into a target the lingering user-manager actually reaches (default.target) and orders it after whatever provides the display:

# adjust the unit name to whichever one is installed (see issue #3)
UNIT=app-dev.lizardbyte.app.Sunshine.service
mkdir -p ~/.config/systemd/user/$UNIT.d
cat > ~/.config/systemd/user/$UNIT.d/headless-boot.conf <<'EOF'
[Unit]
# Order after the headless display unit (xorg-headless.service for the X11
# path, sway-headless.service for the wlr path).
Requires=xorg-headless.service
After=xorg-headless.service

[Install]
# graphical-session.target never activates on a headless host; default.target
# is reached by the lingering user manager, so this is what makes boot work.
WantedBy=default.target
EOF
systemctl --user daemon-reload
systemctl --user reenable $UNIT     # recreates default.target.wants symlink
systemctl --user restart $UNIT

Confirm the load-bearing symlink exists: ls ~/.config/systemd/user/default.target.wants/ | grep -i sunshine. The real proof is a reboot — enabled alone isn't enough on a headless box.

Note: dependency keys (Requires=, After=) belong in the [Unit] section. A drop-in that puts them in [Service] is silently ignored — systemd logs Unknown key name 'Requires' in section 'Service', ignoring in systemctl status, and the ordering never takes effect.

13. X11/NVENC headless path (Ubuntu / non-Hyprland hosts)

Symptom / context

This repo's headless mode assumes Hyprland + capture = wlr. On a host that isn't running Hyprland (e.g. an Ubuntu box, or one where you want NVIDIA NVENC with a guaranteed GL context), the wlr path has nothing to capture and every encoder probe fails at startup (same red banner as issue #4).

The X11/NVENC alternative (as deployed on at least one host)

Instead of a wlroots compositor, run a headless Xorg on :0 and have Sunshine grab the X root window. This is the systemd-service equivalent of the upstream "Remote SSH Headless Setup" guide's TwinView trick — NVIDIA Xorg is told a monitor is connected so it produces a hardware-accelerated virtual display.

• A xorg-headless.service (user unit) runs Xorg :0 -config xorg-headless.conf … with an NVIDIA ConnectedMonitor + ModeValidation block (see the upstream guide). It's the headless display the Sunshine unit orders against in issue #12.
• ~/.config/sunshine/sunshine.conf uses capture = x11, output_name = 0, encoder = nvenc. (Hand-edited — delete the # managed-by: marker line so the Arch installer leaves it alone.)
• A service drop-in pins the X11 environment so Sunshine doesn't auto-probe Wayland:

  [Service]
  Environment=
  Environment=DISPLAY=:0
  Environment=XDG_SESSION_TYPE=x11
  UnsetEnvironment=WAYLAND_DISPLAY XDG_SESSION_TYPE
  

• Input injection works the normal way — user in the input group + 60-sunshine.rules udev rule on /dev/uinput. The upstream guide's chown-via-passwordless-sudo /dev/uinput workaround is not needed; it only applies when you SSH in without group membership.

Gotcha: leftover wlr drop-ins poison the X11 env. If a host was first set up for the wlr path and later migrated to X11, a stale sunshine.service.d/sway-headless.conf drop-in can linger. Because sunshine.service is an alias of app-dev.lizardbyte.app.Sunshine.service, systemd merges drop-ins from both name directories, so that leftover keeps injecting XDG_SESSION_TYPE=wayland, WAYLAND_DISPLAY=wayland-1, and a hard Requires=sway-headless.service (a dead unit). It "works" only because capture = x11 is explicit in the conf, but at boot it tries to pull in the dead sway unit. Delete the stale drop-in and verify the effective env:

systemctl --user show $UNIT -p Environment -p Requires -p After
# should show DISPLAY=:0 + XDG_SESSION_TYPE=x11, xorg-headless.service, no wayland/sway

Gotcha: black screen in Moonlight even though everything "works". Pairing succeeds, NVENC loads, mouse/keyboard input reaches the host — but the client sees only black. Cause: the headless Xorg on :0 is up, but nothing is rendering on it. Unlike the wlr path (where the compositor is the desktop), a bare Xorg server draws nothing on its own, so capture = x11 grabs an empty black root window. Confirm with:

DISPLAY=:0 xlsclients                              # only 'sunshine' = no WM/desktop
DISPLAY=:0 xprop -root _NET_SUPPORTING_WM_CHECK    # 'not found' = no window manager

Fix: run a desktop on :0. The installer ships a headless-desktop.service for exactly this and enables it whenever it detects capture = x11. It defaults to a full GNOME session (gnome-session --session=ubuntu, forced to the X11 path); set HEADLESS_DESKTOP=openbox for a lightweight bare WM instead (lower overhead, but no panel/launcher — right-click menu only). The unit forces XDG_SESSION_TYPE=x11 and does not wrap GNOME in dbus-run-session — GNOME must share the systemd user bus the service already inherits.

systemctl --user enable --now headless-desktop.service
DISPLAY=:0 xprop -root _NET_SUPPORTING_WM_CHECK    # now reports a 'window id'
# read the running desktop's name off that window:
DISPLAY=:0 xprop -id <id> _NET_WM_NAME             # e.g. "GNOME Shell"

status.sh checks for this directly: with capture = x11 it now FAILs if no window manager is present on :0, instead of only verifying the X server answers.

---

Custom keybinding to escape Moonlight (Mac)

Three options. Option C is most reliable.

A. macOS App Shortcut (5 min, no extra software)

System Settings → Keyboard → Keyboard Shortcuts → App Shortcuts → +:

• Application: Moonlight Game Streaming
• Menu Title: Quit Moonlight Game Streaming (or Disconnect)
• Keyboard Shortcut: e.g. ⌘⎋ (Cmd+Esc)

Requires Moonlight's "Capture system keyboard shortcuts" to be Only in fullscreen so the shortcut reaches macOS in windowed mode.

B. Karabiner-Elements (best for fullscreen)

Karabiner intercepts at HID, below the Moonlight app layer. The setup pasted in the troubleshooting chat: bind Right ⌘ + Esc → killall Moonlight.

{
  "title": "Force-quit Moonlight",
  "rules": [{
    "description": "Right-Cmd + Escape → killall Moonlight",
    "manipulators": [{
      "type": "basic",
      "from": {
        "key_code": "escape",
        "modifiers": { "mandatory": ["right_command"] }
      },
      "to": [{ "shell_command": "killall Moonlight" }]
    }]
  }]
}

C. Stream Deck (since you have one)

System → Multimedia → Hotkey → send ⌃⌥⇧Q (Moonlight's built-in quit-stream combo) or System → Open with command /usr/bin/killall Moonlight. Stream Deck input is HID; Moonlight can't capture it.

---

Outstanding follow-ups

1. Inhibit hypridle / screensaver during stream

Currently the omarchy screensaver fires after the host's idle timeout even when a stream is active. Either:

• Have sunshine-stream-do.sh send a systemd-inhibit lock or hyprctl dispatch global hyprlock:lock-off (does that exist?) when a stream starts, release in sunshine-stream-undo.sh.
• Or add an unbind for the input watcher in ~/.config/hypr/hypridle.conf when a SUNSHINE_STREAMING=1 env marker is present.

Outcome wanted: no screensaver while any Moonlight client is connected.

2. Auto-switch to the workspace with the most windows on connect

Currently sunshine-stream-do.sh migrates the active workspace to the headless. If active was empty (workspace 4) and the user's apps are on workspace 1, they see nothing until pressing Cmd+1.

Two-line patch in sunshine-stream-do.sh:

TARGET_WS="$(hyprctl workspaces -j | jq -r 'sort_by(-.windows) | .[0].id')"
PREV_WS="$TARGET_WS"  # instead of activeworkspace

Pick the busiest workspace and migrate that. User lands on their work.

3. 1Password window streams as a black rectangle

By design — 1Password masks its window from screen capture. Workarounds in order of preference:

1. Launch 1Password with --disable-gpu --no-sandbox — software rendering path may evade the anti-capture.
2. Launch with --ozone-platform=x11 — XWayland surface captured differently than native-Wayland surface.
3. Use the 1Password browser extension during the streamed session — the extension's UI lives in Chromium, which captures normally.
4. Use op CLI for everything CLI-driven (cert pipeline already does this).
5. Unlock 1Password locally on the host before walking away; other apps that talk to the 1Password agent (browser extension, op CLI, SSH signing agent) work fine even though the desktop app's window can't be seen.

We haven't picked one yet — see chat for next steps.

4. mDNS / your-host.lan doesn't resolve

Even on the host, getent hosts your-host.lan returns nothing. To use the friendly hostname instead of the LAN IP from clients, set up the DNS entry in Unifi (the user's network gear). Until then, clients use 192.168.x.y directly.

5. Periodic 1Password SSH-agent timeout breaks git signing

During this install the 1P SSH agent went idle several times mid-session, causing git commit to fail with 1Password: failed to fill whole buffer. Touching the 1Password desktop app revives it.

Possible mitigations:

• Set a longer SSH-agent timeout in 1Password app settings.
• Add a watchdog that warns if ssh-add -l fails for >N minutes.
• Disable commit signing for this repo specifically (not recommended; just noting the option).

---

Quick reference — the install sequence as it actually runs end-to-end

After ./install.sh (with all the fixes above):

1. Preflight checks (GPU, modeset, audio, headless prereqs)
2. yay -S sunshine-bin moonlight-qt pipewire-pulse vulkan-tools libva-utils jq
3. If sunshine-bin has unresolved libs → rebuild from source (sunshine)
4. Add user to input group, install uinput udev rule, setcap cap_sys_admin+p
5. Write tuned ~/.config/sunshine/sunshine.conf:
   • capture = wlr, output_name = HEADLESS-1 (will be re-synced by prestart)
   • encoder = nvenc + low-latency NVENC params
   • global_prep_cmd → discovery-based do/undo scripts
   • origin_web_ui_allowed = pc
6. Install hook scripts (do, undo, prestart) into ~/.local/share/omarchy-moonlight/bin/
7. Install prestart drop-in into ~/.config/systemd/user/<unit-name>.service.d/headless-prestart.conf
8. Open Sunshine's LAN ports on whatever firewall is active
9. Enable the service, loginctl enable-linger, start
10. (1Password-CLI signed-in) Fetch the CA from 1Password, mint a host cert with SANs for <hostname>.lan, LAN IP, localhost, 127.0.0.1. Install CA into /etc/ca-certificates.
11. Restart service so it picks up the new cert
12. Verify: cap_sys_admin set, service active on :47990, encoder reachable, hooks present, CA in trust store

Manual one-time setup on each Moonlight client:

• Mac: ./client/install-macos.sh (brew cask + CA add to System keychain)
• Other Linux host: same ./install.sh puts the CA into its system trust store
• iOS/Android/Apple TV: install Moonlight from app store + import CA manually