Omarchy-Stream/docs/HEADLESS-PRIMARY.md

# Headless-primary streaming (JARVIS as-built)

How **JARVIS** streams its desktop to Moonlight so that it works **whether or not
the physical monitor is on** — the machine's normal remote-use case.

> **Companion note — keep in sync.** This document is mirrored in the Obsidian
> vault at `Documents/Sync Vault/Network/Network — Sunshine + Moonlight (JARVIS).md`.
> They are **not** auto-linked. **If you edit one, update the other.**

---

## TL;DR

- The desktop permanently lives on a **persistent virtual `HEADLESS-1` output**.
  Sunshine captures it. It exists independent of the physical monitor.
- The physical **`DP-1` mirrors `HEADLESS-1`** when present, so the at-desk view
  equals the stream.
- When `DP-1` is off (remote), nothing is orphaned and the stream keeps working.
- Managed by `~/.local/share/omarchy-moonlight/bin/sunshine-headless-primary.sh`.

---

## Why this design (the three iterations)

1. **Headless-move (stock omarchy-moonlight headless mode)** — created a
   client-sized `HEADLESS-1` and *moved the active workspace onto it* via the
   `global_prep_cmd` hooks. Not a clone; and because a permanent off-screen
   output existed, apps that remember their monitor (OBS) reopened invisibly on
   it. Rejected.

2. **Mirror-clone (`HEADLESS-1` mirrors `DP-1`)** — a true clone while the
   monitor is on, and mirrored outputs are excluded from the layout so nothing
   strands on them. **But** when the physical monitor fully powers off, `DP-1`
   disappears from Hyprland, the mirror collapses to a standalone empty output,
   and every workspace bound to `DP-1` is orphaned off-screen. Fatal for a
   remote-first host whose monitor is usually off.

3. **Headless-primary (current)** — invert the mirror. The **headless output is
   the source of truth** (always present); `DP-1` mirrors *it*. The capture
   target is never absent, so nothing is ever orphaned. This is the only model
   that satisfies "the stream is a clone AND survives the monitor being off."

**Tradeoff:** the stream is the full `5120x1440` ultrawide, letterboxed on 16:9
clients. Inherent to cloning an ultrawide.

---

## Components

| Path | Role |
|---|---|
| `bin/sunshine-headless-primary.sh` | The manager. `apply` establishes/repairs state (idempotent); `rescue` pulls off-screen floating windows back on-screen; `reconcile` heals drift only if broken; `watch` the self-healing daemon (events + periodic reconcile backstop + auto-reconnect). |
| `bin/sunshine-prestart.sh` | Sunshine `ExecStartPre` — thin wrapper that runs `sunshine-headless-primary.sh apply` before the encoder probe. |
| `~/.config/hypr/monitors.conf` | `DP-1` kept **normal** (safety fallback); `HEADLESS-1..4` default to `0x0`. The manager applies the `DP-1 → mirror HEADLESS-1` on top. |
| `~/.config/hypr/autostart.conf` | `exec-once` runs `apply` at login and starts the `watch` daemon. |
| `~/.config/sunshine/sunshine.conf` | `capture = wlr`, `output_name = HEADLESS-1`, `encoder = nvenc`, `global_prep_cmd = []` (no workspace moving). |
| systemd drop-in `…Sunshine.service.d/headless-prestart.conf` | wires `ExecStartPre` to prestart. |

### What `apply` does (idempotent)

1. Ensures exactly one `HEADLESS` output (creates if none; **disables** extras —
   `hyprctl output remove` is unreliable for these, returns "output not found").
2. Sizes it `5120x1440@60` at `0x0`.
3. If `DP-1` is present → `hyprctl keyword monitor "DP-1,…,mirror,HEADLESS-1"`.
4. **Relocates any workspace stranded on a mirrored output** onto the primary.
   On swap-back Hyprland brings `DP-1` up *normal*, binds a workspace to it, and
   then we mirror it — leaving that workspace trapped on a layout-excluded
   mirror (unreachable). This moves it off. (You can move a workspace *off* a
   mirror but not *onto* one, which is why this only bites via hotplug.)
5. Rescues any **orphaned** workspace (`monitorID == -1`) onto the headless
   primary — deliberately narrow, so a healthy second monitor (DP-2) is left
   alone.
5. **Rescues off-screen floating windows** — recenters any floating, mapped
   window whose rectangle doesn't intersect the monitor its workspace lives on
   (stranded by a topology change, or opened off-screen).
6. Syncs `sunshine.conf`'s `output_name` to the live headless name.

### Self-healing (the watcher)

`watch` is designed so the setup **always converges**, not just when an event
fires. Three layers:

1. **Events** (fast path) — subscribes to Hyprland's socket:
   - `monitoradded` / `monitorremoved` → `apply` immediately (debounced 2s to
     coalesce the v1+v2 burst), then a `reconcile` 1.5s later to catch a
     workspace Hyprland reassigns to `DP-1` *after* apply already ran.
   - `openwindow` → targeted `rescue` if that window opened off-screen.
2. **Periodic `reconcile` backstop** (every `RECONCILE_SECS`, default 8s) — a
   read-only drift check that runs `apply` **only when something is actually
   wrong** (no headless / DP-1 not mirroring / workspace trapped on a mirror /
   orphaned workspace / off-screen window). This is the guarantee: even if an
   event is missed entirely, the state self-heals within a few seconds. It does
   nothing when healthy, so there's no steady-state flicker.
3. **Auto-reconnect** — if `socat` drops, the loop reconnects; if the session
   socket disappears (Hyprland gone) it exits cleanly so the next login's
   `exec-once` starts a fresh watcher (no cross-session duplicates).

So a monitor swap, a trapped workspace ("can't show workspace 1"), or an app
that opens off-screen (OBS, `bluetui`, kdenlive have all done this) gets fixed
automatically — near-instant via the event, or within ~8s via the backstop.
Force a check manually anytime with `sunshine-headless-primary.sh reconcile`.

### Critical implementation notes

- **Always `hyprctl monitors all`**, never plain `hyprctl monitors` — a mirrored
  output is excluded from the plain list, i.e. invisible to exactly the outputs
  we manage. Plain form → the dedup loop spawns a new headless every run.
- **`DP-1` must stay `normal` in `monitors.conf`.** It is the safety fallback: if
  the manager never runs, the physical screen still displays. Never hard-code
  `DP-1 … mirror,HEADLESS-1` statically — at cold boot `HEADLESS-1` doesn't exist
  yet and the screen could blank. The mirror is only ever added at runtime.
- **Hotplug:** Omarchy's own `omarchy-hyprland-monitor-watch` only handles
  `monitorremoved`; our `watch` handles `monitoradded` to re-impose the mirror
  when the monitor is plugged back in.

---

## The network side (this bit is what actually blocks connections)

Streaming failing is usually **not** Sunshine. Two real blockers hit on JARVIS:

### 1. ufw rules pinned to the OLD subnet (the real outage)

The omarchy-moonlight installer opens the Sunshine ports in **ufw scoped to the
LAN subnet**. After the LAN was renumbered `192.168.1.0/24 → 10.0.0.0/24`, the
rules still said `192.168.1.0/24`, so every client on `10.0.0.x` was silently
rejected (default-deny input) while `ssh` — rule `Anywhere` — kept working.

```bash
# symptom: from another LAN host, tcp/22 OPEN but 47984/47989/48010 BLOCKED,
# yet the ports listen on 0.0.0.0 and answer locally.
sudo ufw status verbose
# fix — re-scope to the current subnet:
sudo ufw allow from 10.0.0.0/24 to any port 47984,47989,47990,48010 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 47998,47999,48000,48010 proto udp
sudo ufw reload
# then delete the stale 192.168.1.0/24 rules (ufw status numbered; ufw delete N)
```

> **Rule of thumb: re-scope the omarchy-moonlight ufw rules after any network
> renumber.** The ports are TCP `47984/47989/47990/48010` + UDP
> `47998/47999/48000/48010`. `47990` (web UI) can stay closed — it's
> localhost-locked anyway.

### 2. mDNS resolves to the Docker bridge

With Docker running, avahi advertises on the docker interfaces and
`JARVIS.local` resolves to `172.17.0.1` (docker0) or an IPv6 link-local address —
not `10.0.0.13`. Moonlight auto-discovery then targets an unreachable address.
**Workaround: add the host in Moonlight by IP `10.0.0.13`.** Proper fix: restrict
avahi to the real NIC(s):

```bash
# /etc/avahi/avahi-daemon.conf, under [server]:
#   allow-interfaces=enp7s0,wlan0
#   deny-interfaces=docker0,br-8482440f28f0
sudo systemctl restart avahi-daemon
avahi-resolve -4 -n JARVIS.local     # should print 10.0.0.13
```

---

## Verify

```bash
# headless primary at 0x0, workspaces on it, DP-1 (if present) mirroring it
hyprctl monitors all -j | jq -r '.[] | "\(.name) pos=\(.x)x\(.y) mirrorOf=\(.mirrorOf // "-")"'
hyprctl workspaces -j  | jq -r '.[] | "ws \(.id) -> \(.monitor)"'

# manager is idempotent (run twice; stays one headless)
~/.local/share/omarchy-moonlight/bin/sunshine-headless-primary.sh apply

# hotplug watcher alive
pgrep -af 'sunshine-headless-primary.sh watch'

# from another LAN host: all Sunshine ports reachable
for p in 47984 47989 48010; do nc -vz 10.0.0.13 $p; done
```

Then connect Moonlight (add host by IP `10.0.0.13`) → you should see your desktop
whether or not the physical monitor is on.

---

## Gotchas & recovery

- **Stuck on the wrong/empty workspace after a monitor state change** — re-run
  the manager, or nudge manually:
  ```bash
  ~/.local/share/omarchy-moonlight/bin/sunshine-headless-primary.sh apply
  # or, targeted:
  hyprctl dispatch moveworkspacetomonitor "1 HEADLESS-1"; hyprctl dispatch workspace 1
  ```
- **`hyprctl output remove HEADLESS-N`** returns "output not found" and exits 0
  for these — use `hyprctl keyword monitor "HEADLESS-N,disable"` or reboot.
- **Re-running the omarchy-moonlight `install.sh`** regenerates `sunshine.conf`
  (re-enabling `global_prep_cmd`) and re-adds subnet-scoped ufw rules. Re-apply
  the headless-primary `sunshine.conf` settings and re-check ufw after any
  reinstall.
- **Admin UI** is localhost-only (`origin_web_ui_allowed = pc`). Reach it via
  `ssh -L 47990:localhost:47990 lwoodard@10.0.0.13` then `https://localhost:47990`
  (note: `-L` local forward, not `-R`).

See also `docs/ARCHITECTURE.md`, `docs/TROUBLESHOOTING.md`, `docs/FOLLOWUPS.md`.