@ddtcorex/dsh-maestro-supervisor 0.7.10 → 0.8.2

package/skills/dsh-safe-restart/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: dsh-safe-restart
-description: Use before updating any dsh-maestro-* client bundle or DSH Web asset, or before reloading the running dsh web host process; validate first and perform a user-approved host restart through the bundled guarded recipe. In-session agents use the `dsh_web_restart` tool; hand off to a human only if the tool is unavailable.
+description: Use before updating any dsh-maestro-* client bundle or DSH Web asset, before reloading the running dsh web host process, or when a rebuilt plugin's lib/ is newer than the running supervisor daemon; validate first and perform a user-approved host restart through the bundled guarded recipe. In-session agents use the `dsh_web_restart` tool; hand off to a human only if the tool is unavailable.
 compatibility: dsh
 ---
 
@@ -37,7 +37,18 @@ validate first and get explicit user consent before a real swap.
    or run a no-server composition check instead.
 3. Verify HTTP 200 and the new marker on the candidate. Retain last-known-good
    assets until the real swap has passed post-swap checks.
-4. Ask for explicit consent and timing. “restart đi” is consent; silence is
+4. Check the **second** long-lived process, not only `dsh web`: the standalone
+   `dsh-web-supervisor` daemon also caches this package's `lib/*.js` in RAM at
+   start, and it is the process that judges the new boot. A daemon older than
+   the newest build still applies the OLD rollback rules — on 2026-09-13 that
+   is exactly what rolled `dsh web` back in a loop every ~90s
+   (`rollback — degraded: This operation was aborted`) while `:3080` answered
+   in 1.4ms. Compare the daemon's start with the newest build:
+   `ps -o lstart= -p "$(pgrep -f 'lib/bin.js daemon')"` against
+   `stat -c '%y' <package>/lib/*.js`; a build newer than the daemon's start
+   means the daemon is **stale** and must be reloaded BEFORE the swap.
+   `restart-dsh-web.sh --check-supervisor` reports this and changes nothing.
+5. Ask for explicit consent and timing. “restart đi” is consent; silence is
    not.
 
 ## Run the bundled helper only after consent
@@ -62,6 +73,52 @@ the append-only log destination. The helper refuses a real swap without
 and refuses to launch if ports are still occupied. `--dry-run` never runs
 `kill` or `setsid`.
 
+The helper reloads a stale supervisor daemon itself, before it stops `dsh web`,
+and aborts the swap (exit 70) when that reload fails — keeping the old process
+beats handing a fresh boot to old rollback rules. Both daemon modes are
+standalone: no `--repo`, no listeners, no `--confirm`, and they never touch
+`dsh web`.
+
+```bash
+bash <skill-resource-base>/scripts/restart-dsh-web.sh --check-supervisor   # fresh|stale|absent
+bash <skill-resource-base>/scripts/restart-dsh-web.sh --reload-supervisor  # reload only when stale
+```
+
+`DSH_SUPERVISOR_RELOAD_WAIT` (seconds, default 15) bounds the wait for systemd
+to bring the daemon back. Only systemd owns that relaunch
+(`dsh-web-supervisor.service` is `Restart=always`), so a hand-started daemon is
+reported and left running, never killed.
+
+## In-session validation without restart
+
+In-session agents have three tools (same row as `dsh_web_restart`) that never
+touch the live process:
+
+- `dsh_web_dryboot` (`{ timeoutMs? }`) — boots a copy of the live profile on an
+  ephemeral port with an isolated `DSH_HOME` and returns `{ ok, detail }`.
+  Prefer this over hand-rolled `cp -r` + spawn: the copy repairs relative
+  `link:` symlinks and the child is always killed with the temp home removed.
+- `dsh_web_gc` (`{ confirm? }`) — lists orphaned dry-boot processes
+  (`DSH_HOME=/tmp/dsh-dryboot-*` + ephemeral listener, never self / live ports
+  / real-home processes). Default returns the preview list; `confirm:true`
+  SIGKILLs and verifies absence.
+- `dsh_web_restart_status` (no params) — reads the calling session's restart
+  outcome: `pending` until the daemon swaps, then `ok`/`failed` with
+  `oldPid`/`newPid`/`httpStatus`. `dsh_web_restart` itself now returns
+  `{ ok, detail, oldPid, intentPath }` — quote all four when reporting.
+
+Settings-staging rule: config-lib memoizes settings per process, so an
+out-of-band settings edit is invisible to the host until restart. Stage ALL
+such edits, then restart ONCE. Prefer in-host Settings UI saves — they are
+visible immediately with no restart at all.
+
+TLS note for plugin authors: Node/undici reads `NODE_EXTRA_CA_CERTS` from the
+birth environment only; assigning it at runtime is silently ignored. The
+durable path for a local CA (e.g. a Govard/Caddy dev CA) is a systemd user
+drop-in (`~/.config/systemd/user/dsh-web.service.d/*.conf` with
+`Environment=NODE_EXTRA_CA_CERTS=<path>`), followed by a daemon reload and a
+host restart.
+
 ## Post-swap checks
 
 Do not read the top of an old append-only log as liveness evidence. Instead:
@@ -84,6 +141,8 @@ Do not read the top of an old append-only log as liveness evidence. Instead:
 - Restart for a client-only or static change.
 - Hard-code a PID or a local developer's workspace path.
 - Treat a 200 response alone as proof that a rebuilt plugin was loaded.
+- Treat a rebuilt `lib/` as loaded while the supervisor daemon that judges the
+  boot still runs the previous one; the two processes load the code separately.
 
 ## Agent handoff (required when you are an agent)
 
@@ -92,3 +151,4 @@ If you are running inside `dsh web` (any `dsh-*` skill, `maestro-*` skill, or su
 1. Run all preflight checks (`--dry-run`, `pnpm verify`, marker grep) and report the results.
 2. Run the `dsh_web_restart` tool instead of executing the helper — the helper is for external agents and humans.
 3. If the tool reports it is unavailable, print the exact detached `setsid nohup ... --confirm` command for the human to run; then wait for explicit human confirmation in a new terminal. The interrupted turn will rehydrate from the log when the browser reconnects — no data is lost.
+4. A stale supervisor daemon blocks a correct restart just as much as a stale `lib/`, and in-session you cannot run the helper (`restart-dsh-web` in a command is refused by the self-kill guard, as is any `systemctl`). Compare the daemon start (`ps -o lstart= -p "$(pgrep -f 'lib/bin.js daemon')"` — quote the pid you actually resolved) with `stat -c '%y' <package>/lib/*.js`; when the build is newer, reload with a bare `kill -TERM <daemon pid>` after confirming `/proc/<pid>/cmdline` contains `bin.js daemon`. systemd restarts it in ~2s and `dsh web`, the browser and the session are untouched — never signal the `dsh web` pid for this. If that form is refused, or the daemon does not come back as a new pid, stop and hand the human the exact pid plus the reload command (same handoff as step 3); never fall back to `systemctl` or a broad `pkill`.

package/skills/dsh-safe-restart/scripts/restart-dsh-web.sh

@@ -27,6 +27,8 @@ PLUGIN_DIR="${DSH_SUPERVISOR_PLUGIN_DIR:-$(cd "$(dirname "$SCRIPT_SRC")/../../..
 confirmed=false
 dry_run=false
 auto_mode=false
+check_supervisor=false
+reload_supervisor=false
 
 dry_boot_and_verify() {
   local dsh_repo="$1"
@@ -57,16 +59,25 @@ dry_boot_and_verify() {
 usage() {
   cat <<'EOF'
 Usage: restart-dsh-web.sh --repo <deepseek-harness> [--log <path>] [--confirm|--auto] [--dry-run]
+       restart-dsh-web.sh --check-supervisor
+       restart-dsh-web.sh --reload-supervisor
 
 Safely hand over the DSH Web process that owns ports 3000 and 3080.
 
 Options:
-  --repo <path>  DeepSeek Harness checkout (or set DSH_REPO).
-  --log <path>   Append-only launch log (or set DSH_RESTART_LOG).
-  --confirm      Permit a real process handover (human-gated). Required unless --dry-run.
-  --auto         Permit auto handover (supervisor, no consent prompt). Alias for --confirm with auto log prefix.
-  --dry-run      Print the resolved process tree; never stop or launch anything.
-  -h, --help     Show this help text.
+  --repo <path>        DeepSeek Harness checkout (or set DSH_REPO).
+  --log <path>         Append-only launch log (or set DSH_RESTART_LOG).
+  --confirm            Permit a real process handover (human-gated). Required unless --dry-run.
+  --auto               Permit auto handover (supervisor, no consent prompt). Alias for --confirm with auto log prefix.
+  --dry-run            Print the resolved process tree; never stop or launch anything.
+  --check-supervisor   Report supervisor-daemon freshness (fresh|stale|absent); changes nothing.
+  --reload-supervisor  Reload the daemon only when it predates the newest lib/*.js build.
+  -h, --help           Show this help text.
+
+The supervisor daemon caches this package's lib/*.js in RAM like `dsh web`
+does, and a stale daemon rolls back a healthy boot with its old rules. Every
+real restart therefore reloads it first when stale; DSH_SUPERVISOR_RELOAD_WAIT
+(seconds, default 15) bounds the wait for systemd to bring it back.
 EOF
 }
 
@@ -75,6 +86,120 @@ fail() {
   exit "${2:-1}"
 }
 
+# --- supervisor daemon freshness -------------------------------------------------
+# `dsh web` is not the only process that caches this package's lib/*.js in RAM:
+# the standalone dsh-web-supervisor daemon does too, and it is the process that
+# decides whether a boot is healthy or must be rolled back. A daemon started
+# BEFORE the newest build keeps running the old rollback rules — on 2026-09-13
+# exactly that judged a slow boot as down and rolled `dsh web` back in a loop
+# ("rollback - degraded: This operation was aborted") while the port answered in
+# 1.4ms. So: never swap `dsh web` under a stale daemon.
+#
+# Only systemd owns a relaunch (dsh-web-supervisor.service has Restart=always);
+# a daemon someone started by hand has no owner to bring it back, so this step
+# reports and leaves it alone rather than killing it.
+
+# MainPID of the supervisor unit, or empty unless that pid really is the daemon.
+supervisor_main_pid() {
+  local pid cmdline
+  pid="$(systemctl --user show -p MainPID --value dsh-web-supervisor.service 2>/dev/null | tr -dc '0-9' || true)"
+  [[ -n "$pid" && "$pid" != 0 ]] || return 0
+  [[ -d "/proc/$pid" ]] || return 0
+  # Never signal a pid that is not this daemon, whatever systemd reports.
+  cmdline="$(tr '\0' ' ' <"/proc/$pid/cmdline" 2>/dev/null || true)"
+  [[ "$cmdline" == *"bin.js daemon"* ]] || return 0
+  printf '%s' "$pid"
+}
+
+# Newest mtime among the package's built lib/*.js, or empty when unbuilt.
+newest_lib_mtime() {
+  local newest="" file stamp
+  for file in "$PLUGIN_DIR"/lib/*.js; do
+    [[ -f "$file" ]] || continue
+    stamp="$(stat -c %Y "$file" 2>/dev/null || true)"
+    [[ -n "$stamp" ]] || continue
+    if [[ -z "$newest" || "$stamp" -gt "$newest" ]]; then newest="$stamp"; fi
+  done
+  printf '%s' "$newest"
+}
+
+# Classify the daemon: fresh | stale | absent | unknown.
+supervisor_daemon_state() {
+  SUPERVISOR_PID="$(supervisor_main_pid)"
+  SUPERVISOR_LIB="$(newest_lib_mtime)"
+  if [[ -z "$SUPERVISOR_PID" ]]; then
+    SUPERVISOR_VERDICT=absent
+    return 0
+  fi
+  SUPERVISOR_START="$(stat -c %Y "/proc/$SUPERVISOR_PID" 2>/dev/null || true)"
+  if [[ -z "$SUPERVISOR_START" || -z "$SUPERVISOR_LIB" ]]; then
+    SUPERVISOR_VERDICT=unknown
+    return 0
+  fi
+  if (( SUPERVISOR_LIB > SUPERVISOR_START )); then
+    SUPERVISOR_VERDICT=stale
+  else
+    SUPERVISOR_VERDICT=fresh
+  fi
+}
+
+print_supervisor_state() {
+  case "$SUPERVISOR_VERDICT" in
+    fresh) printf '[restart] supervisor daemon: fresh pid=%s start=%s lib=%s\n' "$SUPERVISOR_PID" "$SUPERVISOR_START" "$SUPERVISOR_LIB" ;;
+    stale) printf '[restart] supervisor daemon: stale pid=%s start=%s lib=%s (built after this daemon started)\n' "$SUPERVISOR_PID" "$SUPERVISOR_START" "$SUPERVISOR_LIB" ;;
+    unknown) printf '[restart] supervisor daemon: unknown pid=%s (lib build time unavailable)\n' "$SUPERVISOR_PID" ;;
+    *) printf '[restart] supervisor daemon: absent\n' ;;
+  esac
+}
+
+# Reload the daemon when it predates the newest build; report the verdict either
+# way. A failed reload is fatal: continuing would hand the swap to the very
+# process that can roll it back in a loop.
+ensure_supervisor_current() {
+  supervisor_daemon_state
+  if [[ "$SUPERVISOR_VERDICT" != stale ]]; then
+    print_supervisor_state
+    return 0
+  fi
+  local old_pid="$SUPERVISOR_PID"
+  local wait_s="${DSH_SUPERVISOR_RELOAD_WAIT:-15}"
+  local new_pid=""
+  print_supervisor_state
+  printf '[restart] reloading the supervisor daemon so it runs the built lib/*.js\n'
+  kill -TERM "$old_pid" 2>/dev/null || true
+  for _ in $(seq 1 "$wait_s"); do
+    sleep 1
+    new_pid="$(supervisor_main_pid)"
+    if [[ -n "$new_pid" && "$new_pid" != "$old_pid" ]]; then break; fi
+    new_pid=""
+  done
+  if [[ -z "$new_pid" ]]; then
+    printf '[restart] FAIL: supervisor daemon %s did not come back within %ss\n' "$old_pid" "$wait_s" >&2
+    return 1
+  fi
+  printf '[restart] supervisor daemon: reloaded %s -> %s\n' "$old_pid" "$new_pid"
+}
+
+# Single-flight + log scoping (D6, spec 2026-09-13-supervisor-safety-net-design):
+# take the SAME boot.lock and append the SAME boot-boundary sentinel as the
+# supervisor's own performSingleBootRestart, by calling the package's helper —
+# never by re-implementing either marker here. The lock is owned by this shell's
+# PID ($$), so it stays valid for as long as the script runs, and it is released
+# on every exit path via the trap below.
+BOOT_GUARD_HELPER="${PLUGIN_DIR}/lib/bin.js"
+boot_guard_acquire() {
+  if [[ ! -f "$BOOT_GUARD_HELPER" ]]; then
+    fail "boot.lock helper is missing: $BOOT_GUARD_HELPER (run 'pnpm build' in the supervisor package)" 69
+  fi
+  if ! node "$BOOT_GUARD_HELPER" boot-guard acquire --pid "$$" >>"$log" 2>&1; then
+    fail 'another boot holds boot.lock (a dsh web boot or another restart is in flight)' 75
+  fi
+}
+boot_guard_release() {
+  [[ -f "$BOOT_GUARD_HELPER" ]] || return 0
+  node "$BOOT_GUARD_HELPER" boot-guard release --pid "$$" >>"$log" 2>&1 || true
+}
+
 while (($#)); do
   case "$1" in
     --repo)
@@ -100,6 +225,14 @@ while (($#)); do
       dry_run=true
       shift
       ;;
+    --check-supervisor)
+      check_supervisor=true
+      shift
+      ;;
+    --reload-supervisor)
+      reload_supervisor=true
+      shift
+      ;;
     -h|--help)
       usage
       exit 0
@@ -110,6 +243,18 @@ while (($#)); do
   esac
 done
 
+# Standalone supervisor-daemon modes: they never touch dsh web, so they need no
+# --repo, no listeners and no consent.
+if [[ "$check_supervisor" == true ]]; then
+  supervisor_daemon_state
+  print_supervisor_state
+  exit 0
+fi
+if [[ "$reload_supervisor" == true ]]; then
+  ensure_supervisor_current || fail 'supervisor daemon could not be reloaded — fix it before swapping dsh web' 70
+  exit 0
+fi
+
 [[ -n "$repo" ]] || fail 'provide --repo or DSH_REPO' 64
 [[ -f "$repo/package.json" ]] || fail "repo has no package.json: $repo" 64
 if [[ "$dry_run" != true && "$confirmed" != true ]]; then
@@ -222,10 +367,19 @@ printf '[restart] stopping process tree: %s\n' "$(tr '\n' ' ' <<<"$tree_pids")"
 
 # Mark this as an intentional restart before the port goes down, so
 # dsh-web-supervisor's health poll does not race us with its own rollback.
-# Removed on every exit path (success or failure) via the trap.
+# The boot.lock below is the single-flight half of the same contract; both are
+# released on every exit path (success or failure) via the trap.
+mkdir -p "$(dirname "$log")"
+boot_guard_acquire
 mkdir -p "$(dirname "$marker")"
 date -Iseconds > "$marker"
-trap 'rm -f "$marker"' EXIT
+trap 'boot_guard_release; rm -f "$marker"' EXIT
+
+# Reload a stale supervisor daemon BEFORE the swap: it is the process that
+# decides whether the boot below is healthy, and a daemon running pre-build
+# rules can roll it back in a loop. Done while dsh web is still up, so the
+# fresh daemon observes the planned-restart marker written just above.
+ensure_supervisor_current || fail 'stale supervisor daemon could not be reloaded — refusing to swap dsh web under old rollback rules' 70
 
 if [[ "$systemd_managed" == true ]]; then
   # systemctl stop is a clean, intentional stop -- Restart=always does not