muse-crew 0.4.5 → 0.4.7

package/lib/AGENTS.md

@@ -4,12 +4,13 @@ Shell scripts for the crew's infrastructure. Called by workflow scripts, cron, a
 
 - `build-registry.js` — deterministic extractor that generates `workflows/registry.json` (workflow step registry) from the workflow files' `meta` blocks at release time; invoked by `crew-release.sh` deploy
 - `crew-release.sh` — immutable release manager: deploy, rollback, prune
-- `merge-lock.sh` — serialized merge lock for concurrent agents; records owner PID
-- `worktree-lifecycle.sh` — the worktree lifecycle seam: prepare/cleanup/inspect/integrate/status/post-deploy over git worktrees (`.worktrees/<id>`, branch `task/<id>`). The crew registry (`<repo>/.worktrees/.registry/<id>`) is the source of truth for task→branch/path — never reconstruct it from git state. Prepare fails closed on dirty `main`; cleanup is forgiving.
+- `merge-lock.sh` — serialized merge lock for concurrent agents: time-based holder lease (bug 2fc8f52f — an unexpired lease is held regardless of process liveness; only an expired lease may be broken). Lock file is key=value: task_id, opaque holder identity (never a PID), acquired_at epoch, lease_seconds (default 600, override via MERGE_LOCK_LEASE_SECONDS). acquire/refresh/release/status/force-release; holder-only refresh and release; every op appends to $CREW_HOME/.merge-lock.log
+- `worktree-lifecycle.sh` — the worktree lifecycle seam: prepare/cleanup/inspect/integrate/status/post-deploy/refresh-lock/lock-status over git worktrees (`.worktrees/<id>`, branch `task/<id>`). The crew registry (`<repo>/.worktrees/.registry/<id>`) is the source of truth for task→branch/path — never reconstruct it from git state. Prepare fails closed on dirty `main`; cleanup is forgiving. `lock-status` reports the merge-lock state explicitly (`UNLOCKED`, or `LOCKED by <holder> since <ts> (pid <pid>)`, always exit 0) so Publish can distinguish an empty-diff Integrate (no lock taken) from a refresh failure.
 - `test-worktree-backend.sh` — regression tests for the lifecycle script (validate, prepare/reuse, inspect, status, cleanup, idempotent cleanup, dirty-main preflight) on a scratch repo
-- `test-version-write.sh` — regression tests for the escape-preserving step-8 version write in publish-npm.sh (fixture: current package.json with the \u2014 escape; extracts the shipped block by anchor)
+- `test-version-write.sh` — regression tests for the escape-preserving step-8 version write in publish-npm.sh (fixture: current package.json with the \\u2014 escape; extracts the shipped block by anchor)
 - `test-publish-verify.sh` — regression tests for the retry-tolerant step-12 verification in publish-npm.sh (canary 5a027278): extracts the shipped block by anchor and runs it against a fake npm whose read replica lags (non-zero exits, then the old version, then the target) — requires convergence on success, fail-closed `PUBLISH_FAILED=verify` on exhaustion, and `--prefer-online` on every read
-- `orphan-sweep.sh` — find and clean stale worktrees and merge locks
+- `test-publish-skip.sh` — regression tests for the no-lock graceful publish skip (park 2026-09-11): `lock-status` returns exact `UNLOCKED`/`LOCKED by <holder> since <ts> (pid <pid>)`; a fake empty-diff integrate takes the MERGED_EMPTY branch and never takes the lock; the shipped step-5 block from publish-npm.sh (extracted by anchor) skips gracefully on UNLOCKED (no park, no `PUBLISH_COMPLETE`), follows the refresh path when a lock is held, and fails closed on a `lock-status` query failure; the gate sits inside the ALREADY_PUBLISHED=0 branch and mutation steps are gated on NO_LOCK_HELD=0
+- `orphan-sweep.sh` — find and clean stale worktrees and merge locks: parses `merge-lock.sh status` key=value fields (never a PID); a lock whose task has a running session is untouched regardless of lease age; a lock with no running session is broken only when its lease is expired (via merge-lock.sh release so the break lands in the audit log)
 - `publish-npm.sh` — deterministic npm publish: lock refresh, release install, version write/commit, pack, registry publish, verify, push, post-deploy. Takes TARGET_VERSION as input; idempotent on retry/resume.
 - `compose-evidence.py` — deterministic visual-evidence compositor (Pillow): pairs identical PNG stems from baseline/ and postchange/ dirs, emits `<stem>-sidebyside.png` and amplified-difference `<stem>-overlay.png` into composites/, prints `PAIR`/`SKIP` manifests. Byte-deterministic; nonzero exit on errors.
 - `test-orphan-sweep.sh` — regression tests for orphan-sweep.sh (active-run guard, verified removal, fail-closed)

package/lib/merge-lock.sh

@@ -1,16 +1,89 @@
 #!/usr/bin/env bash
-# Serialized merge lock for the orchestra-dashboard repo.
+# merge-lock.sh — serialized merge lock for the crew's repos.
+#
+# Time-based holder lease (bug 2fc8f52f): the old PID-liveness staleness
+# check never excluded anyone. The recorded PID was the short-lived shell
+# that ran the acquire, so for essentially the whole Integrate→Publish→
+# post-deploy hold window the holder's PID was dead, every concurrent
+# acquire took the STALE path, and two Publish phases both believed they
+# owned version assignment. PID is the wrong liveness signal (orphan-sweep
+# already treats it as such) — the lease is time-based and refreshed by
+# the workflow while the hold is live.
+#
+# The lock file ($CREW_REPO/.worktrees/.merge-lock) is key=value:
+#   task_id=<task id that holds the lock>
+#   holder=<opaque task+run identity, never a PID>
+#   acquired_at=<epoch seconds; refresh rewrites it to extend the lease>
+#   lease_seconds=<lease window; MERGE_LOCK_LEASE_SECONDS, default 600>
+#
+# The lock is HELD while now - acquired_at < lease_seconds. An expired
+# (or malformed) lease is broken with logging and re-acquired. Every op
+# appends one line to $CREW_HOME/.merge-lock.log (the audit trail).
+#
 # Usage:
-#   merge-lock.sh acquire <task_id> [pid]  — acquires lock, exits 0 on success, 1 if held
-#   merge-lock.sh release <task_id>        — releases lock if held by this task
-#   merge-lock.sh refresh <task_id> [pid]  — refresh lock timestamp (resets staleness clock)
-#   merge-lock.sh status                   — prints current lock holder or "unlocked"
-#   merge-lock.sh force-release            — breaks the lock (admin use only)
+#   merge-lock.sh acquire <task_id> <holder>  — acquire the lock (0 ok, 1 held)
+#   merge-lock.sh refresh <task_id>           — extend the lease (holder task only)
+#   merge-lock.sh release <task_id>           — release the lock (holder task only)
+#   merge-lock.sh status                      — key=value holder/age/remaining, or UNLOCKED
+#   merge-lock.sh force-release                — break the lock (admin use only)
 
 set -euo pipefail
 
 LOCK_DIR="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}/.worktrees"
 LOCK_FILE="$LOCK_DIR/.merge-lock"
+LOG_FILE="${CREW_HOME:-$HOME/workspace/.jarvis}/.merge-lock.log"
+LEASE_SECONDS="${MERGE_LOCK_LEASE_SECONDS:-600}"
+
+log_op() { # log_op <op> <detail> — audit trail; logging never breaks a lock op
+  mkdir -p "$(dirname "$LOG_FILE")" 2>/dev/null || true
+  printf '%s op=%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" "$2" >> "$LOG_FILE" 2>/dev/null || true
+}
+
+# read_lock — parse the lock file into lock_task_id / lock_holder /
+# lock_acquired_at / lock_lease_seconds. Returns 1 when there is no lock.
+read_lock() {
+  lock_task_id=""; lock_holder=""; lock_acquired_at=""; lock_lease_seconds=""
+  [ -f "$LOCK_FILE" ] || return 1
+  local key val
+  while IFS='=' read -r key val; do
+    case "$key" in
+      task_id) lock_task_id="$val" ;;
+      holder) lock_holder="$val" ;;
+      acquired_at) lock_acquired_at="$val" ;;
+      lease_seconds) lock_lease_seconds="$val" ;;
+    esac
+  done < "$LOCK_FILE"
+  return 0
+}
+
+lock_age() { # seconds since acquired_at, or "unknown" when unparsable
+  if [[ "${lock_acquired_at:-}" =~ ^[0-9]+$ ]]; then
+    echo $(( $(date +%s) - lock_acquired_at ))
+  else
+    echo "unknown"
+  fi
+}
+
+lock_remaining() { # lease seconds remaining (<=0 means expired), or "unknown"
+  local age="${1:-$(lock_age)}" lease="$lock_lease_seconds"
+  [[ "$lease" =~ ^[0-9]+$ ]] || lease="$LEASE_SECONDS"
+  if [ "$age" = "unknown" ]; then
+    echo "unknown"
+  else
+    echo $(( lease - age ))
+  fi
+}
+
+# write_lock <task_id> <holder> — atomic create via noclobber; 0 on success.
+# A concurrent writer wins exactly one create; the loser retries the read.
+write_lock() {
+  local task_id="$1" holder="$2"
+  mkdir -p "$LOCK_DIR"
+  ( set -C
+    printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
+      "$task_id" "$holder" "$(date +%s)" "$LEASE_SECONDS" > "$LOCK_FILE"
+  ) 2>/dev/null
+}
 
 cmd="${1:-}"
 task_id="${2:-}"
@@ -18,90 +91,98 @@ task_id="${2:-}"
 case "$cmd" in
   acquire)
     [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
-    owner_pid="${3:-$PPID}"
-    mkdir -p "$LOCK_DIR"
-    if (set -C; echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $owner_pid" > "$LOCK_FILE") 2>/dev/null; then
-      echo "ACQUIRED by $task_id"
+    holder="${3:-$task_id}"
+    if write_lock "$task_id" "$holder"; then
+      log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
+      echo "ACQUIRED by $task_id (holder $holder)"
       exit 0
-    else
-      holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-      acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-      lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
-      # Self-healing: if the holder's PID is dead, missing, or invalid, the lock
-      # is stale (the workflow that acquired it terminated without releasing
-      # via post-deploy). Break the stale lock and retry the acquire. A live
-      # PID means the lock is genuinely held — report HELD. This makes terminal
-      # cleanup mechanical: a stopped/failed workflow's lock is reclaimed on
-      # the next acquire, not via manual release or a background sweeper.
-      if [ -z "$lock_pid" ] || [ "$lock_pid" = "-" ] || ! kill -0 "$lock_pid" 2>/dev/null; then
-        echo "STALE: lock held by $holder since $acquired_at (pid $lock_pid dead) — breaking" >&2
-        rm -f "$LOCK_FILE"
-        if (set -C; echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $owner_pid" > "$LOCK_FILE") 2>/dev/null; then
-          echo "ACQUIRED by $task_id (reclaimed stale lock from $holder)"
-          exit 0
-        fi
-        # Lost the race: another task acquired it after we broke the stale lock.
-        holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-        acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-        lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
+    fi
+    # A lock file exists (or raced into existence) — read it.
+    if ! read_lock; then
+      # Raced with a release between create and read: retry the create once.
+      if write_lock "$task_id" "$holder"; then
+        log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
+        echo "ACQUIRED by $task_id (holder $holder)"
+        exit 0
       fi
-      echo "HELD by $holder since $acquired_at (pid $lock_pid)"
+      read_lock || { echo "ERROR: lock state unreadable"; log_op acquire "task_id=$task_id holder=$holder result=ERROR"; exit 1; }
+    fi
+    remaining="$(lock_remaining)"
+    if [ "$remaining" != "unknown" ] && [ "$remaining" -gt 0 ]; then
+      log_op acquire "task_id=$task_id holder=$holder result=HELD holder_task=$lock_task_id holder_id=$lock_holder remaining=${remaining}s"
+      echo "HELD by $lock_task_id (holder $lock_holder, ${remaining}s of lease remaining)"
       exit 1
     fi
+    # Expired or malformed lease — break it (logged) and re-acquire.
+    echo "STALE: lease for ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) expired — breaking" >&2
+    log_op acquire "task_id=$task_id holder=$holder result=STALE-BREAK previous_task=${lock_task_id:-unknown} previous_holder=${lock_holder:-unknown}"
+    rm -f "$LOCK_FILE"
+    if write_lock "$task_id" "$holder"; then
+      log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED-RECLAIMED previous_task=${lock_task_id:-unknown}"
+      echo "ACQUIRED by $task_id (holder $holder; reclaimed expired lease from ${lock_task_id:-unknown})"
+      exit 0
+    fi
+    # Lost the race: whoever won the create holds it now.
+    read_lock 2>/dev/null || true
+    log_op acquire "task_id=$task_id holder=$holder result=HELD-RACE holder_task=${lock_task_id:-unknown}"
+    echo "HELD by ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) — lost reclaim race"
+    exit 1
+    ;;
+  refresh)
+    [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
+    if ! read_lock; then
+      log_op refresh "task_id=$task_id result=NO-LOCK"
+      echo "ERROR: no lock to refresh"
+      exit 1
+    fi
+    if [ "$lock_task_id" != "$task_id" ]; then
+      log_op refresh "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
+      echo "ERROR: lock held by $lock_task_id, not $task_id"
+      exit 1
+    fi
+    # Extend the lease: acquired_at=now, holder and lease window unchanged.
+    printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
+      "$lock_task_id" "$lock_holder" "$(date +%s)" "${lock_lease_seconds:-$LEASE_SECONDS}" > "$LOCK_FILE"
+    log_op refresh "task_id=$task_id holder=$lock_holder result=REFRESHED"
+    echo "REFRESHED by $task_id (holder $lock_holder)"
+    exit 0
     ;;
   release)
     [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
-    if [ -f "$LOCK_FILE" ]; then
-      holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "")
-      if [ "$holder" = "$task_id" ]; then
-        rm -f "$LOCK_FILE"
-        echo "RELEASED by $task_id"
-        exit 0
-      else
-        echo "ERROR: lock held by $holder, not $task_id"
-        exit 1
-      fi
-    else
+    if ! read_lock; then
+      log_op release "task_id=$task_id result=NOT-LOCKED"
       echo "RELEASED (was not locked)"
       exit 0
     fi
+    if [ "$lock_task_id" = "$task_id" ]; then
+      rm -f "$LOCK_FILE"
+      log_op release "task_id=$task_id holder=$lock_holder result=RELEASED"
+      echo "RELEASED by $task_id"
+      exit 0
+    fi
+    log_op release "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
+    echo "ERROR: lock held by $lock_task_id, not $task_id"
+    exit 1
     ;;
   status)
-    if [ -f "$LOCK_FILE" ]; then
-      holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-      acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-      lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
-      echo "LOCKED by $holder since $acquired_at (pid $lock_pid)"
-    else
+    if ! read_lock; then
       echo "UNLOCKED"
+      exit 0
     fi
+    printf 'locked=true\ntask_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\nage_seconds=%s\nremaining_seconds=%s\n' \
+      "$lock_task_id" "$lock_holder" "$lock_acquired_at" "${lock_lease_seconds:-$LEASE_SECONDS}" \
+      "$(lock_age)" "$(lock_remaining)"
     exit 0
     ;;
   force-release)
+    read_lock 2>/dev/null || true
+    log_op force-release "result=FORCE-RELEASED previous_task=${lock_task_id:-unknown}"
     rm -f "$LOCK_FILE"
     echo "FORCE RELEASED"
     exit 0
     ;;
-  refresh)
-    [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
-    new_pid="${3:-$PPID}"
-    if [ -f "$LOCK_FILE" ]; then
-      holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "")
-      if [ "$holder" = "$task_id" ]; then
-        echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $new_pid" > "$LOCK_FILE"
-        echo "REFRESHED by $task_id"
-        exit 0
-      else
-        echo "ERROR: lock held by $holder, not $task_id"
-        exit 1
-      fi
-    else
-      echo "ERROR: no lock to refresh"
-      exit 1
-    fi
-    ;;
   *)
-    echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release} [task_id] [pid]"
+    echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release} <task_id> [holder]"
     exit 1
     ;;
 esac

package/lib/orphan-sweep.sh

@@ -4,9 +4,9 @@
 #
 # Usage:
 #   orphan-sweep.sh report  — list orphans (read-only for worktrees);
-#                              stale locks with dead PIDs are released
+#                              expired-lease locks are released
 #   orphan-sweep.sh clean   — remove safe-to-clean orphans (merged branches only),
-#                              release stale merge locks, and remove pins whose
+#                              release expired-lease merge locks, and remove pins whose
 #                              task has no running session
 #
 # Active-run knowledge is injected by the caller, never fetched here:
@@ -44,7 +44,7 @@ set -euo pipefail
 REPO="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}"
 WORKTREE_DIR="$REPO/.worktrees"
 LOCK_FILE="$WORKTREE_DIR/.merge-lock"
-STALE_LOCK_MIN=30
+MERGE_LOCK_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/merge-lock.sh"
 
 cmd="${1:-report}"
 found=0
@@ -100,40 +100,44 @@ if [ "$cmd" != "clean" ] && [ "$have_active_data" -eq 0 ]; then
   echo "NOTE: no active-run data provided; ACTIVE checks skipped"
 fi
 
-# --- Stale merge lock ---
+# --- Stale merge lock (lease-based; bug 2fc8f52f) ---
+# Owner identity comes from merge-lock.sh status key=value fields — never
+# from a PID. Fail-closed: a lock whose task has a running session is
+# untouched regardless of lease age; a lock whose task has no running
+# session is broken only when its lease is expired. The break goes through
+# merge-lock.sh release (or force-release for an unparseable file with no
+# identifiable holder) so it lands in the merge-lock audit log.
 if [ -f "$LOCK_FILE" ]; then
-  lock_holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
-  if task_is_active "$lock_holder"; then
-    echo "ACTIVE_LOCK: held by $lock_holder (active run on dashboard) — skipping"
-    found=1
+  lock_status="$("$MERGE_LOCK_SCRIPT" status 2>/dev/null || echo UNLOCKED)"
+  if [ "$lock_status" = "UNLOCKED" ]; then
+    : # lock file vanished between the check and the read — nothing held
   else
-    lock_time=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "")
-    lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "")
-    if [ -n "$lock_time" ]; then
-      lock_epoch=$(date -d "$lock_time" +%s 2>/dev/null || echo 0)
-      now_epoch=$(date -u +%s)
-      age_min=$(( (now_epoch - lock_epoch) / 60 ))
-      if [ "$age_min" -gt "$STALE_LOCK_MIN" ]; then
-        # Check if owner process is still alive
-        pid_alive=0
-        if [ -n "$lock_pid" ] && [ "$lock_pid" != "-" ] && kill -0 "$lock_pid" 2>/dev/null; then
-          pid_alive=1
-        fi
-
-        if [ "$pid_alive" -eq 1 ]; then
-          echo "ACTIVE_LOCK: held ${age_min}m by $lock_holder (pid $lock_pid alive) — skipping"
-        else
-          echo "STALE_LOCK: held ${age_min}m by $lock_holder (pid ${lock_pid:-none} dead) — threshold ${STALE_LOCK_MIN}m"
-          found=1
-          rm -f "$LOCK_FILE"
-          if [ ! -f "$LOCK_FILE" ]; then
-            echo "  → released (dead PID)"
-          else
-            echo "  → FAILED: lock file still present after rm"
-            any_failed=1
-          fi
-        fi
+    lock_holder=$(printf '%s\n' "$lock_status" | sed -n 's/^task_id=//p' | head -1)
+    lock_remaining=$(printf '%s\n' "$lock_status" | sed -n 's/^remaining_seconds=//p' | head -1)
+    lock_age=$(printf '%s\n' "$lock_status" | sed -n 's/^age_seconds=//p' | head -1)
+    if task_is_active "$lock_holder"; then
+      echo "ACTIVE_LOCK: held by ${lock_holder:-unknown} (active run on dashboard) — skipping"
+      found=1
+    elif [ "${lock_remaining:-unknown}" != "unknown" ] && [ "$lock_remaining" -gt 0 ]; then
+      echo "HELD_LOCK: held by ${lock_holder:-unknown} (lease ${lock_remaining}s remaining) — skipping"
+      found=1
+    else
+      echo "STALE_LOCK: held by ${lock_holder:-unknown} (age ${lock_age:-unknown}s, lease expired, no running session)"
+      found=1
+      if [ -n "$lock_holder" ]; then
+        rel_out=$("$MERGE_LOCK_SCRIPT" release "$lock_holder" 2>&1) || true
+      else
+        rel_out=$("$MERGE_LOCK_SCRIPT" force-release 2>&1) || true
       fi
+      case "$rel_out" in
+        *RELEASED*)
+          echo "  → released (lease expired)"
+          ;;
+        *)
+          echo "  → FAILED: could not release lock: $rel_out"
+          any_failed=1
+          ;;
+      esac
     fi
   fi
 fi