muse-crew 0.4.4 → 0.4.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/API.md CHANGED
@@ -71,14 +71,14 @@ Send a stuck or failed task to a specific workflow phase for recovery.
71
71
 
72
72
  ### `parktask`
73
73
 
74
- Atomically park a task for human attention: sets `state` to `parked` and writes the explanatory `note` event in one transaction. Either both land or neither does callers never see a note without a park or a park without a note. The dispatcher skips parked tasks; moving the task back to `todo` is the human re-queue path (which stamps `retry_reset_at` see `updatetask`).
74
+ Atomically park a task for human attention: sets `state` to `parked`, writes the explanatory `note` event, and settles every `running` session for the task — in one transaction. Either all land or none do: callers never see a note without a park, a park without a note, or a parked task with a ghost `running` session left behind (a terminal task transition settles its sessions). Settled sessions move to `failed` the status `recovertask` accepts, so recovery works immediately after a park — with `ended_at` stamped and `Parked: <message>` appended to their notes; `failure_reason` stays null.
75
75
 
76
76
  | Field | Type | Required | Notes |
77
77
  |-------|------|----------|-------|
78
78
  | `task_id` | uuid | yes | |
79
79
  | `message` | string (1–1000) | yes | Why the task needs a human; written to the activity feed as a `note` |
80
80
 
81
- Returns `{ "ok": true, "task": {...} }` with the updated task record.
81
+ Returns `{ "ok": true, "task": {...}, "settled_sessions": <n> }` with the updated task record and the count of sessions settled (`0` when none were running).
82
82
 
83
83
  ---
84
84
 
package/lib/AGENTS.md CHANGED
@@ -4,11 +4,12 @@ Shell scripts for the crew's infrastructure. Called by workflow scripts, cron, a
4
4
 
5
5
  - `build-registry.js` — deterministic extractor that generates `workflows/registry.json` (workflow step registry) from the workflow files' `meta` blocks at release time; invoked by `crew-release.sh` deploy
6
6
  - `crew-release.sh` — immutable release manager: deploy, rollback, prune
7
- - `merge-lock.sh` — serialized merge lock for concurrent agents; records owner PID
7
+ - `merge-lock.sh` — serialized merge lock for concurrent agents: time-based holder lease (bug 2fc8f52f — an unexpired lease is held regardless of process liveness; only an expired lease may be broken). Lock file is key=value: task_id, opaque holder identity (never a PID), acquired_at epoch, lease_seconds (default 600, override via MERGE_LOCK_LEASE_SECONDS). acquire/refresh/release/status/force-release; holder-only refresh and release; every op appends to $CREW_HOME/.merge-lock.log
8
8
  - `worktree-lifecycle.sh` — the worktree lifecycle seam: prepare/cleanup/inspect/integrate/status/post-deploy over git worktrees (`.worktrees/<id>`, branch `task/<id>`). The crew registry (`<repo>/.worktrees/.registry/<id>`) is the source of truth for task→branch/path — never reconstruct it from git state. Prepare fails closed on dirty `main`; cleanup is forgiving.
9
9
  - `test-worktree-backend.sh` — regression tests for the lifecycle script (validate, prepare/reuse, inspect, status, cleanup, idempotent cleanup, dirty-main preflight) on a scratch repo
10
10
  - `test-version-write.sh` — regression tests for the escape-preserving step-8 version write in publish-npm.sh (fixture: current package.json with the \u2014 escape; extracts the shipped block by anchor)
11
- - `orphan-sweep.sh` — find and clean stale worktrees and merge locks
11
+ - `test-publish-verify.sh` — regression tests for the retry-tolerant step-12 verification in publish-npm.sh (canary 5a027278): extracts the shipped block by anchor and runs it against a fake npm whose read replica lags (non-zero exits, then the old version, then the target) — requires convergence on success, fail-closed `PUBLISH_FAILED=verify` on exhaustion, and `--prefer-online` on every read
12
+ - `orphan-sweep.sh` — find and clean stale worktrees and merge locks: parses `merge-lock.sh status` key=value fields (never a PID); a lock whose task has a running session is untouched regardless of lease age; a lock with no running session is broken only when its lease is expired (via merge-lock.sh release so the break lands in the audit log)
12
13
  - `publish-npm.sh` — deterministic npm publish: lock refresh, release install, version write/commit, pack, registry publish, verify, push, post-deploy. Takes TARGET_VERSION as input; idempotent on retry/resume.
13
14
  - `compose-evidence.py` — deterministic visual-evidence compositor (Pillow): pairs identical PNG stems from baseline/ and postchange/ dirs, emits `<stem>-sidebyside.png` and amplified-difference `<stem>-overlay.png` into composites/, prints `PAIR`/`SKIP` manifests. Byte-deterministic; nonzero exit on errors.
14
15
  - `test-orphan-sweep.sh` — regression tests for orphan-sweep.sh (active-run guard, verified removal, fail-closed)
package/lib/merge-lock.sh CHANGED
@@ -1,16 +1,89 @@
1
1
  #!/usr/bin/env bash
2
- # Serialized merge lock for the orchestra-dashboard repo.
2
+ # merge-lock.sh — serialized merge lock for the crew's repos.
3
+ #
4
+ # Time-based holder lease (bug 2fc8f52f): the old PID-liveness staleness
5
+ # check never excluded anyone. The recorded PID was the short-lived shell
6
+ # that ran the acquire, so for essentially the whole Integrate→Publish→
7
+ # post-deploy hold window the holder's PID was dead, every concurrent
8
+ # acquire took the STALE path, and two Publish phases both believed they
9
+ # owned version assignment. PID is the wrong liveness signal (orphan-sweep
10
+ # already treats it as such) — the lease is time-based and refreshed by
11
+ # the workflow while the hold is live.
12
+ #
13
+ # The lock file ($CREW_REPO/.worktrees/.merge-lock) is key=value:
14
+ # task_id=<task id that holds the lock>
15
+ # holder=<opaque task+run identity, never a PID>
16
+ # acquired_at=<epoch seconds; refresh rewrites it to extend the lease>
17
+ # lease_seconds=<lease window; MERGE_LOCK_LEASE_SECONDS, default 600>
18
+ #
19
+ # The lock is HELD while now - acquired_at < lease_seconds. An expired
20
+ # (or malformed) lease is broken with logging and re-acquired. Every op
21
+ # appends one line to $CREW_HOME/.merge-lock.log (the audit trail).
22
+ #
3
23
  # Usage:
4
- # merge-lock.sh acquire <task_id> [pid]acquires lock, exits 0 on success, 1 if held
5
- # merge-lock.sh release <task_id> releases lock if held by this task
6
- # merge-lock.sh refresh <task_id> [pid] refresh lock timestamp (resets staleness clock)
7
- # merge-lock.sh status prints current lock holder or "unlocked"
8
- # merge-lock.sh force-release breaks the lock (admin use only)
24
+ # merge-lock.sh acquire <task_id> <holder>acquire the lock (0 ok, 1 held)
25
+ # merge-lock.sh refresh <task_id> extend the lease (holder task only)
26
+ # merge-lock.sh release <task_id> release the lock (holder task only)
27
+ # merge-lock.sh status key=value holder/age/remaining, or UNLOCKED
28
+ # merge-lock.sh force-release break the lock (admin use only)
9
29
 
10
30
  set -euo pipefail
11
31
 
12
32
  LOCK_DIR="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}/.worktrees"
13
33
  LOCK_FILE="$LOCK_DIR/.merge-lock"
34
+ LOG_FILE="${CREW_HOME:-$HOME/workspace/.jarvis}/.merge-lock.log"
35
+ LEASE_SECONDS="${MERGE_LOCK_LEASE_SECONDS:-600}"
36
+
37
+ log_op() { # log_op <op> <detail> — audit trail; logging never breaks a lock op
38
+ mkdir -p "$(dirname "$LOG_FILE")" 2>/dev/null || true
39
+ printf '%s op=%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" "$2" >> "$LOG_FILE" 2>/dev/null || true
40
+ }
41
+
42
+ # read_lock — parse the lock file into lock_task_id / lock_holder /
43
+ # lock_acquired_at / lock_lease_seconds. Returns 1 when there is no lock.
44
+ read_lock() {
45
+ lock_task_id=""; lock_holder=""; lock_acquired_at=""; lock_lease_seconds=""
46
+ [ -f "$LOCK_FILE" ] || return 1
47
+ local key val
48
+ while IFS='=' read -r key val; do
49
+ case "$key" in
50
+ task_id) lock_task_id="$val" ;;
51
+ holder) lock_holder="$val" ;;
52
+ acquired_at) lock_acquired_at="$val" ;;
53
+ lease_seconds) lock_lease_seconds="$val" ;;
54
+ esac
55
+ done < "$LOCK_FILE"
56
+ return 0
57
+ }
58
+
59
+ lock_age() { # seconds since acquired_at, or "unknown" when unparsable
60
+ if [[ "${lock_acquired_at:-}" =~ ^[0-9]+$ ]]; then
61
+ echo $(( $(date +%s) - lock_acquired_at ))
62
+ else
63
+ echo "unknown"
64
+ fi
65
+ }
66
+
67
+ lock_remaining() { # lease seconds remaining (<=0 means expired), or "unknown"
68
+ local age="${1:-$(lock_age)}" lease="$lock_lease_seconds"
69
+ [[ "$lease" =~ ^[0-9]+$ ]] || lease="$LEASE_SECONDS"
70
+ if [ "$age" = "unknown" ]; then
71
+ echo "unknown"
72
+ else
73
+ echo $(( lease - age ))
74
+ fi
75
+ }
76
+
77
+ # write_lock <task_id> <holder> — atomic create via noclobber; 0 on success.
78
+ # A concurrent writer wins exactly one create; the loser retries the read.
79
+ write_lock() {
80
+ local task_id="$1" holder="$2"
81
+ mkdir -p "$LOCK_DIR"
82
+ ( set -C
83
+ printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
84
+ "$task_id" "$holder" "$(date +%s)" "$LEASE_SECONDS" > "$LOCK_FILE"
85
+ ) 2>/dev/null
86
+ }
14
87
 
15
88
  cmd="${1:-}"
16
89
  task_id="${2:-}"
@@ -18,90 +91,98 @@ task_id="${2:-}"
18
91
  case "$cmd" in
19
92
  acquire)
20
93
  [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
21
- owner_pid="${3:-$PPID}"
22
- mkdir -p "$LOCK_DIR"
23
- if (set -C; echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $owner_pid" > "$LOCK_FILE") 2>/dev/null; then
24
- echo "ACQUIRED by $task_id"
94
+ holder="${3:-$task_id}"
95
+ if write_lock "$task_id" "$holder"; then
96
+ log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
97
+ echo "ACQUIRED by $task_id (holder $holder)"
25
98
  exit 0
26
- else
27
- holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
28
- acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
29
- lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
30
- # Self-healing: if the holder's PID is dead, missing, or invalid, the lock
31
- # is stale (the workflow that acquired it terminated without releasing
32
- # via post-deploy). Break the stale lock and retry the acquire. A live
33
- # PID means the lock is genuinely held — report HELD. This makes terminal
34
- # cleanup mechanical: a stopped/failed workflow's lock is reclaimed on
35
- # the next acquire, not via manual release or a background sweeper.
36
- if [ -z "$lock_pid" ] || [ "$lock_pid" = "-" ] || ! kill -0 "$lock_pid" 2>/dev/null; then
37
- echo "STALE: lock held by $holder since $acquired_at (pid $lock_pid dead) — breaking" >&2
38
- rm -f "$LOCK_FILE"
39
- if (set -C; echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $owner_pid" > "$LOCK_FILE") 2>/dev/null; then
40
- echo "ACQUIRED by $task_id (reclaimed stale lock from $holder)"
41
- exit 0
42
- fi
43
- # Lost the race: another task acquired it after we broke the stale lock.
44
- holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
45
- acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
46
- lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
99
+ fi
100
+ # A lock file exists (or raced into existence) — read it.
101
+ if ! read_lock; then
102
+ # Raced with a release between create and read: retry the create once.
103
+ if write_lock "$task_id" "$holder"; then
104
+ log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
105
+ echo "ACQUIRED by $task_id (holder $holder)"
106
+ exit 0
47
107
  fi
48
- echo "HELD by $holder since $acquired_at (pid $lock_pid)"
108
+ read_lock || { echo "ERROR: lock state unreadable"; log_op acquire "task_id=$task_id holder=$holder result=ERROR"; exit 1; }
109
+ fi
110
+ remaining="$(lock_remaining)"
111
+ if [ "$remaining" != "unknown" ] && [ "$remaining" -gt 0 ]; then
112
+ log_op acquire "task_id=$task_id holder=$holder result=HELD holder_task=$lock_task_id holder_id=$lock_holder remaining=${remaining}s"
113
+ echo "HELD by $lock_task_id (holder $lock_holder, ${remaining}s of lease remaining)"
49
114
  exit 1
50
115
  fi
116
+ # Expired or malformed lease — break it (logged) and re-acquire.
117
+ echo "STALE: lease for ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) expired — breaking" >&2
118
+ log_op acquire "task_id=$task_id holder=$holder result=STALE-BREAK previous_task=${lock_task_id:-unknown} previous_holder=${lock_holder:-unknown}"
119
+ rm -f "$LOCK_FILE"
120
+ if write_lock "$task_id" "$holder"; then
121
+ log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED-RECLAIMED previous_task=${lock_task_id:-unknown}"
122
+ echo "ACQUIRED by $task_id (holder $holder; reclaimed expired lease from ${lock_task_id:-unknown})"
123
+ exit 0
124
+ fi
125
+ # Lost the race: whoever won the create holds it now.
126
+ read_lock 2>/dev/null || true
127
+ log_op acquire "task_id=$task_id holder=$holder result=HELD-RACE holder_task=${lock_task_id:-unknown}"
128
+ echo "HELD by ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) — lost reclaim race"
129
+ exit 1
130
+ ;;
131
+ refresh)
132
+ [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
133
+ if ! read_lock; then
134
+ log_op refresh "task_id=$task_id result=NO-LOCK"
135
+ echo "ERROR: no lock to refresh"
136
+ exit 1
137
+ fi
138
+ if [ "$lock_task_id" != "$task_id" ]; then
139
+ log_op refresh "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
140
+ echo "ERROR: lock held by $lock_task_id, not $task_id"
141
+ exit 1
142
+ fi
143
+ # Extend the lease: acquired_at=now, holder and lease window unchanged.
144
+ printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
145
+ "$lock_task_id" "$lock_holder" "$(date +%s)" "${lock_lease_seconds:-$LEASE_SECONDS}" > "$LOCK_FILE"
146
+ log_op refresh "task_id=$task_id holder=$lock_holder result=REFRESHED"
147
+ echo "REFRESHED by $task_id (holder $lock_holder)"
148
+ exit 0
51
149
  ;;
52
150
  release)
53
151
  [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
54
- if [ -f "$LOCK_FILE" ]; then
55
- holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "")
56
- if [ "$holder" = "$task_id" ]; then
57
- rm -f "$LOCK_FILE"
58
- echo "RELEASED by $task_id"
59
- exit 0
60
- else
61
- echo "ERROR: lock held by $holder, not $task_id"
62
- exit 1
63
- fi
64
- else
152
+ if ! read_lock; then
153
+ log_op release "task_id=$task_id result=NOT-LOCKED"
65
154
  echo "RELEASED (was not locked)"
66
155
  exit 0
67
156
  fi
157
+ if [ "$lock_task_id" = "$task_id" ]; then
158
+ rm -f "$LOCK_FILE"
159
+ log_op release "task_id=$task_id holder=$lock_holder result=RELEASED"
160
+ echo "RELEASED by $task_id"
161
+ exit 0
162
+ fi
163
+ log_op release "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
164
+ echo "ERROR: lock held by $lock_task_id, not $task_id"
165
+ exit 1
68
166
  ;;
69
167
  status)
70
- if [ -f "$LOCK_FILE" ]; then
71
- holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
72
- acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
73
- lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
74
- echo "LOCKED by $holder since $acquired_at (pid $lock_pid)"
75
- else
168
+ if ! read_lock; then
76
169
  echo "UNLOCKED"
170
+ exit 0
77
171
  fi
172
+ printf 'locked=true\ntask_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\nage_seconds=%s\nremaining_seconds=%s\n' \
173
+ "$lock_task_id" "$lock_holder" "$lock_acquired_at" "${lock_lease_seconds:-$LEASE_SECONDS}" \
174
+ "$(lock_age)" "$(lock_remaining)"
78
175
  exit 0
79
176
  ;;
80
177
  force-release)
178
+ read_lock 2>/dev/null || true
179
+ log_op force-release "result=FORCE-RELEASED previous_task=${lock_task_id:-unknown}"
81
180
  rm -f "$LOCK_FILE"
82
181
  echo "FORCE RELEASED"
83
182
  exit 0
84
183
  ;;
85
- refresh)
86
- [ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
87
- new_pid="${3:-$PPID}"
88
- if [ -f "$LOCK_FILE" ]; then
89
- holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "")
90
- if [ "$holder" = "$task_id" ]; then
91
- echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $new_pid" > "$LOCK_FILE"
92
- echo "REFRESHED by $task_id"
93
- exit 0
94
- else
95
- echo "ERROR: lock held by $holder, not $task_id"
96
- exit 1
97
- fi
98
- else
99
- echo "ERROR: no lock to refresh"
100
- exit 1
101
- fi
102
- ;;
103
184
  *)
104
- echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release} [task_id] [pid]"
185
+ echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release} <task_id> [holder]"
105
186
  exit 1
106
187
  ;;
107
188
  esac
@@ -4,9 +4,9 @@
4
4
  #
5
5
  # Usage:
6
6
  # orphan-sweep.sh report — list orphans (read-only for worktrees);
7
- # stale locks with dead PIDs are released
7
+ # expired-lease locks are released
8
8
  # orphan-sweep.sh clean — remove safe-to-clean orphans (merged branches only),
9
- # release stale merge locks, and remove pins whose
9
+ # release expired-lease merge locks, and remove pins whose
10
10
  # task has no running session
11
11
  #
12
12
  # Active-run knowledge is injected by the caller, never fetched here:
@@ -44,7 +44,7 @@ set -euo pipefail
44
44
  REPO="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}"
45
45
  WORKTREE_DIR="$REPO/.worktrees"
46
46
  LOCK_FILE="$WORKTREE_DIR/.merge-lock"
47
- STALE_LOCK_MIN=30
47
+ MERGE_LOCK_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/merge-lock.sh"
48
48
 
49
49
  cmd="${1:-report}"
50
50
  found=0
@@ -100,40 +100,44 @@ if [ "$cmd" != "clean" ] && [ "$have_active_data" -eq 0 ]; then
100
100
  echo "NOTE: no active-run data provided; ACTIVE checks skipped"
101
101
  fi
102
102
 
103
- # --- Stale merge lock ---
103
+ # --- Stale merge lock (lease-based; bug 2fc8f52f) ---
104
+ # Owner identity comes from merge-lock.sh status key=value fields — never
105
+ # from a PID. Fail-closed: a lock whose task has a running session is
106
+ # untouched regardless of lease age; a lock whose task has no running
107
+ # session is broken only when its lease is expired. The break goes through
108
+ # merge-lock.sh release (or force-release for an unparseable file with no
109
+ # identifiable holder) so it lands in the merge-lock audit log.
104
110
  if [ -f "$LOCK_FILE" ]; then
105
- lock_holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
106
- if task_is_active "$lock_holder"; then
107
- echo "ACTIVE_LOCK: held by $lock_holder (active run on dashboard)skipping"
108
- found=1
111
+ lock_status="$("$MERGE_LOCK_SCRIPT" status 2>/dev/null || echo UNLOCKED)"
112
+ if [ "$lock_status" = "UNLOCKED" ]; then
113
+ : # lock file vanished between the check and the read nothing held
109
114
  else
110
- lock_time=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "")
111
- lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "")
112
- if [ -n "$lock_time" ]; then
113
- lock_epoch=$(date -d "$lock_time" +%s 2>/dev/null || echo 0)
114
- now_epoch=$(date -u +%s)
115
- age_min=$(( (now_epoch - lock_epoch) / 60 ))
116
- if [ "$age_min" -gt "$STALE_LOCK_MIN" ]; then
117
- # Check if owner process is still alive
118
- pid_alive=0
119
- if [ -n "$lock_pid" ] && [ "$lock_pid" != "-" ] && kill -0 "$lock_pid" 2>/dev/null; then
120
- pid_alive=1
121
- fi
122
-
123
- if [ "$pid_alive" -eq 1 ]; then
124
- echo "ACTIVE_LOCK: held ${age_min}m by $lock_holder (pid $lock_pid alive) — skipping"
125
- else
126
- echo "STALE_LOCK: held ${age_min}m by $lock_holder (pid ${lock_pid:-none} dead) — threshold ${STALE_LOCK_MIN}m"
127
- found=1
128
- rm -f "$LOCK_FILE"
129
- if [ ! -f "$LOCK_FILE" ]; then
130
- echo " → released (dead PID)"
131
- else
132
- echo " → FAILED: lock file still present after rm"
133
- any_failed=1
134
- fi
135
- fi
115
+ lock_holder=$(printf '%s\n' "$lock_status" | sed -n 's/^task_id=//p' | head -1)
116
+ lock_remaining=$(printf '%s\n' "$lock_status" | sed -n 's/^remaining_seconds=//p' | head -1)
117
+ lock_age=$(printf '%s\n' "$lock_status" | sed -n 's/^age_seconds=//p' | head -1)
118
+ if task_is_active "$lock_holder"; then
119
+ echo "ACTIVE_LOCK: held by ${lock_holder:-unknown} (active run on dashboard) — skipping"
120
+ found=1
121
+ elif [ "${lock_remaining:-unknown}" != "unknown" ] && [ "$lock_remaining" -gt 0 ]; then
122
+ echo "HELD_LOCK: held by ${lock_holder:-unknown} (lease ${lock_remaining}s remaining) — skipping"
123
+ found=1
124
+ else
125
+ echo "STALE_LOCK: held by ${lock_holder:-unknown} (age ${lock_age:-unknown}s, lease expired, no running session)"
126
+ found=1
127
+ if [ -n "$lock_holder" ]; then
128
+ rel_out=$("$MERGE_LOCK_SCRIPT" release "$lock_holder" 2>&1) || true
129
+ else
130
+ rel_out=$("$MERGE_LOCK_SCRIPT" force-release 2>&1) || true
136
131
  fi
132
+ case "$rel_out" in
133
+ *RELEASED*)
134
+ echo " → released (lease expired)"
135
+ ;;
136
+ *)
137
+ echo " → FAILED: could not release lock: $rel_out"
138
+ any_failed=1
139
+ ;;
140
+ esac
137
141
  fi
138
142
  fi
139
143
  fi
@@ -150,13 +150,38 @@ if [ "$ALREADY_PUBLISHED" = "0" ]; then
150
150
  rm -f "$TGZ"
151
151
 
152
152
  # 12. Verify: ground truth is the registry, not any agent's summary.
153
- REG="$(npm view "$PKG" version 2>/dev/null)" \
154
- || fail "verify" "npm view $PKG failed"
153
+ # The registry is eventually consistent: a publish followed by an
154
+ # immediate read can observe the pre-publish version (canary 5a027278
155
+ # hit a stale read replica ~6s after publish). A single stale read must
156
+ # never park a landed publish, so this block retries with backoff and
157
+ # client cache-busting instead of failing on the first mismatch.
158
+ # STEP-12-ANCHOR: publish verification (retry-tolerant)
159
+ VERIFY_ATTEMPTS="${VERIFY_ATTEMPTS:-12}"
160
+ VERIFY_SLEEP_SECS="${VERIFY_SLEEP_SECS:-10}"
161
+ REG=""
162
+ for attempt in $(seq 1 "$VERIFY_ATTEMPTS"); do
163
+ # --prefer-online busts npm's client-side packument cache; the retry
164
+ # loop absorbs server-side replica lag. A non-zero exit is a miss,
165
+ # not an immediate failure.
166
+ REG="$(npm view "$PKG" version --prefer-online 2>/dev/null || true)"
167
+ [ "$REG" = "$TARGET_VERSION" ] && break
168
+ echo "PUBLISH_VERIFY_RETRY=$attempt registry=$REG"
169
+ [ "$attempt" -lt "$VERIFY_ATTEMPTS" ] && sleep "$VERIFY_SLEEP_SECS"
170
+ done
155
171
  [ "$REG" = "$TARGET_VERSION" ] \
156
- || fail "verify" "registry=$REG"
172
+ || fail "verify" "registry=$REG after $VERIFY_ATTEMPTS attempts"
157
173
  echo "PUBLISH_VERIFIED=$TARGET_VERSION"
174
+ # STEP-12-END
158
175
  fi
159
176
 
177
+ # 12b. Refresh the merge lock again — release-deploy, pack, publish, and the
178
+ # verify retry loop can together consume more than half the lease on a slow
179
+ # registry. The push and the post-deploy below must happen under our lock;
180
+ # if the refresh fails (expired lease, someone else holds it), stop — never
181
+ # push a version-bump commit or release a lock we no longer own.
182
+ LOCK_OUT2="$(CREW_REPO="$REPO_PATH" bash "${LIFECYCLE:?LIFECYCLE is required}" refresh-lock "$TASK_ID" 2>&1)" \
183
+ || fail "lock-refresh" "$LOCK_OUT2"
184
+
160
185
  # 13. Push the version-bump commit (idempotent: no-op if already pushed).
161
186
  PUSH_OUT="$(git -C "$REPO_PATH" push origin main 2>&1)" \
162
187
  || fail "push" "$PUSH_OUT"
@@ -2,9 +2,12 @@
2
2
  # test-merge-lock.sh — regression tests for merge-lock.sh
3
3
  #
4
4
  # Self-contained: uses a temp CREW_REPO and drives lib/merge-lock.sh through
5
- # acquire/release/status. Tests the self-healing acquire: a lock held by a
6
- # dead PID is reclaimed automatically; a lock held by a live PID reports HELD.
7
- # Exits 0 only if every case passes.
5
+ # acquire/refresh/release/status. Tests the time-based lease model (bug
6
+ # 2fc8f52f): an unexpired lease blocks contenders even when no holder
7
+ # process is alive (PID liveness is never consulted); an expired lease is
8
+ # reclaimed with logging; refresh extends the lease; release is
9
+ # holder-only; status emits parseable holder/age/remaining fields; every
10
+ # op appends to the merge-lock audit log. Exits 0 only if every case passes.
8
11
 
9
12
  set -uo pipefail
10
13
 
@@ -20,49 +23,89 @@ TMPBASE="$(mktemp -d)"
20
23
  trap 'rm -rf "$TMPBASE"' EXIT
21
24
 
22
25
  export CREW_REPO="$TMPBASE/repo"
26
+ export CREW_HOME="$TMPBASE/home"
27
+ export MERGE_LOCK_LEASE_SECONDS=600
23
28
  mkdir -p "$CREW_REPO/.worktrees"
24
29
 
25
- # --- Case 1: acquire on unlocked succeeds ---
26
- # Use $$ (this shell's PID) which is definitely alive.
27
- out=$("$LOCK" acquire task-1 $$ 2>&1); code=$?
28
- [ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-1"* ]] && ok "1: acquire on unlocked succeeds" || no "1: exit $code, out: $out"
30
+ # --- Case 1: acquire on unlocked succeeds (opaque holder, not a PID) ---
31
+ out=$("$LOCK" acquire task-1 "task-1/run-aaa" 2>&1); code=$?
32
+ [ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-1"* ]] && [[ "$out" == *"task-1/run-aaa"* ]] \
33
+ && ok "1: acquire on unlocked succeeds with opaque holder" || no "1: exit $code, out: $out"
29
34
 
30
- # --- Case 2: second acquire with live PID reports HELD ---
31
- out=$("$LOCK" acquire task-2 99999 2>&1); code=$?
32
- [ "$code" -eq 1 ] && [[ "$out" == *"HELD by task-1"* ]] && ok "2: acquire with live holder PID reports HELD" || no "2: exit $code, out: $out"
35
+ # --- Case 2: contender blocked while lease unexpired (no PID involved) ---
36
+ out=$("$LOCK" acquire task-2 "task-2/run-bbb" 2>&1); code=$?
37
+ [ "$code" -eq 1 ] && [[ "$out" == *"HELD by task-1"* ]] \
38
+ && ok "2: unexpired lease blocks contender" || no "2: exit $code, out: $out"
33
39
 
34
- # --- Case 3: release by owner succeeds ---
35
- out=$("$LOCK" release task-1 2>&1); code=$?
36
- [ "$code" -eq 0 ] && [[ "$out" == *"RELEASED by task-1"* ]] && ok "3: release by owner succeeds" || no "3: exit $code, out: $out"
37
-
38
- # --- Case 4: status reports UNLOCKED after release ---
40
+ # --- Case 3: status emits parseable holder/age/remaining fields ---
39
41
  out=$("$LOCK" status 2>&1); code=$?
40
- [ "$code" -eq 0 ] && [[ "$out" == "UNLOCKED" ]] && ok "4: status UNLOCKED after release" || no "4: exit $code, out: $out"
42
+ [ "$code" -eq 0 ] \
43
+ && [[ "$out" == *"locked=true"* ]] \
44
+ && [[ "$out" == *"task_id=task-1"* ]] \
45
+ && [[ "$out" == *"holder=task-1/run-aaa"* ]] \
46
+ && [[ "$out" == *"age_seconds="* ]] \
47
+ && [[ "$out" == *"remaining_seconds="* ]] \
48
+ && ok "3: status has holder/age/remaining fields" || no "3: exit $code, out: $out"
49
+
50
+ # --- Case 4: refresh extends the lease (holder-only) ---
51
+ # Sleep long enough that integer-second lease math must show decay; a
52
+ # refresh then resets the clock and the remaining lease grows again.
53
+ sleep 3
54
+ just_before=$("$LOCK" status | sed -n 's/^remaining_seconds=//p')
55
+ "$LOCK" refresh task-1 >/dev/null 2>&1; code=$?
56
+ just_after=$("$LOCK" status | sed -n 's/^remaining_seconds=//p')
57
+ [ "$code" -eq 0 ] && [ "$just_after" -gt "$just_before" ] \
58
+ && ok "4: refresh extends the lease" || no "4: code $code, before $just_before, after $just_after"
59
+
60
+ # --- Case 5: refresh by non-holder fails ---
61
+ out=$("$LOCK" refresh task-2 2>&1); code=$?
62
+ [ "$code" -eq 1 ] && [[ "$out" == *"held by task-1"* ]] \
63
+ && ok "5: refresh by non-holder fails" || no "5: exit $code, out: $out"
41
64
 
42
- # --- Case 5: stale lock (dead PID) is reclaimed on acquire ---
43
- # Write a lock file with a PID that is certainly dead (99999999).
44
- echo "stale-task 2026-09-10T06:21:15Z 99999999" > "$CREW_REPO/.worktrees/.merge-lock"
45
- out=$("$LOCK" acquire task-3 22222 2>&1); code=$?
46
- [ "$code" -eq 0 ] && [[ "$out" == *"reclaimed stale lock from stale-task"* ]] && ok "5: stale lock (dead PID) reclaimed on acquire" || no "5: exit $code, out: $out"
65
+ # --- Case 6: release by non-holder fails ---
66
+ out=$("$LOCK" release task-2 2>&1); code=$?
67
+ [ "$code" -eq 1 ] && [[ "$out" == *"held by task-1"* ]] \
68
+ && ok "6: release by non-holder fails" || no "6: exit $code, out: $out"
47
69
 
48
- # --- Case 6: after reclaim, the new holder owns the lock ---
70
+ # --- Case 7: release by holder succeeds and removes the lock ---
71
+ out=$("$LOCK" release task-1 2>&1); code=$?
72
+ [ "$code" -eq 0 ] && [[ "$out" == *"RELEASED by task-1"* ]] && [ ! -f "$CREW_REPO/.worktrees/.merge-lock" ] \
73
+ && ok "7: release by holder removes lock file" || no "7: exit $code, out: $out"
74
+
75
+ # --- Case 8: status reports UNLOCKED after release ---
49
76
  out=$("$LOCK" status 2>&1); code=$?
50
- [ "$code" -eq 0 ] && [[ "$out" == *"LOCKED by task-3"* ]] && ok "6: reclaimed lock owned by new task" || no "6: exit $code, out: $out"
77
+ [ "$code" -eq 0 ] && [ "$out" = "UNLOCKED" ] \
78
+ && ok "8: status UNLOCKED after release" || no "8: exit $code, out: $out"
79
+
80
+ # --- Case 9: expired lease is reclaimed (logged) by a contender ---
81
+ # Write a lock file whose lease expired long ago (epoch 2020, lease 600s).
82
+ printf 'task_id=old-task\nholder=old-task/run-old\nacquired_at=1577836800\nlease_seconds=600\n' \
83
+ > "$CREW_REPO/.worktrees/.merge-lock"
84
+ out=$("$LOCK" acquire task-3 "task-3/run-ccc" 2>&1); code=$?
85
+ [ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-3"* ]] && [[ "$out" == *"reclaimed expired lease from old-task"* ]] \
86
+ && ok "9: expired lease reclaimed by contender" || no "9: exit $code, out: $out"
87
+ [ -f "$TMPBASE/home/.merge-lock.log" ] && grep -q "op=acquire.*result=STALE-BREAK.*previous_task=old-task" "$TMPBASE/home/.merge-lock.log" \
88
+ && ok "9b: reclaim logged to the audit log" || no "9b: no STALE-BREAK line in audit log"
51
89
 
52
- # --- Case 7: release by non-owner fails ---
53
- out=$("$LOCK" release wrong-task 2>&1); code=$?
54
- [ "$code" -eq 1 ] && [[ "$out" == *"ERROR: lock held by task-3"* ]] && ok "7: release by non-owner fails" || no "7: exit $code, out: $out"
90
+ # --- Case 10: malformed lock file is treated as expired and reclaimed ---
91
+ printf 'garbage without any key=value\n' > "$CREW_REPO/.worktrees/.merge-lock"
92
+ out=$("$LOCK" acquire task-4 "task-4/run-ddd" 2>&1); code=$?
93
+ [ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-4"* ]] \
94
+ && ok "10: malformed lock reclaimed" || no "10: exit $code, out: $out"
55
95
 
56
- # --- Case 8: release by owner cleans up ---
57
- out=$("$LOCK" release task-3 2>&1); code=$?
58
- [ "$code" -eq 0 ] && [ ! -f "$CREW_REPO/.worktrees/.merge-lock" ] && ok "8: release removes lock file" || no "8: exit $code, out: $out"
96
+ # --- Case 11: every op appends to the audit log ---
97
+ lines=$(wc -l < "$TMPBASE/home/.merge-lock.log")
98
+ [ "$lines" -ge 8 ] && ok "11: audit log has $lines lines" || no "11: audit log has only $lines lines"
59
99
 
60
- # --- Case 9: lock with missing PID field is treated as stale ---
61
- # A lock file with "-" as PID (or empty) should be reclaimable.
62
- echo "old-task 2026-09-10T06:21:15Z -" > "$CREW_REPO/.worktrees/.merge-lock"
63
- out=$("$LOCK" acquire task-4 33333 2>&1); code=$?
64
- [ "$code" -eq 0 ] && [[ "$out" == *"reclaimed stale lock from old-task"* ]] && ok "9: lock with '-' PID reclaimed as stale" || no "9: exit $code, out: $out"
100
+ # --- Case 12: concurrent acquirers serialize exactly one wins ---
65
101
  "$LOCK" release task-4 >/dev/null 2>&1
102
+ for n in $(seq 1 20); do
103
+ ( "$LOCK" acquire "racer-$n" "racer-$n/run" >/dev/null 2>&1 ) &
104
+ done
105
+ wait
106
+ holder_count=$("$LOCK" status | grep -c '^task_id=racer-' || true)
107
+ [ "$holder_count" -eq 1 ] && ok "12: exactly one concurrent acquirer holds the lock" || no "12: holder_count=$holder_count"
108
+ grep -q "result=ACQUIRED" "$TMPBASE/home/.merge-lock.log" && ok "12b: at least one ACQUIRED logged" || no "12b: no ACQUIRED in audit log"
66
109
 
67
110
  echo ""
68
111
  echo "merge-lock: $pass passed, $fail failed"
@@ -22,6 +22,9 @@ no() { echo "FAIL: $1"; fail=$((fail + 1)); }
22
22
  TMPBASE="$(mktemp -d)"
23
23
  trap 'rm -rf "$TMPBASE"' EXIT
24
24
 
25
+ # Keep merge-lock.sh's audit log out of the real $CREW_HOME during tests.
26
+ export CREW_HOME="$TMPBASE/home"
27
+
25
28
  # new_fixture <name> — fresh git repo, one commit on main; exports CREW_REPO.
26
29
  new_fixture() {
27
30
  local dir="$TMPBASE/$1"
@@ -90,21 +93,32 @@ echo "$out" | grep -q "→ FAILED" && ok "4: FAILED line printed (no false 'remo
90
93
  [ -d .worktrees/T4 ] && ok "4: failed worktree left in place" || no "4: worktree gone despite failed removal"
91
94
  git worktree unlock .worktrees/T4 2>/dev/null || true
92
95
 
93
- # --- Case 5: lock path ---
96
+ # --- Case 5: lock path (lease-based; bug 2fc8f52f) ---
94
97
  new_fixture case5
95
98
  mkdir -p .worktrees
96
- printf 'T5 2020-01-01T00:00:00Z 999999\n' > .worktrees/.merge-lock
99
+ # 5a: expired lease, no running session → released (logged to the audit log).
100
+ printf 'task_id=T5\nholder=T5/run-old\nacquired_at=1577836800\nlease_seconds=600\n' > .worktrees/.merge-lock
97
101
  out=$(CREW_ACTIVE_TASKS="" "$SWEEP" clean 2>&1); code=$?
98
- [ ! -f .worktrees/.merge-lock ] && ok "5a: stale lock released" || no "5a: stale lock file still present"
99
- echo "$out" | grep -q "released (dead PID)" && ok "5a: release reported" || no "5a: no release line in output"
102
+ [ ! -f .worktrees/.merge-lock ] && ok "5a: expired-lease lock released" || no "5a: lock file still present"
103
+ echo "$out" | grep -q "STALE_LOCK" && ok "5a: STALE_LOCK line printed" || no "5a: no STALE_LOCK line in output"
104
+ echo "$out" | grep -q "released (lease expired)" && ok "5a: release reported" || no "5a: no release line in output"
100
105
  [ "$code" -eq 0 ] && ok "5a: exit 0" || no "5a: exit $code, want 0"
101
106
 
102
- printf '%s 2020-01-01T00:00:00Z 999999\n' "$UUID_LOCK" > .worktrees/.merge-lock
107
+ # 5b: valid lease held by an active run → untouched regardless of lease age.
108
+ now=$(date +%s)
109
+ printf 'task_id=%s\nholder=%s/run-live\nacquired_at=%s\nlease_seconds=600\n' "$UUID_LOCK" "$UUID_LOCK" "$now" > .worktrees/.merge-lock
103
110
  out=$(CREW_ACTIVE_TASKS="$UUID_LOCK" "$SWEEP" clean 2>&1); code=$?
104
111
  [ -f .worktrees/.merge-lock ] && ok "5b: active holder's lock not released" || no "5b: active holder's lock was released"
105
112
  echo "$out" | grep -q "ACTIVE_LOCK" && ok "5b: ACTIVE_LOCK line printed" || no "5b: no ACTIVE_LOCK line in output"
106
113
  [ "$code" -eq 0 ] && ok "5b: exit 0" || no "5b: exit $code, want 0"
107
114
 
115
+ # 5c: valid lease, no running session → held, NOT released (lease not expired).
116
+ printf 'task_id=T6\nholder=T6/run-live\nacquired_at=%s\nlease_seconds=600\n' "$now" > .worktrees/.merge-lock
117
+ out=$(CREW_ACTIVE_TASKS="" "$SWEEP" clean 2>&1); code=$?
118
+ [ -f .worktrees/.merge-lock ] && ok "5c: unexpired-lease lock not released" || no "5c: unexpired-lease lock was released"
119
+ echo "$out" | grep -q "HELD_LOCK" && ok "5c: HELD_LOCK line printed" || no "5c: no HELD_LOCK line in output"
120
+ [ "$code" -eq 0 ] && ok "5c: exit 0" || no "5c: exit $code, want 0"
121
+
108
122
  # --- Case 6: report mode without active data ---
109
123
  new_fixture case6
110
124
  merged_worktree T7