muse-crew 0.4.4 → 0.4.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/API.md +2 -2
- package/lib/AGENTS.md +3 -2
- package/lib/merge-lock.sh +149 -68
- package/lib/orphan-sweep.sh +38 -34
- package/lib/publish-npm.sh +28 -3
- package/lib/test-merge-lock.sh +77 -34
- package/lib/test-orphan-sweep.sh +19 -5
- package/lib/test-publish-verify.sh +157 -0
- package/lib/test-version-write.sh +9 -4
- package/lib/worktree-lifecycle.sh +33 -7
- package/package.json +1 -1
- package/workflows/bugfix.js +117 -16
- package/workflows/chore.js +117 -16
- package/workflows/standard.js +114 -23
package/API.md
CHANGED
|
@@ -71,14 +71,14 @@ Send a stuck or failed task to a specific workflow phase for recovery.
|
|
|
71
71
|
|
|
72
72
|
### `parktask`
|
|
73
73
|
|
|
74
|
-
Atomically park a task for human attention: sets `state` to `parked
|
|
74
|
+
Atomically park a task for human attention: sets `state` to `parked`, writes the explanatory `note` event, and settles every `running` session for the task — in one transaction. Either all land or none do: callers never see a note without a park, a park without a note, or a parked task with a ghost `running` session left behind (a terminal task transition settles its sessions). Settled sessions move to `failed` — the status `recovertask` accepts, so recovery works immediately after a park — with `ended_at` stamped and `Parked: <message>` appended to their notes; `failure_reason` stays null.
|
|
75
75
|
|
|
76
76
|
| Field | Type | Required | Notes |
|
|
77
77
|
|-------|------|----------|-------|
|
|
78
78
|
| `task_id` | uuid | yes | |
|
|
79
79
|
| `message` | string (1–1000) | yes | Why the task needs a human; written to the activity feed as a `note` |
|
|
80
80
|
|
|
81
|
-
Returns `{ "ok": true, "task": {...} }` with the updated task record.
|
|
81
|
+
Returns `{ "ok": true, "task": {...}, "settled_sessions": <n> }` with the updated task record and the count of sessions settled (`0` when none were running).
|
|
82
82
|
|
|
83
83
|
---
|
|
84
84
|
|
package/lib/AGENTS.md
CHANGED
|
@@ -4,11 +4,12 @@ Shell scripts for the crew's infrastructure. Called by workflow scripts, cron, a
|
|
|
4
4
|
|
|
5
5
|
- `build-registry.js` — deterministic extractor that generates `workflows/registry.json` (workflow step registry) from the workflow files' `meta` blocks at release time; invoked by `crew-release.sh` deploy
|
|
6
6
|
- `crew-release.sh` — immutable release manager: deploy, rollback, prune
|
|
7
|
-
- `merge-lock.sh` — serialized merge lock for concurrent agents;
|
|
7
|
+
- `merge-lock.sh` — serialized merge lock for concurrent agents: time-based holder lease (bug 2fc8f52f — an unexpired lease is held regardless of process liveness; only an expired lease may be broken). Lock file is key=value: task_id, opaque holder identity (never a PID), acquired_at epoch, lease_seconds (default 600, override via MERGE_LOCK_LEASE_SECONDS). acquire/refresh/release/status/force-release; holder-only refresh and release; every op appends to $CREW_HOME/.merge-lock.log
|
|
8
8
|
- `worktree-lifecycle.sh` — the worktree lifecycle seam: prepare/cleanup/inspect/integrate/status/post-deploy over git worktrees (`.worktrees/<id>`, branch `task/<id>`). The crew registry (`<repo>/.worktrees/.registry/<id>`) is the source of truth for task→branch/path — never reconstruct it from git state. Prepare fails closed on dirty `main`; cleanup is forgiving.
|
|
9
9
|
- `test-worktree-backend.sh` — regression tests for the lifecycle script (validate, prepare/reuse, inspect, status, cleanup, idempotent cleanup, dirty-main preflight) on a scratch repo
|
|
10
10
|
- `test-version-write.sh` — regression tests for the escape-preserving step-8 version write in publish-npm.sh (fixture: current package.json with the \u2014 escape; extracts the shipped block by anchor)
|
|
11
|
-
- `
|
|
11
|
+
- `test-publish-verify.sh` — regression tests for the retry-tolerant step-12 verification in publish-npm.sh (canary 5a027278): extracts the shipped block by anchor and runs it against a fake npm whose read replica lags (non-zero exits, then the old version, then the target) — requires convergence on success, fail-closed `PUBLISH_FAILED=verify` on exhaustion, and `--prefer-online` on every read
|
|
12
|
+
- `orphan-sweep.sh` — find and clean stale worktrees and merge locks: parses `merge-lock.sh status` key=value fields (never a PID); a lock whose task has a running session is untouched regardless of lease age; a lock with no running session is broken only when its lease is expired (via merge-lock.sh release so the break lands in the audit log)
|
|
12
13
|
- `publish-npm.sh` — deterministic npm publish: lock refresh, release install, version write/commit, pack, registry publish, verify, push, post-deploy. Takes TARGET_VERSION as input; idempotent on retry/resume.
|
|
13
14
|
- `compose-evidence.py` — deterministic visual-evidence compositor (Pillow): pairs identical PNG stems from baseline/ and postchange/ dirs, emits `<stem>-sidebyside.png` and amplified-difference `<stem>-overlay.png` into composites/, prints `PAIR`/`SKIP` manifests. Byte-deterministic; nonzero exit on errors.
|
|
14
15
|
- `test-orphan-sweep.sh` — regression tests for orphan-sweep.sh (active-run guard, verified removal, fail-closed)
|
package/lib/merge-lock.sh
CHANGED
|
@@ -1,16 +1,89 @@
|
|
|
1
1
|
#!/usr/bin/env bash
|
|
2
|
-
#
|
|
2
|
+
# merge-lock.sh — serialized merge lock for the crew's repos.
|
|
3
|
+
#
|
|
4
|
+
# Time-based holder lease (bug 2fc8f52f): the old PID-liveness staleness
|
|
5
|
+
# check never excluded anyone. The recorded PID was the short-lived shell
|
|
6
|
+
# that ran the acquire, so for essentially the whole Integrate→Publish→
|
|
7
|
+
# post-deploy hold window the holder's PID was dead, every concurrent
|
|
8
|
+
# acquire took the STALE path, and two Publish phases both believed they
|
|
9
|
+
# owned version assignment. PID is the wrong liveness signal (orphan-sweep
|
|
10
|
+
# already treats it as such) — the lease is time-based and refreshed by
|
|
11
|
+
# the workflow while the hold is live.
|
|
12
|
+
#
|
|
13
|
+
# The lock file ($CREW_REPO/.worktrees/.merge-lock) is key=value:
|
|
14
|
+
# task_id=<task id that holds the lock>
|
|
15
|
+
# holder=<opaque task+run identity, never a PID>
|
|
16
|
+
# acquired_at=<epoch seconds; refresh rewrites it to extend the lease>
|
|
17
|
+
# lease_seconds=<lease window; MERGE_LOCK_LEASE_SECONDS, default 600>
|
|
18
|
+
#
|
|
19
|
+
# The lock is HELD while now - acquired_at < lease_seconds. An expired
|
|
20
|
+
# (or malformed) lease is broken with logging and re-acquired. Every op
|
|
21
|
+
# appends one line to $CREW_HOME/.merge-lock.log (the audit trail).
|
|
22
|
+
#
|
|
3
23
|
# Usage:
|
|
4
|
-
# merge-lock.sh acquire <task_id>
|
|
5
|
-
# merge-lock.sh
|
|
6
|
-
# merge-lock.sh
|
|
7
|
-
# merge-lock.sh status
|
|
8
|
-
# merge-lock.sh force-release
|
|
24
|
+
# merge-lock.sh acquire <task_id> <holder> — acquire the lock (0 ok, 1 held)
|
|
25
|
+
# merge-lock.sh refresh <task_id> — extend the lease (holder task only)
|
|
26
|
+
# merge-lock.sh release <task_id> — release the lock (holder task only)
|
|
27
|
+
# merge-lock.sh status — key=value holder/age/remaining, or UNLOCKED
|
|
28
|
+
# merge-lock.sh force-release — break the lock (admin use only)
|
|
9
29
|
|
|
10
30
|
set -euo pipefail
|
|
11
31
|
|
|
12
32
|
LOCK_DIR="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}/.worktrees"
|
|
13
33
|
LOCK_FILE="$LOCK_DIR/.merge-lock"
|
|
34
|
+
LOG_FILE="${CREW_HOME:-$HOME/workspace/.jarvis}/.merge-lock.log"
|
|
35
|
+
LEASE_SECONDS="${MERGE_LOCK_LEASE_SECONDS:-600}"
|
|
36
|
+
|
|
37
|
+
log_op() { # log_op <op> <detail> — audit trail; logging never breaks a lock op
|
|
38
|
+
mkdir -p "$(dirname "$LOG_FILE")" 2>/dev/null || true
|
|
39
|
+
printf '%s op=%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" "$2" >> "$LOG_FILE" 2>/dev/null || true
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
# read_lock — parse the lock file into lock_task_id / lock_holder /
|
|
43
|
+
# lock_acquired_at / lock_lease_seconds. Returns 1 when there is no lock.
|
|
44
|
+
read_lock() {
|
|
45
|
+
lock_task_id=""; lock_holder=""; lock_acquired_at=""; lock_lease_seconds=""
|
|
46
|
+
[ -f "$LOCK_FILE" ] || return 1
|
|
47
|
+
local key val
|
|
48
|
+
while IFS='=' read -r key val; do
|
|
49
|
+
case "$key" in
|
|
50
|
+
task_id) lock_task_id="$val" ;;
|
|
51
|
+
holder) lock_holder="$val" ;;
|
|
52
|
+
acquired_at) lock_acquired_at="$val" ;;
|
|
53
|
+
lease_seconds) lock_lease_seconds="$val" ;;
|
|
54
|
+
esac
|
|
55
|
+
done < "$LOCK_FILE"
|
|
56
|
+
return 0
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
lock_age() { # seconds since acquired_at, or "unknown" when unparsable
|
|
60
|
+
if [[ "${lock_acquired_at:-}" =~ ^[0-9]+$ ]]; then
|
|
61
|
+
echo $(( $(date +%s) - lock_acquired_at ))
|
|
62
|
+
else
|
|
63
|
+
echo "unknown"
|
|
64
|
+
fi
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
lock_remaining() { # lease seconds remaining (<=0 means expired), or "unknown"
|
|
68
|
+
local age="${1:-$(lock_age)}" lease="$lock_lease_seconds"
|
|
69
|
+
[[ "$lease" =~ ^[0-9]+$ ]] || lease="$LEASE_SECONDS"
|
|
70
|
+
if [ "$age" = "unknown" ]; then
|
|
71
|
+
echo "unknown"
|
|
72
|
+
else
|
|
73
|
+
echo $(( lease - age ))
|
|
74
|
+
fi
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
# write_lock <task_id> <holder> — atomic create via noclobber; 0 on success.
|
|
78
|
+
# A concurrent writer wins exactly one create; the loser retries the read.
|
|
79
|
+
write_lock() {
|
|
80
|
+
local task_id="$1" holder="$2"
|
|
81
|
+
mkdir -p "$LOCK_DIR"
|
|
82
|
+
( set -C
|
|
83
|
+
printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
|
|
84
|
+
"$task_id" "$holder" "$(date +%s)" "$LEASE_SECONDS" > "$LOCK_FILE"
|
|
85
|
+
) 2>/dev/null
|
|
86
|
+
}
|
|
14
87
|
|
|
15
88
|
cmd="${1:-}"
|
|
16
89
|
task_id="${2:-}"
|
|
@@ -18,90 +91,98 @@ task_id="${2:-}"
|
|
|
18
91
|
case "$cmd" in
|
|
19
92
|
acquire)
|
|
20
93
|
[ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
echo "ACQUIRED by $task_id"
|
|
94
|
+
holder="${3:-$task_id}"
|
|
95
|
+
if write_lock "$task_id" "$holder"; then
|
|
96
|
+
log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
|
|
97
|
+
echo "ACQUIRED by $task_id (holder $holder)"
|
|
25
98
|
exit 0
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
# cleanup mechanical: a stopped/failed workflow's lock is reclaimed on
|
|
35
|
-
# the next acquire, not via manual release or a background sweeper.
|
|
36
|
-
if [ -z "$lock_pid" ] || [ "$lock_pid" = "-" ] || ! kill -0 "$lock_pid" 2>/dev/null; then
|
|
37
|
-
echo "STALE: lock held by $holder since $acquired_at (pid $lock_pid dead) — breaking" >&2
|
|
38
|
-
rm -f "$LOCK_FILE"
|
|
39
|
-
if (set -C; echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $owner_pid" > "$LOCK_FILE") 2>/dev/null; then
|
|
40
|
-
echo "ACQUIRED by $task_id (reclaimed stale lock from $holder)"
|
|
41
|
-
exit 0
|
|
42
|
-
fi
|
|
43
|
-
# Lost the race: another task acquired it after we broke the stale lock.
|
|
44
|
-
holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
|
|
45
|
-
acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
|
|
46
|
-
lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
|
|
99
|
+
fi
|
|
100
|
+
# A lock file exists (or raced into existence) — read it.
|
|
101
|
+
if ! read_lock; then
|
|
102
|
+
# Raced with a release between create and read: retry the create once.
|
|
103
|
+
if write_lock "$task_id" "$holder"; then
|
|
104
|
+
log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED"
|
|
105
|
+
echo "ACQUIRED by $task_id (holder $holder)"
|
|
106
|
+
exit 0
|
|
47
107
|
fi
|
|
48
|
-
echo "
|
|
108
|
+
read_lock || { echo "ERROR: lock state unreadable"; log_op acquire "task_id=$task_id holder=$holder result=ERROR"; exit 1; }
|
|
109
|
+
fi
|
|
110
|
+
remaining="$(lock_remaining)"
|
|
111
|
+
if [ "$remaining" != "unknown" ] && [ "$remaining" -gt 0 ]; then
|
|
112
|
+
log_op acquire "task_id=$task_id holder=$holder result=HELD holder_task=$lock_task_id holder_id=$lock_holder remaining=${remaining}s"
|
|
113
|
+
echo "HELD by $lock_task_id (holder $lock_holder, ${remaining}s of lease remaining)"
|
|
49
114
|
exit 1
|
|
50
115
|
fi
|
|
116
|
+
# Expired or malformed lease — break it (logged) and re-acquire.
|
|
117
|
+
echo "STALE: lease for ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) expired — breaking" >&2
|
|
118
|
+
log_op acquire "task_id=$task_id holder=$holder result=STALE-BREAK previous_task=${lock_task_id:-unknown} previous_holder=${lock_holder:-unknown}"
|
|
119
|
+
rm -f "$LOCK_FILE"
|
|
120
|
+
if write_lock "$task_id" "$holder"; then
|
|
121
|
+
log_op acquire "task_id=$task_id holder=$holder result=ACQUIRED-RECLAIMED previous_task=${lock_task_id:-unknown}"
|
|
122
|
+
echo "ACQUIRED by $task_id (holder $holder; reclaimed expired lease from ${lock_task_id:-unknown})"
|
|
123
|
+
exit 0
|
|
124
|
+
fi
|
|
125
|
+
# Lost the race: whoever won the create holds it now.
|
|
126
|
+
read_lock 2>/dev/null || true
|
|
127
|
+
log_op acquire "task_id=$task_id holder=$holder result=HELD-RACE holder_task=${lock_task_id:-unknown}"
|
|
128
|
+
echo "HELD by ${lock_task_id:-unknown} (holder ${lock_holder:-unknown}) — lost reclaim race"
|
|
129
|
+
exit 1
|
|
130
|
+
;;
|
|
131
|
+
refresh)
|
|
132
|
+
[ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
|
|
133
|
+
if ! read_lock; then
|
|
134
|
+
log_op refresh "task_id=$task_id result=NO-LOCK"
|
|
135
|
+
echo "ERROR: no lock to refresh"
|
|
136
|
+
exit 1
|
|
137
|
+
fi
|
|
138
|
+
if [ "$lock_task_id" != "$task_id" ]; then
|
|
139
|
+
log_op refresh "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
|
|
140
|
+
echo "ERROR: lock held by $lock_task_id, not $task_id"
|
|
141
|
+
exit 1
|
|
142
|
+
fi
|
|
143
|
+
# Extend the lease: acquired_at=now, holder and lease window unchanged.
|
|
144
|
+
printf 'task_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\n' \
|
|
145
|
+
"$lock_task_id" "$lock_holder" "$(date +%s)" "${lock_lease_seconds:-$LEASE_SECONDS}" > "$LOCK_FILE"
|
|
146
|
+
log_op refresh "task_id=$task_id holder=$lock_holder result=REFRESHED"
|
|
147
|
+
echo "REFRESHED by $task_id (holder $lock_holder)"
|
|
148
|
+
exit 0
|
|
51
149
|
;;
|
|
52
150
|
release)
|
|
53
151
|
[ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
|
|
54
|
-
if
|
|
55
|
-
|
|
56
|
-
if [ "$holder" = "$task_id" ]; then
|
|
57
|
-
rm -f "$LOCK_FILE"
|
|
58
|
-
echo "RELEASED by $task_id"
|
|
59
|
-
exit 0
|
|
60
|
-
else
|
|
61
|
-
echo "ERROR: lock held by $holder, not $task_id"
|
|
62
|
-
exit 1
|
|
63
|
-
fi
|
|
64
|
-
else
|
|
152
|
+
if ! read_lock; then
|
|
153
|
+
log_op release "task_id=$task_id result=NOT-LOCKED"
|
|
65
154
|
echo "RELEASED (was not locked)"
|
|
66
155
|
exit 0
|
|
67
156
|
fi
|
|
157
|
+
if [ "$lock_task_id" = "$task_id" ]; then
|
|
158
|
+
rm -f "$LOCK_FILE"
|
|
159
|
+
log_op release "task_id=$task_id holder=$lock_holder result=RELEASED"
|
|
160
|
+
echo "RELEASED by $task_id"
|
|
161
|
+
exit 0
|
|
162
|
+
fi
|
|
163
|
+
log_op release "task_id=$task_id result=NOT-HOLDER holder_task=$lock_task_id"
|
|
164
|
+
echo "ERROR: lock held by $lock_task_id, not $task_id"
|
|
165
|
+
exit 1
|
|
68
166
|
;;
|
|
69
167
|
status)
|
|
70
|
-
if
|
|
71
|
-
holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "unknown")
|
|
72
|
-
acquired_at=$(cut -d' ' -f2 "$LOCK_FILE" 2>/dev/null || echo "unknown")
|
|
73
|
-
lock_pid=$(awk '{print $3}' "$LOCK_FILE" 2>/dev/null || echo "-")
|
|
74
|
-
echo "LOCKED by $holder since $acquired_at (pid $lock_pid)"
|
|
75
|
-
else
|
|
168
|
+
if ! read_lock; then
|
|
76
169
|
echo "UNLOCKED"
|
|
170
|
+
exit 0
|
|
77
171
|
fi
|
|
172
|
+
printf 'locked=true\ntask_id=%s\nholder=%s\nacquired_at=%s\nlease_seconds=%s\nage_seconds=%s\nremaining_seconds=%s\n' \
|
|
173
|
+
"$lock_task_id" "$lock_holder" "$lock_acquired_at" "${lock_lease_seconds:-$LEASE_SECONDS}" \
|
|
174
|
+
"$(lock_age)" "$(lock_remaining)"
|
|
78
175
|
exit 0
|
|
79
176
|
;;
|
|
80
177
|
force-release)
|
|
178
|
+
read_lock 2>/dev/null || true
|
|
179
|
+
log_op force-release "result=FORCE-RELEASED previous_task=${lock_task_id:-unknown}"
|
|
81
180
|
rm -f "$LOCK_FILE"
|
|
82
181
|
echo "FORCE RELEASED"
|
|
83
182
|
exit 0
|
|
84
183
|
;;
|
|
85
|
-
refresh)
|
|
86
|
-
[ -z "$task_id" ] && { echo "ERROR: task_id required"; exit 1; }
|
|
87
|
-
new_pid="${3:-$PPID}"
|
|
88
|
-
if [ -f "$LOCK_FILE" ]; then
|
|
89
|
-
holder=$(cut -d' ' -f1 "$LOCK_FILE" 2>/dev/null || echo "")
|
|
90
|
-
if [ "$holder" = "$task_id" ]; then
|
|
91
|
-
echo "$task_id $(date -u +%Y-%m-%dT%H:%M:%SZ) $new_pid" > "$LOCK_FILE"
|
|
92
|
-
echo "REFRESHED by $task_id"
|
|
93
|
-
exit 0
|
|
94
|
-
else
|
|
95
|
-
echo "ERROR: lock held by $holder, not $task_id"
|
|
96
|
-
exit 1
|
|
97
|
-
fi
|
|
98
|
-
else
|
|
99
|
-
echo "ERROR: no lock to refresh"
|
|
100
|
-
exit 1
|
|
101
|
-
fi
|
|
102
|
-
;;
|
|
103
184
|
*)
|
|
104
|
-
echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release}
|
|
185
|
+
echo "Usage: merge-lock.sh {acquire|release|refresh|status|force-release} <task_id> [holder]"
|
|
105
186
|
exit 1
|
|
106
187
|
;;
|
|
107
188
|
esac
|
package/lib/orphan-sweep.sh
CHANGED
|
@@ -4,9 +4,9 @@
|
|
|
4
4
|
#
|
|
5
5
|
# Usage:
|
|
6
6
|
# orphan-sweep.sh report — list orphans (read-only for worktrees);
|
|
7
|
-
#
|
|
7
|
+
# expired-lease locks are released
|
|
8
8
|
# orphan-sweep.sh clean — remove safe-to-clean orphans (merged branches only),
|
|
9
|
-
# release
|
|
9
|
+
# release expired-lease merge locks, and remove pins whose
|
|
10
10
|
# task has no running session
|
|
11
11
|
#
|
|
12
12
|
# Active-run knowledge is injected by the caller, never fetched here:
|
|
@@ -44,7 +44,7 @@ set -euo pipefail
|
|
|
44
44
|
REPO="${CREW_REPO:-$HOME/workspace/ts-spaces/orchestra-dashboard}"
|
|
45
45
|
WORKTREE_DIR="$REPO/.worktrees"
|
|
46
46
|
LOCK_FILE="$WORKTREE_DIR/.merge-lock"
|
|
47
|
-
|
|
47
|
+
MERGE_LOCK_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/merge-lock.sh"
|
|
48
48
|
|
|
49
49
|
cmd="${1:-report}"
|
|
50
50
|
found=0
|
|
@@ -100,40 +100,44 @@ if [ "$cmd" != "clean" ] && [ "$have_active_data" -eq 0 ]; then
|
|
|
100
100
|
echo "NOTE: no active-run data provided; ACTIVE checks skipped"
|
|
101
101
|
fi
|
|
102
102
|
|
|
103
|
-
# --- Stale merge lock ---
|
|
103
|
+
# --- Stale merge lock (lease-based; bug 2fc8f52f) ---
|
|
104
|
+
# Owner identity comes from merge-lock.sh status key=value fields — never
|
|
105
|
+
# from a PID. Fail-closed: a lock whose task has a running session is
|
|
106
|
+
# untouched regardless of lease age; a lock whose task has no running
|
|
107
|
+
# session is broken only when its lease is expired. The break goes through
|
|
108
|
+
# merge-lock.sh release (or force-release for an unparseable file with no
|
|
109
|
+
# identifiable holder) so it lands in the merge-lock audit log.
|
|
104
110
|
if [ -f "$LOCK_FILE" ]; then
|
|
105
|
-
|
|
106
|
-
if
|
|
107
|
-
|
|
108
|
-
found=1
|
|
111
|
+
lock_status="$("$MERGE_LOCK_SCRIPT" status 2>/dev/null || echo UNLOCKED)"
|
|
112
|
+
if [ "$lock_status" = "UNLOCKED" ]; then
|
|
113
|
+
: # lock file vanished between the check and the read — nothing held
|
|
109
114
|
else
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
echo "STALE_LOCK: held ${age_min}m by $lock_holder (pid ${lock_pid:-none} dead) — threshold ${STALE_LOCK_MIN}m"
|
|
127
|
-
found=1
|
|
128
|
-
rm -f "$LOCK_FILE"
|
|
129
|
-
if [ ! -f "$LOCK_FILE" ]; then
|
|
130
|
-
echo " → released (dead PID)"
|
|
131
|
-
else
|
|
132
|
-
echo " → FAILED: lock file still present after rm"
|
|
133
|
-
any_failed=1
|
|
134
|
-
fi
|
|
135
|
-
fi
|
|
115
|
+
lock_holder=$(printf '%s\n' "$lock_status" | sed -n 's/^task_id=//p' | head -1)
|
|
116
|
+
lock_remaining=$(printf '%s\n' "$lock_status" | sed -n 's/^remaining_seconds=//p' | head -1)
|
|
117
|
+
lock_age=$(printf '%s\n' "$lock_status" | sed -n 's/^age_seconds=//p' | head -1)
|
|
118
|
+
if task_is_active "$lock_holder"; then
|
|
119
|
+
echo "ACTIVE_LOCK: held by ${lock_holder:-unknown} (active run on dashboard) — skipping"
|
|
120
|
+
found=1
|
|
121
|
+
elif [ "${lock_remaining:-unknown}" != "unknown" ] && [ "$lock_remaining" -gt 0 ]; then
|
|
122
|
+
echo "HELD_LOCK: held by ${lock_holder:-unknown} (lease ${lock_remaining}s remaining) — skipping"
|
|
123
|
+
found=1
|
|
124
|
+
else
|
|
125
|
+
echo "STALE_LOCK: held by ${lock_holder:-unknown} (age ${lock_age:-unknown}s, lease expired, no running session)"
|
|
126
|
+
found=1
|
|
127
|
+
if [ -n "$lock_holder" ]; then
|
|
128
|
+
rel_out=$("$MERGE_LOCK_SCRIPT" release "$lock_holder" 2>&1) || true
|
|
129
|
+
else
|
|
130
|
+
rel_out=$("$MERGE_LOCK_SCRIPT" force-release 2>&1) || true
|
|
136
131
|
fi
|
|
132
|
+
case "$rel_out" in
|
|
133
|
+
*RELEASED*)
|
|
134
|
+
echo " → released (lease expired)"
|
|
135
|
+
;;
|
|
136
|
+
*)
|
|
137
|
+
echo " → FAILED: could not release lock: $rel_out"
|
|
138
|
+
any_failed=1
|
|
139
|
+
;;
|
|
140
|
+
esac
|
|
137
141
|
fi
|
|
138
142
|
fi
|
|
139
143
|
fi
|
package/lib/publish-npm.sh
CHANGED
|
@@ -150,13 +150,38 @@ if [ "$ALREADY_PUBLISHED" = "0" ]; then
|
|
|
150
150
|
rm -f "$TGZ"
|
|
151
151
|
|
|
152
152
|
# 12. Verify: ground truth is the registry, not any agent's summary.
|
|
153
|
-
|
|
154
|
-
|
|
153
|
+
# The registry is eventually consistent: a publish followed by an
|
|
154
|
+
# immediate read can observe the pre-publish version (canary 5a027278
|
|
155
|
+
# hit a stale read replica ~6s after publish). A single stale read must
|
|
156
|
+
# never park a landed publish, so this block retries with backoff and
|
|
157
|
+
# client cache-busting instead of failing on the first mismatch.
|
|
158
|
+
# STEP-12-ANCHOR: publish verification (retry-tolerant)
|
|
159
|
+
VERIFY_ATTEMPTS="${VERIFY_ATTEMPTS:-12}"
|
|
160
|
+
VERIFY_SLEEP_SECS="${VERIFY_SLEEP_SECS:-10}"
|
|
161
|
+
REG=""
|
|
162
|
+
for attempt in $(seq 1 "$VERIFY_ATTEMPTS"); do
|
|
163
|
+
# --prefer-online busts npm's client-side packument cache; the retry
|
|
164
|
+
# loop absorbs server-side replica lag. A non-zero exit is a miss,
|
|
165
|
+
# not an immediate failure.
|
|
166
|
+
REG="$(npm view "$PKG" version --prefer-online 2>/dev/null || true)"
|
|
167
|
+
[ "$REG" = "$TARGET_VERSION" ] && break
|
|
168
|
+
echo "PUBLISH_VERIFY_RETRY=$attempt registry=$REG"
|
|
169
|
+
[ "$attempt" -lt "$VERIFY_ATTEMPTS" ] && sleep "$VERIFY_SLEEP_SECS"
|
|
170
|
+
done
|
|
155
171
|
[ "$REG" = "$TARGET_VERSION" ] \
|
|
156
|
-
|| fail "verify" "registry=$REG"
|
|
172
|
+
|| fail "verify" "registry=$REG after $VERIFY_ATTEMPTS attempts"
|
|
157
173
|
echo "PUBLISH_VERIFIED=$TARGET_VERSION"
|
|
174
|
+
# STEP-12-END
|
|
158
175
|
fi
|
|
159
176
|
|
|
177
|
+
# 12b. Refresh the merge lock again — release-deploy, pack, publish, and the
|
|
178
|
+
# verify retry loop can together consume more than half the lease on a slow
|
|
179
|
+
# registry. The push and the post-deploy below must happen under our lock;
|
|
180
|
+
# if the refresh fails (expired lease, someone else holds it), stop — never
|
|
181
|
+
# push a version-bump commit or release a lock we no longer own.
|
|
182
|
+
LOCK_OUT2="$(CREW_REPO="$REPO_PATH" bash "${LIFECYCLE:?LIFECYCLE is required}" refresh-lock "$TASK_ID" 2>&1)" \
|
|
183
|
+
|| fail "lock-refresh" "$LOCK_OUT2"
|
|
184
|
+
|
|
160
185
|
# 13. Push the version-bump commit (idempotent: no-op if already pushed).
|
|
161
186
|
PUSH_OUT="$(git -C "$REPO_PATH" push origin main 2>&1)" \
|
|
162
187
|
|| fail "push" "$PUSH_OUT"
|
package/lib/test-merge-lock.sh
CHANGED
|
@@ -2,9 +2,12 @@
|
|
|
2
2
|
# test-merge-lock.sh — regression tests for merge-lock.sh
|
|
3
3
|
#
|
|
4
4
|
# Self-contained: uses a temp CREW_REPO and drives lib/merge-lock.sh through
|
|
5
|
-
# acquire/release/status. Tests the
|
|
6
|
-
#
|
|
7
|
-
#
|
|
5
|
+
# acquire/refresh/release/status. Tests the time-based lease model (bug
|
|
6
|
+
# 2fc8f52f): an unexpired lease blocks contenders even when no holder
|
|
7
|
+
# process is alive (PID liveness is never consulted); an expired lease is
|
|
8
|
+
# reclaimed with logging; refresh extends the lease; release is
|
|
9
|
+
# holder-only; status emits parseable holder/age/remaining fields; every
|
|
10
|
+
# op appends to the merge-lock audit log. Exits 0 only if every case passes.
|
|
8
11
|
|
|
9
12
|
set -uo pipefail
|
|
10
13
|
|
|
@@ -20,49 +23,89 @@ TMPBASE="$(mktemp -d)"
|
|
|
20
23
|
trap 'rm -rf "$TMPBASE"' EXIT
|
|
21
24
|
|
|
22
25
|
export CREW_REPO="$TMPBASE/repo"
|
|
26
|
+
export CREW_HOME="$TMPBASE/home"
|
|
27
|
+
export MERGE_LOCK_LEASE_SECONDS=600
|
|
23
28
|
mkdir -p "$CREW_REPO/.worktrees"
|
|
24
29
|
|
|
25
|
-
# --- Case 1: acquire on unlocked succeeds ---
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
30
|
+
# --- Case 1: acquire on unlocked succeeds (opaque holder, not a PID) ---
|
|
31
|
+
out=$("$LOCK" acquire task-1 "task-1/run-aaa" 2>&1); code=$?
|
|
32
|
+
[ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-1"* ]] && [[ "$out" == *"task-1/run-aaa"* ]] \
|
|
33
|
+
&& ok "1: acquire on unlocked succeeds with opaque holder" || no "1: exit $code, out: $out"
|
|
29
34
|
|
|
30
|
-
# --- Case 2:
|
|
31
|
-
out=$("$LOCK" acquire task-2
|
|
32
|
-
[ "$code" -eq 1 ] && [[ "$out" == *"HELD by task-1"* ]]
|
|
35
|
+
# --- Case 2: contender blocked while lease unexpired (no PID involved) ---
|
|
36
|
+
out=$("$LOCK" acquire task-2 "task-2/run-bbb" 2>&1); code=$?
|
|
37
|
+
[ "$code" -eq 1 ] && [[ "$out" == *"HELD by task-1"* ]] \
|
|
38
|
+
&& ok "2: unexpired lease blocks contender" || no "2: exit $code, out: $out"
|
|
33
39
|
|
|
34
|
-
# --- Case 3:
|
|
35
|
-
out=$("$LOCK" release task-1 2>&1); code=$?
|
|
36
|
-
[ "$code" -eq 0 ] && [[ "$out" == *"RELEASED by task-1"* ]] && ok "3: release by owner succeeds" || no "3: exit $code, out: $out"
|
|
37
|
-
|
|
38
|
-
# --- Case 4: status reports UNLOCKED after release ---
|
|
40
|
+
# --- Case 3: status emits parseable holder/age/remaining fields ---
|
|
39
41
|
out=$("$LOCK" status 2>&1); code=$?
|
|
40
|
-
[ "$code" -eq 0 ]
|
|
42
|
+
[ "$code" -eq 0 ] \
|
|
43
|
+
&& [[ "$out" == *"locked=true"* ]] \
|
|
44
|
+
&& [[ "$out" == *"task_id=task-1"* ]] \
|
|
45
|
+
&& [[ "$out" == *"holder=task-1/run-aaa"* ]] \
|
|
46
|
+
&& [[ "$out" == *"age_seconds="* ]] \
|
|
47
|
+
&& [[ "$out" == *"remaining_seconds="* ]] \
|
|
48
|
+
&& ok "3: status has holder/age/remaining fields" || no "3: exit $code, out: $out"
|
|
49
|
+
|
|
50
|
+
# --- Case 4: refresh extends the lease (holder-only) ---
|
|
51
|
+
# Sleep long enough that integer-second lease math must show decay; a
|
|
52
|
+
# refresh then resets the clock and the remaining lease grows again.
|
|
53
|
+
sleep 3
|
|
54
|
+
just_before=$("$LOCK" status | sed -n 's/^remaining_seconds=//p')
|
|
55
|
+
"$LOCK" refresh task-1 >/dev/null 2>&1; code=$?
|
|
56
|
+
just_after=$("$LOCK" status | sed -n 's/^remaining_seconds=//p')
|
|
57
|
+
[ "$code" -eq 0 ] && [ "$just_after" -gt "$just_before" ] \
|
|
58
|
+
&& ok "4: refresh extends the lease" || no "4: code $code, before $just_before, after $just_after"
|
|
59
|
+
|
|
60
|
+
# --- Case 5: refresh by non-holder fails ---
|
|
61
|
+
out=$("$LOCK" refresh task-2 2>&1); code=$?
|
|
62
|
+
[ "$code" -eq 1 ] && [[ "$out" == *"held by task-1"* ]] \
|
|
63
|
+
&& ok "5: refresh by non-holder fails" || no "5: exit $code, out: $out"
|
|
41
64
|
|
|
42
|
-
# --- Case
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
[ "$code" -eq 0 ] && [[ "$out" == *"reclaimed stale lock from stale-task"* ]] && ok "5: stale lock (dead PID) reclaimed on acquire" || no "5: exit $code, out: $out"
|
|
65
|
+
# --- Case 6: release by non-holder fails ---
|
|
66
|
+
out=$("$LOCK" release task-2 2>&1); code=$?
|
|
67
|
+
[ "$code" -eq 1 ] && [[ "$out" == *"held by task-1"* ]] \
|
|
68
|
+
&& ok "6: release by non-holder fails" || no "6: exit $code, out: $out"
|
|
47
69
|
|
|
48
|
-
# --- Case
|
|
70
|
+
# --- Case 7: release by holder succeeds and removes the lock ---
|
|
71
|
+
out=$("$LOCK" release task-1 2>&1); code=$?
|
|
72
|
+
[ "$code" -eq 0 ] && [[ "$out" == *"RELEASED by task-1"* ]] && [ ! -f "$CREW_REPO/.worktrees/.merge-lock" ] \
|
|
73
|
+
&& ok "7: release by holder removes lock file" || no "7: exit $code, out: $out"
|
|
74
|
+
|
|
75
|
+
# --- Case 8: status reports UNLOCKED after release ---
|
|
49
76
|
out=$("$LOCK" status 2>&1); code=$?
|
|
50
|
-
[ "$code" -eq 0 ] && [
|
|
77
|
+
[ "$code" -eq 0 ] && [ "$out" = "UNLOCKED" ] \
|
|
78
|
+
&& ok "8: status UNLOCKED after release" || no "8: exit $code, out: $out"
|
|
79
|
+
|
|
80
|
+
# --- Case 9: expired lease is reclaimed (logged) by a contender ---
|
|
81
|
+
# Write a lock file whose lease expired long ago (epoch 2020, lease 600s).
|
|
82
|
+
printf 'task_id=old-task\nholder=old-task/run-old\nacquired_at=1577836800\nlease_seconds=600\n' \
|
|
83
|
+
> "$CREW_REPO/.worktrees/.merge-lock"
|
|
84
|
+
out=$("$LOCK" acquire task-3 "task-3/run-ccc" 2>&1); code=$?
|
|
85
|
+
[ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-3"* ]] && [[ "$out" == *"reclaimed expired lease from old-task"* ]] \
|
|
86
|
+
&& ok "9: expired lease reclaimed by contender" || no "9: exit $code, out: $out"
|
|
87
|
+
[ -f "$TMPBASE/home/.merge-lock.log" ] && grep -q "op=acquire.*result=STALE-BREAK.*previous_task=old-task" "$TMPBASE/home/.merge-lock.log" \
|
|
88
|
+
&& ok "9b: reclaim logged to the audit log" || no "9b: no STALE-BREAK line in audit log"
|
|
51
89
|
|
|
52
|
-
# --- Case
|
|
53
|
-
|
|
54
|
-
|
|
90
|
+
# --- Case 10: malformed lock file is treated as expired and reclaimed ---
|
|
91
|
+
printf 'garbage without any key=value\n' > "$CREW_REPO/.worktrees/.merge-lock"
|
|
92
|
+
out=$("$LOCK" acquire task-4 "task-4/run-ddd" 2>&1); code=$?
|
|
93
|
+
[ "$code" -eq 0 ] && [[ "$out" == *"ACQUIRED by task-4"* ]] \
|
|
94
|
+
&& ok "10: malformed lock reclaimed" || no "10: exit $code, out: $out"
|
|
55
95
|
|
|
56
|
-
# --- Case
|
|
57
|
-
|
|
58
|
-
[ "$
|
|
96
|
+
# --- Case 11: every op appends to the audit log ---
|
|
97
|
+
lines=$(wc -l < "$TMPBASE/home/.merge-lock.log")
|
|
98
|
+
[ "$lines" -ge 8 ] && ok "11: audit log has $lines lines" || no "11: audit log has only $lines lines"
|
|
59
99
|
|
|
60
|
-
# --- Case
|
|
61
|
-
# A lock file with "-" as PID (or empty) should be reclaimable.
|
|
62
|
-
echo "old-task 2026-09-10T06:21:15Z -" > "$CREW_REPO/.worktrees/.merge-lock"
|
|
63
|
-
out=$("$LOCK" acquire task-4 33333 2>&1); code=$?
|
|
64
|
-
[ "$code" -eq 0 ] && [[ "$out" == *"reclaimed stale lock from old-task"* ]] && ok "9: lock with '-' PID reclaimed as stale" || no "9: exit $code, out: $out"
|
|
100
|
+
# --- Case 12: concurrent acquirers serialize — exactly one wins ---
|
|
65
101
|
"$LOCK" release task-4 >/dev/null 2>&1
|
|
102
|
+
for n in $(seq 1 20); do
|
|
103
|
+
( "$LOCK" acquire "racer-$n" "racer-$n/run" >/dev/null 2>&1 ) &
|
|
104
|
+
done
|
|
105
|
+
wait
|
|
106
|
+
holder_count=$("$LOCK" status | grep -c '^task_id=racer-' || true)
|
|
107
|
+
[ "$holder_count" -eq 1 ] && ok "12: exactly one concurrent acquirer holds the lock" || no "12: holder_count=$holder_count"
|
|
108
|
+
grep -q "result=ACQUIRED" "$TMPBASE/home/.merge-lock.log" && ok "12b: at least one ACQUIRED logged" || no "12b: no ACQUIRED in audit log"
|
|
66
109
|
|
|
67
110
|
echo ""
|
|
68
111
|
echo "merge-lock: $pass passed, $fail failed"
|
package/lib/test-orphan-sweep.sh
CHANGED
|
@@ -22,6 +22,9 @@ no() { echo "FAIL: $1"; fail=$((fail + 1)); }
|
|
|
22
22
|
TMPBASE="$(mktemp -d)"
|
|
23
23
|
trap 'rm -rf "$TMPBASE"' EXIT
|
|
24
24
|
|
|
25
|
+
# Keep merge-lock.sh's audit log out of the real $CREW_HOME during tests.
|
|
26
|
+
export CREW_HOME="$TMPBASE/home"
|
|
27
|
+
|
|
25
28
|
# new_fixture <name> — fresh git repo, one commit on main; exports CREW_REPO.
|
|
26
29
|
new_fixture() {
|
|
27
30
|
local dir="$TMPBASE/$1"
|
|
@@ -90,21 +93,32 @@ echo "$out" | grep -q "→ FAILED" && ok "4: FAILED line printed (no false 'remo
|
|
|
90
93
|
[ -d .worktrees/T4 ] && ok "4: failed worktree left in place" || no "4: worktree gone despite failed removal"
|
|
91
94
|
git worktree unlock .worktrees/T4 2>/dev/null || true
|
|
92
95
|
|
|
93
|
-
# --- Case 5: lock path ---
|
|
96
|
+
# --- Case 5: lock path (lease-based; bug 2fc8f52f) ---
|
|
94
97
|
new_fixture case5
|
|
95
98
|
mkdir -p .worktrees
|
|
96
|
-
|
|
99
|
+
# 5a: expired lease, no running session → released (logged to the audit log).
|
|
100
|
+
printf 'task_id=T5\nholder=T5/run-old\nacquired_at=1577836800\nlease_seconds=600\n' > .worktrees/.merge-lock
|
|
97
101
|
out=$(CREW_ACTIVE_TASKS="" "$SWEEP" clean 2>&1); code=$?
|
|
98
|
-
[ ! -f .worktrees/.merge-lock ] && ok "5a:
|
|
99
|
-
echo "$out" | grep -q "
|
|
102
|
+
[ ! -f .worktrees/.merge-lock ] && ok "5a: expired-lease lock released" || no "5a: lock file still present"
|
|
103
|
+
echo "$out" | grep -q "STALE_LOCK" && ok "5a: STALE_LOCK line printed" || no "5a: no STALE_LOCK line in output"
|
|
104
|
+
echo "$out" | grep -q "released (lease expired)" && ok "5a: release reported" || no "5a: no release line in output"
|
|
100
105
|
[ "$code" -eq 0 ] && ok "5a: exit 0" || no "5a: exit $code, want 0"
|
|
101
106
|
|
|
102
|
-
|
|
107
|
+
# 5b: valid lease held by an active run → untouched regardless of lease age.
|
|
108
|
+
now=$(date +%s)
|
|
109
|
+
printf 'task_id=%s\nholder=%s/run-live\nacquired_at=%s\nlease_seconds=600\n' "$UUID_LOCK" "$UUID_LOCK" "$now" > .worktrees/.merge-lock
|
|
103
110
|
out=$(CREW_ACTIVE_TASKS="$UUID_LOCK" "$SWEEP" clean 2>&1); code=$?
|
|
104
111
|
[ -f .worktrees/.merge-lock ] && ok "5b: active holder's lock not released" || no "5b: active holder's lock was released"
|
|
105
112
|
echo "$out" | grep -q "ACTIVE_LOCK" && ok "5b: ACTIVE_LOCK line printed" || no "5b: no ACTIVE_LOCK line in output"
|
|
106
113
|
[ "$code" -eq 0 ] && ok "5b: exit 0" || no "5b: exit $code, want 0"
|
|
107
114
|
|
|
115
|
+
# 5c: valid lease, no running session → held, NOT released (lease not expired).
|
|
116
|
+
printf 'task_id=T6\nholder=T6/run-live\nacquired_at=%s\nlease_seconds=600\n' "$now" > .worktrees/.merge-lock
|
|
117
|
+
out=$(CREW_ACTIVE_TASKS="" "$SWEEP" clean 2>&1); code=$?
|
|
118
|
+
[ -f .worktrees/.merge-lock ] && ok "5c: unexpired-lease lock not released" || no "5c: unexpired-lease lock was released"
|
|
119
|
+
echo "$out" | grep -q "HELD_LOCK" && ok "5c: HELD_LOCK line printed" || no "5c: no HELD_LOCK line in output"
|
|
120
|
+
[ "$code" -eq 0 ] && ok "5c: exit 0" || no "5c: exit $code, want 0"
|
|
121
|
+
|
|
108
122
|
# --- Case 6: report mode without active data ---
|
|
109
123
|
new_fixture case6
|
|
110
124
|
merged_worktree T7
|