autonomous-sdlc-harness 0.4.0 → 0.4.1

package/templates/scripts/lib/harness-run-lib.sh

@@ -58,8 +58,12 @@
 #      and are written only by the `hr_lane_*` functions.
 #   2. THE RUN REGISTRY writes the registry file its caller names. Fence: that
 #      file is `<root>/<state_dir>/autonomous_logs/registry.json`, resolved
-#      through `hr_state_path`, and it is written only by `hr_registry_init`
-#      and `hr_registry_set`.
+#      through `hr_state_path`, plus the lock directory `<file>.lock` (and its
+#      `.stale.*` move-aside and `.break` mutex while a stale one is broken)
+#      and the temp files
+#      `.registry.*` in the registry's own directory. Written only by
+#      `hr_registry_init`, `hr_registry_set` and the `hr_registry_lock` /
+#      `hr_registry_unlock` pair `hr_registry_set` calls.
 #   3. THE REMOTE STATE BUNDLE writes the files its format lists. Fence: inside
 #      `<root>/<state_dir>/` (resolved through `hr_state_dir`), only
 #      `autonomous_logs/remote_status.json`, `clarifications/<branch>/`,
@@ -71,8 +75,8 @@
 #      there but a writer's own failed temp file is ever removed.
 #
 # A caller that calls no `hr_lane_*`, `hr_registry_init`, `hr_registry_set`,
-# `hr_remote_status_write`, `hr_remote_bundle_write` or
-# `hr_remote_bundle_restore` function still gets a library that only reads. The
+# `hr_registry_lock`, `hr_registry_unlock`, `hr_remote_status_write`,
+# `hr_remote_bundle_write` or `hr_remote_bundle_restore` function still gets a library that only reads. The
 # lane's ceilings are the only environment values here that carry policy, because
 # the lane is machine-scoped and has no configuration key to carry them; each is
 # named where it is used. `XDG_STATE_HOME`, `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`,
@@ -165,7 +169,9 @@
 # because callers capture stdout. The one pass-through is the registry's two
 # writers, `hr_registry_init` and `hr_registry_set`, which leave the shell's,
 # `mktemp`'s and `jq`'s own stderr on a failed write to the caller, as the
-# watcher's bodies they replaced did — that stream is the watcher's log. That silence is why the lane reports a lock it BROKE through a
+# watcher's bodies they replaced did — that stream is the watcher's log — and
+# add one line of their own, naming the lock, when `hr_registry_set` cannot
+# take the registry lock. That silence is why the lane reports a lock it BROKE through a
 # variable instead of a log line — the caller owns the log.
 #
 # NAMING. Every function is prefixed `hr_`; every variable this file touches
@@ -1110,31 +1116,187 @@ hr_push_env_files() {
 # Each function takes the registry file as its first argument — the caller
 # resolves it as `hr_state_path <root> autonomous_logs/registry.json` — and is
 # write exception 2 in the header. Every value is written as a JSON string;
-# every write stamps `branch` and `updated_at` on the record and replaces the
-# file through a `mktemp` + `mv`. No shell option is assumed: the caller may set
-# `-e`, `-u` or neither.
+# every write stamps `branch` and `updated_at` on the record. No shell option is
+# assumed: the caller may set `-e`, `-u` or neither.
+#
+# EVERY WRITE IS A LOCKED READ-MODIFY-WRITE. Every script sharing the file
+# writes it, and so does the watcher's own `( … ) &` engine subshell, so two
+# unserialized writers each read the same record and the second `mv` silently
+# drops the first one's key. The lock is a `mkdir` of `<file>.lock` holding an
+# `owner` file with a per-call token — the `mktemp` name of that call's temp
+# file, because `$$` inside a subshell is the parent's pid and bash 3.2 has no
+# per-subshell pid variable — and only a call whose token is still in `owner`
+# releases it.
+#
+# STALENESS IS THE LOCK DIRECTORY'S AGE, NEVER A PID. The stall watchdog kills
+# the engine subshell mid-write, so a dead holder is expected, and a pid says
+# nothing about a lock the killed subshell took under its parent's `$$`. A write
+# is one `jq` over a file of a few records, well under a second, so a lock 10 s
+# old is a crashed holder; a waiter polls every 0.05 s for up to 12 s — past the
+# stale age, so a waiter that arrives just after a crash breaks the lock rather
+# than giving up. Both are constants here, not environment values: the header
+# allows policy-carrying environment values for the lane only.
+#
+# BREAKERS ARE SERIALIZED, AND NONE RENAMES A LIVE LOCK. A waiter that judges
+# the lock stale takes a second `mkdir` mutex, `<file>.lock.break`, judges the
+# lock's age again under it, and only then renames the lock aside and empties
+# it, as `hr_lane_acquire` does. A lock re-taken since the first judgement is
+# fresh, so the second leaves it alone. Between the second judgement and the
+# rename, nothing but a breaker or the lock's own holder frees the path:
+# breakers are serialized, and the holder of a lock that old is dead by the
+# stale rule. The watchdog can kill a breaker inside the mutex, so a mutex 10 s
+# old is removed by the next waiter. That removal is the one window left open.
+# Two waiters that judge one abandoned mutex old together can both remove it,
+# and the second can remove the fresh mutex the first has just taken. Two
+# breakers then judge the lock at once, so reaching it takes a breaker killed
+# inside the mutex, a stale lock and three concurrent writers.
+#
+# A write that cannot take the lock prints one stderr line naming it, writes
+# nothing and returns 1.
+#
+# ONE CALL, SEVERAL KEYS. `hr_registry_set <file> <branch> <key> <value> [<key>
+# <value> …]` writes every pair in one `jq` pass under one lock and one `mv`.
+# A caller writing two keys that a concurrent reader must never see apart — a
+# status and the reason for it — MUST pass them in one call.
+#
+# READERS TAKE NO LOCK. The temp file sits beside the registry, so the `mv` is a
+# same-filesystem rename and `hr_registry_get`, `hr_registry_branches` and any
+# other `jq` over the file see the old record or the new one, never a partial
+# one. `hr_registry_init` creates the file by `ln`-ing a complete temp file to
+# the registry name, which fails when the name exists, so a reader's create
+# never truncates a registry a writer has just created.
 # ---------------------------------------------------------------------------
 
-# Create an empty registry at <file> when none exists.
+# The directory holding <file> — where its temp files go, so every `mv` over it
+# is a same-filesystem rename.
+hr_registry_dir() {
+  local file="${1-}" dir
+  case "$file" in
+    */*)
+      dir="${file%/*}"
+      [ -n "$dir" ] || dir=/
+      ;;
+    *) dir=. ;;
+  esac
+  printf '%s\n' "$dir"
+}
+
+# Create an empty registry at <file> when none exists. Atomic: the name is
+# either absent or a complete `{"runs":{}}` — never truncated, never partial.
 hr_registry_init() {
-  local file="${1-}"
+  local file="${1-}" dir tmp
   [ -n "$file" ] || return 1
-  [ -f "$file" ] || printf '{"runs":{}}\n' >"$file"
+  [ -f "$file" ] && return 0
+  dir=$(hr_registry_dir "$file")
+  tmp=$(mktemp "$dir/.registry.XXXXXX") || return 1
+  if printf '{"runs":{}}\n' >"$tmp"; then
+    ln "$tmp" "$file" 2>/dev/null || :
+  fi
+  rm -f "$tmp"
+  [ -f "$file" ]
+}
+
+# hr_registry_lock <file> <token> — take `<file>.lock` for <token>, breaking a
+# stale one (see the section comment). 0 when held; 1, with one stderr line,
+# when the wait ceiling passed.
+hr_registry_lock() {
+  local lock="${1-}.lock" token="${2-}" polls=0 start="" now m stale bm
+  # 12 — the wait ceiling and 10 — the stale age, both in seconds. The wait is
+  # judged by the clock, because each poll's forks cost more than its sleep;
+  # 240 polls is the ceiling only when `date` gives no epoch.
+  while :; do
+    if mkdir "$lock" 2>/dev/null; then
+      if printf '%s\n' "$token" >"$lock/owner" 2>/dev/null; then
+        return 0
+      fi
+      rm -f "$lock/owner" 2>/dev/null || :
+      rmdir "$lock" 2>/dev/null || :
+      break
+    fi
+    m=$(hr_lane_mtime "$lock")
+    now=$(date +%s 2>/dev/null) || now=0
+    case "$now" in '' | *[!0-9]*) now=0 ;; esac
+    [ -n "$start" ] || start="$now"
+    if [ "$now" -gt 0 ]; then
+      [ $((now - start)) -lt 12 ] || break
+    else
+      [ "$polls" -lt 240 ] || break
+    fi
+    if [ "$m" -gt 0 ] && [ "$now" -gt 0 ] && [ $((now - m)) -ge 10 ]; then
+      if mkdir "$lock.break" 2>/dev/null; then
+        # One breaker at a time: judge the age again under the mutex, so a
+        # lock re-taken since the judgement above is fresh and left alone.
+        m=$(hr_lane_mtime "$lock")
+        now=$(date +%s 2>/dev/null) || now=0
+        case "$now" in '' | *[!0-9]*) now=0 ;; esac
+        if [ "$m" -gt 0 ] && [ "$now" -gt 0 ] && [ $((now - m)) -ge 10 ]; then
+          stale="$lock.stale.${token##*.}"
+          if mv "$lock" "$stale" 2>/dev/null; then
+            rm -f "$stale/owner" 2>/dev/null || :
+            rmdir "$stale" 2>/dev/null || :
+          fi
+        fi
+        rmdir "$lock.break" 2>/dev/null || :
+        polls=$((polls + 1))
+        continue
+      fi
+      # Another breaker holds the mutex. One the watchdog killed inside it
+      # left it behind: a mutex 10 s old is removed here.
+      bm=$(hr_lane_mtime "$lock.break")
+      if [ "$bm" -gt 0 ] && [ $((now - bm)) -ge 10 ]; then
+        rmdir "$lock.break" 2>/dev/null || :
+      fi
+    fi
+    sleep 0.05
+    polls=$((polls + 1))
+  done
+  printf 'hr_registry_set: could not take the registry lock %s\n' "$lock" >&2
+  return 1
+}
+
+# hr_registry_unlock <file> <token> — release `<file>.lock` only while its
+# `owner` still holds <token>; a lock broken and re-taken is someone else's.
+hr_registry_unlock() {
+  local lock="${1-}.lock" token="${2-}" owner=""
+  [ -r "$lock/owner" ] && { IFS= read -r owner <"$lock/owner" || :; } 2>/dev/null
+  [ -n "$token" ] && [ "$owner" = "$token" ] || return 0
+  rm -f "$lock/owner" 2>/dev/null || :
+  rmdir "$lock" 2>/dev/null || :
 }
 
-# hr_registry_set <file> <branch> <key> <value>
+# hr_registry_set <file> <branch> <key> <value> [<key> <value> …]
+# 1, writing nothing, on no key/value pair or an odd count of them. Each pair
+# reaches `jq` as `--arg kN` / `--arg vN` — jq 1.5 has no `$ARGS` — and only
+# those generated names enter the program text, never a value.
 hr_registry_set() {
-  local file="${1-}" branch="${2-}" key="${3-}" value="${4-}" tmp
-  hr_registry_init "$file" || :
-  tmp="$(mktemp)" || return 1
-  if jq --arg b "$branch" --arg k "$key" --arg v "$value" --arg now "$(date '+%Y-%m-%dT%H:%M:%S')" '
-    .runs[$b] = ((.runs[$b] // {}) + {($k): $v, "branch": $b, "updated_at": $now})
-  ' "$file" >"$tmp"; then
-    mv "$tmp" "$file"
-  else
+  local file="${1-}" branch="${2-}" dir tmp fields="" i=0 status=0
+  local -a args
+  [ -n "$file" ] && [ "$#" -ge 4 ] && [ $(($# % 2)) -eq 0 ] || return 1
+  shift 2
+  args=(--arg b "$branch" --arg now "$(date '+%Y-%m-%dT%H:%M:%S')")
+  while [ "$#" -gt 0 ]; do
+    args+=(--arg "k$i" "$1" --arg "v$i" "$2")
+    fields="$fields(\$k$i): \$v$i, "
+    i=$((i + 1))
+    shift 2
+  done
+  dir=$(hr_registry_dir "$file")
+  tmp=$(mktemp "$dir/.registry.XXXXXX") || return 1
+  if ! hr_registry_lock "$file" "$tmp"; then
     rm -f "$tmp"
     return 1
   fi
+  hr_registry_init "$file" || :
+  if jq "${args[@]}" "
+    .runs[\$b] = ((.runs[\$b] // {}) + {$fields\"branch\": \$b, \"updated_at\": \$now})
+  " "$file" >"$tmp"; then
+    mv "$tmp" "$file" || status=1
+  else
+    status=1
+  fi
+  [ "$status" -eq 0 ] || rm -f "$tmp"
+  hr_registry_unlock "$file" "$tmp"
+  return "$status"
 }
 
 # hr_registry_get <file> <branch> <key>   -> the value, or nothing

package/templates/scripts/remote-run.sh

@@ -276,7 +276,7 @@
 # no job to stop and may complete before its cancel lands — trying every one even
 # after a failure. (3) Only when (1) and (2) all succeeded, and only when a
 # local registry record exists, it writes `remote_stopped_at` and sets `status`
-# to `failed` through `hr_registry_set`; a partial stop leaves the record alone
+# to `failed` in one `hr_registry_set` call; a partial stop leaves the record alone
 # and exits 3, so running `stop` again is the remedy.
 #
 # `warm` dispatches action=warm on GitHub's OWN default branch (`gh repo view
@@ -811,8 +811,7 @@ verb_stop() {
 
   registry=$(hr_state_path "$root" autonomous_logs/registry.json) || registry=""
   if [ -n "$registry" ] && [ -f "$registry" ] && [ -n "$(hr_registry_get "$registry" "$branch" branch)" ]; then
-    hr_registry_set "$registry" "$branch" remote_stopped_at "$stopped_at" \
-      && hr_registry_set "$registry" "$branch" status failed \
+    hr_registry_set "$registry" "$branch" remote_stopped_at "$stopped_at" status failed \
       || echo "remote-run.sh: stopped on GitHub, but the local record of $branch could not be updated" >&2
   fi
   echo "remote-run.sh: stopped $branch"
@@ -904,6 +903,17 @@ set_or_fail() {
   }
 }
 
+# set_many_or_fail <key> <value> [<key> <value> …] — every pair in one write, so
+# a concurrent reader never sees an outcome half-applied.
+set_many_or_fail() {
+  local keys="" i
+  hr_registry_set "$registry" "$branch" "$@" || {
+    for ((i = 1; i <= $#; i += 2)); do keys="$keys${keys:+, }${!i}"; done
+    echo "remote-run.sh: writing $keys of $branch to '$registry' failed" >&2
+    exit "$EXIT_USAGE"
+  }
+}
+
 verb_status() {
   local runs finished_id synced_id field
   list_runs
@@ -935,18 +945,14 @@ verb_status() {
 sync_expired() {
   local line
   line=$(expired_line "$1")
-  set_or_fail status paused
-  set_or_fail pause_reason expired
-  set_or_fail remote_run_id "$1"
-  set_or_fail remote_run_url "$2"
-  set_or_fail remote_detail "$line"
-  set_or_fail remote_synced_at "$3"
+  set_many_or_fail status paused pause_reason expired remote_run_id "$1" \
+    remote_run_url "$2" remote_detail "$line" remote_synced_at "$3"
   echo "remote-run.sh: $line"
 }
 
 verb_sync() {
   local worktree runs newest id state url synced_id now older download status_file
-  local status reason detail
+  local status reason detail resume_at cycles
   worktree=$(hr_registry_get "$registry" "$branch" worktree)
   if [ -z "$worktree" ] || [ ! -d "$worktree" ]; then
     echo "remote-run.sh: refused, nothing written: the mirror working copy '$worktree' of $branch is missing" >&2
@@ -965,8 +971,7 @@ verb_sync() {
   now=$(date +%s)
 
   if [ "$state" != completed ]; then
-    set_or_fail status running
-    set_or_fail remote_synced_at "$now"
+    set_many_or_fail status running remote_synced_at "$now"
     echo "remote-run.sh: run $id of $branch is $state; the record is running, nothing downloaded"
     return 0
   fi
@@ -1039,14 +1044,11 @@ verb_sync() {
       detail="the job ended mid-run (its bundle still says running): $url"
     fi
     [ -n "$detail" ] || detail="synced from $url"
-    set_or_fail status "$status"
-    set_or_fail pause_reason "$reason"
-    set_or_fail usage_resume_at "$(hr_remote_status_get "$status_file" usage_resume_at || :)"
-    set_or_fail park_loop_cycles "$(hr_remote_status_get "$status_file" park_loop_cycles || :)"
-    set_or_fail remote_run_id "$id"
-    set_or_fail remote_run_url "$url"
-    set_or_fail remote_detail "$detail"
-    set_or_fail remote_synced_at "$now"
+    resume_at=$(hr_remote_status_get "$status_file" usage_resume_at) || resume_at=""
+    cycles=$(hr_remote_status_get "$status_file" park_loop_cycles) || cycles=""
+    set_many_or_fail status "$status" pause_reason "$reason" usage_resume_at "$resume_at" \
+      park_loop_cycles "$cycles" remote_run_id "$id" remote_run_url "$url" \
+      remote_detail "$detail" remote_synced_at "$now"
     echo "remote-run.sh: synced run $id of $branch: $status${reason:+ ($reason)}"
     return 0
   fi
@@ -1061,20 +1063,17 @@ verb_sync() {
     done
   fi
   if [ "$bundle_exists" -eq 1 ]; then
-    set_or_fail status paused
-    set_or_fail pause_reason killed
-    set_or_fail remote_run_id "$id"
-    set_or_fail remote_run_url "$url"
-    set_or_fail remote_detail "run $id ended with no state bundle (killed, cancelled or replaced): $url"
-    set_or_fail remote_synced_at "$now"
+    set_many_or_fail status paused pause_reason killed remote_run_id "$id" remote_run_url "$url" \
+      remote_detail "run $id ended with no state bundle (killed, cancelled or replaced): $url" \
+      remote_synced_at "$now"
     echo "remote-run.sh: run $id of $branch left no bundle; the record is paused (killed), nothing restored"
     return 0
   fi
 
   # Case 5 — no bundle in any run.
-  set_or_fail status failed
-  set_or_fail remote_detail "no run of $branch ever uploaded a state bundle; newest: $url"
-  set_or_fail remote_synced_at "$now"
+  set_many_or_fail status failed \
+    remote_detail "no run of $branch ever uploaded a state bundle; newest: $url" \
+    remote_synced_at "$now"
   echo "remote-run.sh: no run of $branch carries a state bundle; the record is failed"
 }
 

package/templates/state-dir/autonomous_logs/README.md

@@ -1,6 +1,6 @@
 # autonomous_logs/
 
-One readable transcript `<branch>.log` and one raw event log `<branch>.stream.jsonl` per dispatched run, plus the daemon's own `watcher.log`, the service manager's capture of its standard output and error as `watcher.out.log` and `watcher.err.log`, and `registry.json` — the run registry that indexes every run's status. All of it resolves to `<state_dir>/autonomous_logs/` **in the main checkout**, whichever sibling working copy a run actually executes in, so one repository has one set of these files however many runs it has in flight.
+One readable transcript `<branch>.log` and one raw event log `<branch>.stream.jsonl` per dispatched run, plus the daemon's own `watcher.log`, the service manager's capture of its standard output and error as `watcher.out.log` and `watcher.err.log`, and `registry.json` — the run registry that indexes every run's status. All of it resolves to `<state_dir>/autonomous_logs/` **in the main checkout**, whichever sibling working copy a run actually executes in, so one repository has one set of these files however many runs it has in flight. A `registry.json.lock` directory beside the registry exists only for the width of one registry write; one left over after a crash is broken automatically by the next writer, and is never removed by hand while a watcher runs.
 
 Everything here is written by the run daemon and read by the operator watching or diagnosing a run: the transcript is formatted live and tailable while the run is going, and the raw stream beside it is the deep-debug copy and the one thing the usage gate can parse, since a run cannot read the stream it is itself producing. The registry has readers beyond the operator, and that is what makes it different in kind from its neighbours: it is the single record of which runs exist and what state each is in, and the harness's status command, its pause and resume commands, `<scripts_dir>/restart-watcher.sh` and `<scripts_dir>/cleanup-merged-worktrees.sh` all read it before they act, and `<scripts_dir>/remote-run.sh` reads and writes it for a run executing on GitHub Actions.