@matt82198/aesop 0.1.0-beta.3 → 0.1.0-beta.5

package/daemons/CLAUDE.md

@@ -0,0 +1,36 @@
+# daemons/ — Watchdog daemon
+
+**Purpose**: Long-running backup and secret-scan daemon for fleet-wide repo safety.
+
+## Files
+
+- **run-watchdog.sh** (1.7K): Interactive daemon supervisor; spawns backup-fleet.sh every 150s, maintains heartbeat guard (200s dedupe window), logs to FLEET-BACKUP.log. Calls tools/alert_bridge.py --scan after each cycle to post HIGH/CRITICAL alerts to Slack/Discord (opt-in). Traps INT/TERM cleanly.
+- **backup-fleet.sh** (5K): Core backup worker; discovers repos (~/.*, ~/*, ~/dev/*), stashes uncommitted work to backup/* branches, pushes unpushed commits, scans tracked files for secrets. Blocks push if secret-scan fails (marks BLOCKED state).
+
+## Contracts
+
+**Env vars consumed**: `AESOP_ROOT` (defaults to `.`) — root of project; prepends to all state/ and tools/ paths.
+
+**State files written** (git-ignored):
+- `$AESOP_ROOT/state/.watchdog-heartbeat` — epoch seconds; dedupe guard (200s window).
+- `$AESOP_ROOT/state/.watchdog-repos.json` — per-cycle snapshot: [{repo, state: CLEAN|PUSHED|SNAPSHOTTED|BLOCKED, age}].
+- `$AESOP_ROOT/state/FLEET-BACKUP.log` — append-only; cycle start/end, repo statuses, secret-scan blocks.
+- `$AESOP_ROOT/state/SECURITY-ALERTS.log` — append-only; security alerts (read by alert_bridge.py to post to webhooks).
+- `$AESOP_ROOT/state/.alert-bridge-cursor` — line number of last sent alert (ensures idempotent alert dispatch).
+
+**Exit behaviors**: run-watchdog exits 0 always (trap handles clean shutdown). backup-fleet exits 0 even on secret-scan block (marks repo BLOCKED, continues).
+
+**Cadence**: 150s cycle; heartbeat dedupe within 200s.
+
+## Invariants & Gotchas
+
+1. **Single-instance guard (atomic lock)**: run-watchdog.sh uses atomic mkdir-based lockfile (`.watchdog-lock/`) to prevent concurrent daemons.
+   - Atomic acquire: `mkdir $LOCK_DIR` is guaranteed atomic on POSIX systems (returns 0 only to first caller).
+   - Stale-lock recovery: Crashed holder won't wedge daemon forever; lock older than 300s is reclaimed atomically.
+   - Guards both daemon mode and `--once` mode (no bypass).
+2. **Testing override**: Set `AESOP_WATCHDOG_CYCLE_CMD` env var to replace backup-fleet.sh invocation (allows tests to use mock cycle without running real backup).
+3. **CRLF-safe, no line continuations**: Use POSIX-safe heredocs; never add `\` for line wrap.
+4. **Secret-scan gate**: `scan_tracked_files()` calls `$AESOP_ROOT/tools/secret_scan.py` on staged/modified files; non-0 exit blocks push, marks repo BLOCKED.
+5. **Append-only logs**: FLEET-BACKUP.log only grows; rotate via tools/rotate_logs.py.
+6. **Path dedup via realpath**: Avoids processing symlinked repos or dot-dir aliases twice.
+7. **Alert Bridge integration** (NEW wave-14): After each backup-fleet.sh cycle, run-watchdog.sh calls `python $AESOP_ROOT/tools/alert_bridge.py --scan || true` to post HIGH/CRITICAL security alerts and heartbeat staleness to Slack/Discord. No-op if webhook_url null/absent in aesop.config.json (feature opt-in). Uses cursor file for idempotent dispatch (no re-sends). See tools/alert_bridge.py for details.

package/daemons/backup-fleet.sh

@@ -3,16 +3,49 @@ set -uo pipefail
 # Backup fleet repos: stash uncommitted work, push unpushed commits to backup branches.
 # Runs every 150s from run-watchdog.sh. Abort on secret-scan failures.
 # Improvements: dot-directory discovery, path dedup, tracked-files-only secret scanning.
+# P2 FIX: JSON escaping for repo names and NUL-delimited internal protocol.
 
 AESOP_ROOT="${AESOP_ROOT:-.}"
 HEARTBEAT="$AESOP_ROOT/state/.watchdog-heartbeat"
 LOG="$AESOP_ROOT/state/FLEET-BACKUP.log"
 REPOS_STATUS="$AESOP_ROOT/state/.watchdog-repos.json"
 
+# P2 FIX: script-global temp file handles (cycle-status JSON build + per-repo
+# NUL-protocol scratch file), cleaned up unconditionally via trap below so an
+# interrupted cycle (INT/TERM) or unexpected error never leaks mktemp files
+# over the daemon's long-running lifecycle. Initialized empty so `rm -f`
+# under `set -u` is always safe even if the trap fires before mktemp runs.
+temp_json=""
+temp_result=""
+cleanup_temp_files() {
+  rm -f "$temp_json" "$temp_result"
+}
+trap cleanup_temp_files EXIT INT TERM
+
 date +%s > "$HEARTBEAT" 2>/dev/null
 ts() { date '+%Y-%m-%d %H:%M:%S'; }
 log() { echo "[$(ts)] $*" | tee -a "$LOG"; }
 
+# P2 FIX: JSON escape function for safe string interpolation in JSON
+# Escapes backslash, quote, and control characters (newline, tab, carriage return, etc.)
+json_escape() {
+  local s="$1"
+  # Escape backslash first (must be first to avoid double-escaping)
+  s="${s//\\/\\\\}"
+  # Escape double quote
+  s="${s//\"/\\\"}"
+  # Escape newline
+  s="${s//$'\n'/\\n}"
+  # Escape carriage return
+  s="${s//$'\r'/\\r}"
+  # Escape tab
+  s="${s//$'\t'/\\t}"
+  # Escape backspace, form feed, and other C0 controls
+  s="${s//$'\b'/\\b}"
+  s="${s//$'\f'/\\f}"
+  printf '%s' "$s"
+}
+
 is_touched() {
   local repo="$1"
   [ ! -d "$repo/.git" ] && return 1
@@ -33,31 +66,60 @@ get_tracked_modifications() {
   ) | sort -u
 }
 
+get_untracked_files() {
+  local repo="$1"
+  (
+    cd "$repo" || return 1
+    git ls-files --others --exclude-standard 2>/dev/null
+  ) | sort -u
+}
+
 scan_tracked_files() {
   local repo="$1"
   local tracked_files
-  local file_paths=""
+  local untracked_files
+  local -a file_paths=()
+  local repo_win
+
+  # Convert repo path to Windows format for Python compatibility (Git Bash on Windows)
+  repo_win=$(cd "$repo" 2>/dev/null && pwd -W 2>/dev/null || echo "$repo")
+
   tracked_files=$(get_tracked_modifications "$repo")
+  untracked_files=$(get_untracked_files "$repo")
 
-  if [ -z "$tracked_files" ]; then
-    return 0
+  # Collect tracked modifications
+  if [ -n "$tracked_files" ]; then
+    while IFS= read -r file; do
+      if [ -n "$file" ] && [ -f "$repo/$file" ]; then
+        file_paths+=("$repo_win/$file")
+      fi
+    done <<EOF
+$tracked_files
+EOF
   fi
 
-  while IFS= read -r file; do
-    if [ -n "$file" ] && [ -f "$repo/$file" ]; then
-      file_paths="$file_paths $repo/$file"
-    fi
-  done <<EOF
-$tracked_files
+  # Collect untracked files (ITEM 1 fix)
+  if [ -n "$untracked_files" ]; then
+    while IFS= read -r file; do
+      if [ -n "$file" ] && [ -f "$repo/$file" ]; then
+        file_paths+=("$repo_win/$file")
+      fi
+    done <<EOF
+$untracked_files
 EOF
+  fi
 
-  if [ -z "$file_paths" ]; then
+  if [ ${#file_paths[@]} -eq 0 ]; then
     return 0
   fi
 
   if [ -f "$AESOP_ROOT/tools/secret_scan.py" ]; then
-    python "$AESOP_ROOT/tools/secret_scan.py" $file_paths >/dev/null 2>&1
+    # ITEM 2 fix: use "${file_paths[@]}" to properly quote array elements
+    python "$AESOP_ROOT/tools/secret_scan.py" "${file_paths[@]}" >/dev/null 2>&1
+    return $?
   fi
+
+  return 0
 }
 
 get_default_branch() {
@@ -88,11 +150,11 @@ process_repo() {
         WIPREF="backup/wip-$(date +%Y%m%d)"
         if scan_tracked_files "$repo"; then
           if git push -qf origin "$COMMIT:refs/heads/$WIPREF" 2>/dev/null; then
-            printf 'SNAPSHOTTED|%s\n' "$name"
+            printf 'SNAPSHOTTED\0%s\n' "$name"
             exit 0
           fi
         else
-          printf 'BLOCKED|%s\n' "$name"
+          printf 'BLOCKED\0%s\n' "$name"
           exit 0
         fi
       fi
@@ -107,27 +169,27 @@ process_repo() {
         WIPREF="backup/master-wip-$(date +%Y%m%d)"
         if scan_tracked_files "$repo"; then
           if git push -qf origin "HEAD:refs/heads/$WIPREF" 2>/dev/null; then
-            printf 'SNAPSHOTTED|%s\n' "$name"
+            printf 'SNAPSHOTTED\0%s\n' "$name"
             exit 0
           fi
         else
-          printf 'BLOCKED|%s\n' "$name"
+          printf 'BLOCKED\0%s\n' "$name"
           exit 0
         fi
       else
         if scan_tracked_files "$repo"; then
           if git push -q origin "$branch" 2>/dev/null; then
-            printf 'PUSHED|%s\n' "$name"
+            printf 'PUSHED\0%s\n' "$name"
             exit 0
           fi
         else
-          printf 'BLOCKED|%s\n' "$name"
+          printf 'BLOCKED\0%s\n' "$name"
           exit 0
         fi
       fi
     fi
 
-    printf 'CLEAN|%s\n' "$name"
+    printf 'CLEAN\0%s\n' "$name"
   )
 }
 
@@ -154,15 +216,24 @@ for dir in ~/.* ~/* ~/dev/*; do
 $real_dir"
 
   if is_touched "$dir"; then
-    result=$(process_repo "$dir")
-    IFS="|" read -r state name <<< "$result"
+    # P0 FIX: Eliminate $() command substitution to preserve NUL bytes in protocol
+    # Direct redirect maintains NUL delimiters (process_repo emits STATE\0name format)
+    temp_result=$(mktemp)
+    process_repo "$dir" > "$temp_result"
+    # Read first field (state) and second field (name) using NUL delimiter
+    # Portable solution: use awk with RS="\0" instead of GNU sed \x0 (BSD/macOS compatible)
+    state=$(awk 'BEGIN{RS="\0"} NR==1 {print}' "$temp_result")
+    name=$(awk 'BEGIN{RS="\0"} NR==2 {print}' "$temp_result" | tr -d '\n')
+    rm -f "$temp_result"
     [ -z "$state" ] && continue
     if [ "$first" = 1 ]; then
       first=0
     else
       echo "," >> "$temp_json"
     fi
-    printf '{"repo":"%s","state":"%s","age":"%s"}' "$name" "$state" "$(date -Iseconds)" >> "$temp_json"
+    # P2 FIX: JSON-escape all interpolated values before emitting
+    # P2 FIX: Portable date format (not GNU-only date -Iseconds)
+    printf '{"repo":"%s","state":"%s","age":"%s"}' "$(json_escape "$name")" "$(json_escape "$state")" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$temp_json"
     log "$state: $name"
   fi
 done

package/daemons/run-watchdog.sh

@@ -3,16 +3,71 @@
 # Backs up committed + uncommitted fleet work and scans for security issues every 150s.
 # Usage: run-watchdog.sh [--once]
 #
-# Configuration: export AESOP_ROOT=/path/to/aesop before running, or edit default below.
+# Configuration: export AESOP_ROOT=/path/to/aesop before running (defaults to script directory's parent).
+# Testing: export AESOP_WATCHDOG_CYCLE_CMD to override backup-fleet.sh invocation
 
-AESOP_ROOT="${AESOP_ROOT:-.}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+AESOP_ROOT="${AESOP_ROOT:-$(dirname "$SCRIPT_DIR")}"
 MODE="${1:-daemon}"
+LOCK_DIR="$AESOP_ROOT/state/.watchdog-lock"
+LOCK_STALE_THRESHOLD=300
 
-# Heartbeat guard: skip if already running (within 200s)
-__hb=$(cat "$AESOP_ROOT/state/.watchdog-heartbeat" 2>/dev/null)
-__now=$(date +%s)
-if [ "$MODE" != "--once" ] && [ -n "$__hb" ] && [ $((__now - __hb)) -lt 200 ] 2>/dev/null; then
-  echo "watchdog already running (heartbeat $((__now - __hb))s ago) — not starting a duplicate."
+# Atomic lock acquire: mkdir is atomic (POSIX guarantees)
+# Returns 0 if lock acquired, 1 if held by another process, 2 if stale lock reclaimed
+acquire_lock() {
+  local lock_dir="$1"
+  local stale_threshold="$2"
+
+  # Ensure parent directory exists
+  mkdir -p "$(dirname "$lock_dir")" 2>/dev/null || true
+
+  # Try to create lock directory atomically (this is atomic on all POSIX systems)
+  if mkdir "$lock_dir" 2>/dev/null; then
+    # We created it; write timestamp and PID
+    date +%s > "$lock_dir/timestamp" 2>/dev/null
+    echo $$ > "$lock_dir/pid" 2>/dev/null
+    return 0
+  fi
+
+  # Lock directory already exists; check if it's stale
+  if [ -d "$lock_dir" ] && [ -f "$lock_dir/timestamp" ]; then
+    local lock_mtime=$(cat "$lock_dir/timestamp" 2>/dev/null || echo 0)
+    local now=$(date +%s)
+    local lock_age=$((now - lock_mtime))
+
+    if [ "$lock_age" -gt "$stale_threshold" ]; then
+      # Lock is stale; try to reclaim it
+      rm -rf "$lock_dir" 2>/dev/null || true
+      if mkdir "$lock_dir" 2>/dev/null; then
+        date +%s > "$lock_dir/timestamp" 2>/dev/null
+        echo $$ > "$lock_dir/pid" 2>/dev/null
+        echo "watchdog lock was stale (${lock_age}s) — reclaimed." >&2
+        return 2
+      fi
+    fi
+  fi
+
+  # Lock is held by another process
+  return 1
+}
+
+# Release lock: verify ownership before removing (P0 fix)
+release_lock() {
+  local lock_dir="$1"
+  if [ -f "$lock_dir/pid" ]; then
+    local lock_pid=$(cat "$lock_dir/pid" 2>/dev/null || echo "")
+    if [ "$lock_pid" = "$$" ]; then
+      rm -rf "$lock_dir" 2>/dev/null
+    fi
+  fi
+}
+
+# Try to acquire lock (applies to both --once and daemon modes)
+acquire_lock "$LOCK_DIR" "$LOCK_STALE_THRESHOLD"
+lock_result=$?
+
+if [ $lock_result -eq 1 ]; then
+  echo "watchdog already running — not starting a duplicate."
   exit 0
 fi
 
@@ -21,17 +76,45 @@ echo "  FLEET WATCHDOG DAEMON  ·  backup + ensure-push + scan / 150s"
 echo "  logs: $AESOP_ROOT/state/FLEET-BACKUP.log   ·   Ctrl-C to stop"
 echo "==================================================================="
 echo "[$(date '+%F %T')] === watchdog daemon (shell) STARTED ===" >> "$AESOP_ROOT/state/FLEET-BACKUP.log"
-trap 'echo "[$(date "+%F %T")] === watchdog daemon (shell) STOPPED ===" >> "$AESOP_ROOT/state/FLEET-BACKUP.log"; echo "stopped."; exit 0' INT TERM
+trap "release_lock \"$LOCK_DIR\"; echo \"[$(date '+%F %T')] === watchdog daemon (shell) STOPPED ===\" >> \"$AESOP_ROOT/state/FLEET-BACKUP.log\"; echo \"stopped.\"; exit 0" INT TERM
+
+# Allow override of backup cycle command (for testing)
+# Use array to safely handle paths with spaces (P1 fix)
+if [ -n "$AESOP_WATCHDOG_CYCLE_CMD" ]; then
+  # Override: run as-is through bash -c
+  CYCLE_CMD_ARRAY=("bash" "-c" "$AESOP_WATCHDOG_CYCLE_CMD")
+else
+  # Default: array form for proper quoting
+  CYCLE_CMD_ARRAY=("bash" "$AESOP_ROOT/daemons/backup-fleet.sh")
+fi
 
 if [ "$MODE" = "--once" ]; then
-  bash "$AESOP_ROOT/daemons/backup-fleet.sh" 2>&1
-  exit 0
+  full_out=$("${CYCLE_CMD_ARRAY[@]}" 2>&1)
+  cmd_exit=$?
+  echo "$full_out"
+  if [ $cmd_exit -ne 0 ]; then
+    err_msg="[$(date '+%F %T')] ERROR: cycle #1 failed with exit code $cmd_exit"
+    echo "$err_msg" >> "$AESOP_ROOT/state/FLEET-BACKUP.log"
+    echo "[ERROR: exit $cmd_exit]" >&2
+  fi
+  python "$AESOP_ROOT/tools/alert_bridge.py" --scan || true
+  release_lock "$LOCK_DIR"
+  exit $cmd_exit
 fi
 
 n=0
 while true; do
   n=$((n+1))
-  out=$(bash "$AESOP_ROOT/daemons/backup-fleet.sh" 2>&1 | tail -2)
-  printf '%s  cycle #%d\n%s\n' "$(date '+%H:%M:%S')" "$n" "$out"
+  full_out=$("${CYCLE_CMD_ARRAY[@]}" 2>&1)
+  cmd_exit=$?
+  if [ $cmd_exit -eq 0 ]; then
+    out=$(echo "$full_out" | tail -2)
+    printf '%s  cycle #%d\n%s\n' "$(date '+%H:%M:%S')" "$n" "$out"
+  else
+    echo "[$(date '+%F %T')] ERROR: cycle #$n failed with exit code $cmd_exit" >> "$AESOP_ROOT/state/FLEET-BACKUP.log"
+    out=$(echo "$full_out" | tail -2)
+    printf '%s  cycle #%d [ERROR: exit %d]\n%s\n' "$(date '+%H:%M:%S')" "$n" "$cmd_exit" "$out"
+  fi
+  python "$AESOP_ROOT/tools/alert_bridge.py" --scan || true
   sleep 150
 done

package/dash/CLAUDE.md

@@ -0,0 +1,32 @@
+# dash/ — Dashboard TUI
+
+**Purpose**: Real-time TUI fleet monitoring — watchdog daemon + agent activity display.
+
+## Files
+
+- **watchdog-gui.sh** — Read-only terminal UI (4s refresh, double-buffered, no flicker). Displays daemon status, backed-up repos, heartbeats, recent events. Launch in own window; never inside tool shell. CRLF-safe (no line continuations).
+- **dash-extra.mjs** — Node.js agent activity detector. Scans AESOP_TRANSCRIPTS_ROOT for agent-*.jsonl files (last 12 min). TUI output by default; --json for web endpoints. Requires node on PATH (unavailable fallback if missing).
+
+## State contracts
+
+**watchdog-gui.sh reads** (heartbeat staleness thresholds applied):
+- `state/.watchdog-heartbeat` (epoch, threshold: 300s for watchdog)
+- `state/FLEET-BACKUP.log` (append-only, tail 3 lines)
+- `state/SECURITY-ALERTS.log` (counts HIGH/MED, grep filtering RESOLVED-FP)
+- `state/.watchdog-repos.json` (jq-parsed, status per repo)
+- `state/.heartbeats/*` (epoch, per-agent thresholds: 3600s monitor, 300s watchdog, 1800s default)
+
+**dash-extra.mjs reads**:
+- `~/.claude/projects/**/ agent-*.jsonl` (modified time, <12min filter, top 8 recent)
+- `state/SECURITY-ALERTS.log` (severity classification: HIGH/MED/DRIFT)
+
+**Refresh cadence**:
+- watchdog-gui.sh: 4s loop (infinite, Ctrl-C to exit)
+- dash-extra.mjs: called on-demand (no built-in loop)
+
+## Invariants & TUI conventions
+
+- **Never modify fleet state** — both tools are read-only dashboards.
+- **Infinite loop design** — watchdog-gui.sh runs forever; launch in dedicated terminal window, never inside tool shells.
+- **CRLF-safe** — no bash line continuations; portable POSIX.
+- **Node optional** — dash-extra.mjs gracefully prints "unavailable" if node missing; doesn't fail watchdog-gui.sh.