@bookedsolid/rea 0.41.0 → 0.43.0

package/MIGRATING.md

@@ -389,6 +389,145 @@ per finding and produces more consistent verdicts — fewer
 same-code-different-verdict round-trips. Trade-off is push-gate
 latency.
 
+## Node-binary hook scanner (added in 0.32.0)
+
+Pre-0.32.0 every `.claude/hooks/*.sh` carried the full gate body in
+bash. Adversarial review consistently caught bash-only edge cases that
+were structurally unfixable in shell — multi-line awk encodings,
+ANSI-C escapes, deep nested-shell decoding. 0.32.0 pivoted the entire
+hook surface to a Node-binary scanner: hooks became thin shims (~20-80
+LOC each) that delegate the actual gate work to `rea hook <name>` —
+which runs the canonical scanner inside `dist/cli/index.js`.
+
+**Consumer impact:**
+
+- Run `pnpm install` (or `npm install`) after upgrading to 0.32.0+ so
+  `dist/cli/index.js` is built and the shims have something to call.
+- `.claude/hooks/*.sh` files on disk are noticeably smaller after
+  `rea upgrade`; this is the canonical post-0.32.0 shape, not a
+  truncation. `rea doctor` will tell you if a shim is the wrong
+  vintage.
+- The audit trail is unchanged: hooks still emit `rea.bash_scan`-class
+  records to `.rea/audit.jsonl` with the same field shape.
+- Performance is materially better — single Node startup per scan
+  instead of an awk/sed pipeline per pattern.
+
+If `rea doctor` reports `policy-reader Tier 1 (rea CLI)` as `warn:
+dist not found`, you skipped the build step. Run `pnpm install`.
+
+## Graceful-degradation policy reader (added in 0.37.0)
+
+The shimmed hooks need to read `.rea/policy.yaml` from a bash context
+that may or may not have python3, jq, or rea's CLI on PATH. 0.37.0
+formalized a 4-tier reader ladder:
+
+1. **Tier 1** — `rea hook policy-get` (requires `dist/cli/index.js`)
+2. **Tier 2** — `python3 + stdlib yaml` (PyYAML) — handles flow-form
+3. **Tier 3** — POSIX `awk` block-form parser (the always-available floor)
+4. **Fail-closed** — every tier unreachable: shim refuses the action
+
+Tier 1 → 2 → 3 fallthrough is silent at hook-runtime; that's
+intentional (graceful degradation), but means an unreachable Tier 1 +
+unreachable Tier 2 can silently downgrade flow-form policy lookups to
+block-form-only. `rea doctor` (0.39.0+) surfaces all three tier
+reachabilities so you can spot the gap.
+
+**Consumer impact:**
+
+- If you use FLOW-form YAML for any policy block (e.g.
+  `blocked_paths: [.env, ".env.*"]`), make sure either the rea CLI
+  dist is present OR `python3 + PyYAML` is installed. With ONLY awk
+  reachable, flow-form lookups silently no-op on every shim
+  fallthrough path and your declared policy isn't enforced.
+- Install PyYAML on CI runners: `pip3 install pyyaml`. On consumer
+  developer machines, it's almost always already present (macOS ships
+  it; major Linux distros bundle it with python3).
+- For list-valued policy keys (`blocked_paths`, `protected_writes`),
+  the loader iterates the resulting JSON via jq OR python3. Have at
+  least one on PATH or `rea doctor` (0.42.0+) will report `fail` on
+  the `policy-reader Tier 3 (awk)` row with a list-walker-specific
+  remediation message.
+
+## Shim runtime extraction (added in 0.38.0)
+
+Cosmetic-only refactor: every `.claude/hooks/*.sh` shim now sources
+`hooks/_lib/shim-runtime.sh` for shared boilerplate (env loading,
+tier classification, audit-event emission). **No consumer action
+required** — the change is byte-equivalent at the gate surface. New
+shims you author can adopt the same runtime by sourcing the shared
+helper; documented in the shim authoring guide.
+
+## Doctor health surfaces for the policy reader (added in 0.39.0)
+
+`rea doctor` gained explicit reachability checks for the 4-tier
+ladder, the dist invokability probe, and a sandbox-containment check
+on the resolved `dist/cli/index.js` path. Output lines you'll see:
+
+- `policy-reader Tier 1 (rea CLI)` — pass/warn based on dist
+  presence + actual invocation
+- `policy-reader Tier 2 (python3 + PyYAML)` — pass/warn based on
+  python3 + import yaml succeeding
+- `policy-reader Tier 3 (awk)` — pass when awk present; warn or
+  fail conditional on whether other tiers cover the gap (0.40.0
+  refined the verdict logic; 0.42.0 hardened the list-walker
+  predicate)
+- `policy-reader effective floor` — summary verdict across all three
+- `policy-reader jq (JSON accelerator)` — info-level, calls out
+  Tier 1/2 perf when jq is absent
+
+**Consumer action:** run `rea doctor` after each upgrade. The lines
+above accurately reflect what your shims will do at runtime — a
+`warn` is not a hard failure but signals a posture worth knowing
+about (e.g. flow-form policy silently no-ops). A `fail` on any tier
+row IS a hard failure that the doctor exits non-zero on.
+
+## Upgrade preview + audit summary (added in 0.41.0)
+
+Two new consumer-facing commands rolled out:
+
+### `rea upgrade --check`
+
+Dry-run preview of what `rea upgrade` would write, file-by-file, with
+unified diffs. JSON output via `--json`. Always exits 0 — this is a
+preview, not a gate. Use it before any non-trivial rea upgrade to
+sanity-check the diff:
+
+```bash
+rea upgrade --check                       # human-readable table + diffs
+rea upgrade --check --json                # machine-readable for CI
+rea upgrade --check --no-diff             # counts + paths only
+```
+
+0.42.0 added the same settings-schema validation that `rea upgrade`
+itself runs — if the merged settings would fail schema parse (typo'd
+hook event, malformed hook command, …), the preview surfaces the
+`WOULD REFUSE` message rather than promising a write the real
+upgrade would refuse. The `settings_validation` field in the JSON
+output carries the structured outcome.
+
+### `rea audit summary`
+
+High-level rollup of the audit log: counts by `tool_name`, `tier`,
+`status`, `session`, the time window covered, and a sample-verified
+chain-integrity check. `--since <duration>` (e.g. `24h`, `7d`, `2w`)
+narrows to a recent window:
+
+```bash
+rea audit summary                         # all time
+rea audit summary --since 24h             # last 24 hours
+rea audit summary --since 7d --json       # last week, JSON
+```
+
+0.42.0 hardened the rotated-file walk: pre-0.42.0 `--since` pruned
+rotated audit segments by filename stamp, which is wall-clock at the
+rotation INSTANT — not the earliest record contained. A rotated file
+from N days ago can contain records from N+M days ago when the
+rotation cycle was long, so pruning by filename silently dropped
+in-window records. Post-0.42.0 the walker reads every rotated file
+under `--since` and lets the per-record timestamp filter drop the
+out-of-window entries. Correctness over micro-optimization;
+`rea audit summary` performance is unchanged in practice.
+
 ## Policy knobs worth setting
 
 For consumers with a long-running migration branch (>30 commits since

package/dist/cli/audit-summary.d.ts

@@ -81,6 +81,21 @@ export interface AuditSummaryResult {
     window_end: string | null;
     /** Absolute paths of audit files walked. */
     files_scanned: string[];
+    /**
+     * 0.42.0 codex round 4 P2 + round 6 P2 (2026-05-16) — reserved for
+     * future use; ALWAYS EMPTY in 0.42.0. The original intent (round 4)
+     * was to soft-skip rotated segments that the operator could not
+     * read (e.g. EACCES/EPERM after a backup restore). Round 6 showed
+     * the soft-skip was unsound: without per-segment time-range
+     * metadata we cannot prove a skipped file is out-of-scope for the
+     * `--since` window, so a silent skip risks an undercount + a
+     * misleading `chain_integrity: ok`. The current implementation
+     * therefore throws on any non-ENOENT read error; this field is
+     * kept in the public schema so a future release that ships
+     * per-segment time-range metadata can populate it without breaking
+     * JSON consumers.
+     */
+    unreadable_segments: string[];
     total_events: number;
     by_tool_name: Record<string, number>;
     by_tier: Record<string, number>;

package/dist/cli/audit-summary.js

@@ -118,48 +118,49 @@ export function parseDurationSeconds(raw) {
  *     PLUS the current `audit.jsonl`. Round-1 P2: the prior shape
  *     dropped rotated history silently while the header still
  *     advertised "all time", undercounting long-lived repos.
- *   - `windowStart` set: walk every rotated file whose basename
- *     timestamp >= the cutoff, PLUS one rotated file immediately
- *     before the cutoff (the in-flight file at cutoff time may
- *     contain in-window records).
+ *   - `windowStart` set: walk EVERY rotated file. The per-record
+ *     timestamp filter inside `computeAuditSummary` then drops
+ *     out-of-window records during the scan. 0.41.0 round-3 P2 +
+ *     0.42.0 charter item 3: rotated filenames are NOT authoritative
+ *     for "earliest contained record" — they are wall-clock at the
+ *     ROTATION INSTANT, which can be days after the file's earliest
+ *     contents when the rotation size cap is reached late. Pruning
+ *     by filename therefore drops in-window records from
+ *     conservatively-rotated logs (a rotated file from 7 days ago can
+ *     still contain records from 14 days ago because the previous
+ *     rotation event was 14 days ago). The cost of walking every
+ *     rotated segment under `--since` is bounded by the rotation cap
+ *     × number of segments — comfortably manageable in the
+ *     summary-rollup setting where we already read every byte for
+ *     the in-window scan; the win is correctness.
  *
- * Sort order is timestamp-ascending; the current `audit.jsonl` is
- * always appended last (it is the newest segment of the chain).
+ * Sort order is timestamp-ascending (by FILENAME stamp); the current
+ * `audit.jsonl` is always appended last (it is the newest segment
+ * of the chain).
  */
 async function resolveSummaryFileWalk(baseDir, windowStart) {
     const reaDir = path.join(baseDir, REA_DIR);
     const currentAudit = path.join(reaDir, AUDIT_FILE);
     const files = [];
     const rotated = await listRotatedAuditFiles(reaDir);
-    if (windowStart === null) {
-        // Walk every rotated segment. The "all time" header would be a
-        // lie otherwise.
-        for (const name of rotated)
-            files.push(path.join(reaDir, name));
-    }
-    else {
-        // Rotated filenames are `audit-YYYYMMDD-HHMMSS(-N).jsonl` in UTC.
-        // We treat each filename as "rotated at this instant" and include
-        // every file rotated >= windowStart, plus one file immediately
-        // before windowStart (the in-flight file at cutoff time may
-        // contain in-window records).
-        const stampToDate = (name) => {
-            const m = /^audit-(\d{4})(\d{2})(\d{2})-(\d{2})(\d{2})(\d{2})/.exec(name);
-            if (m === null)
-                return null;
-            const iso = `${m[1]}-${m[2]}-${m[3]}T${m[4]}:${m[5]}:${m[6]}Z`;
-            const d = new Date(iso);
-            return Number.isNaN(d.getTime()) ? null : d;
-        };
-        const cutoffIdx = rotated.findIndex((n) => {
-            const d = stampToDate(n);
-            return d !== null && d >= windowStart;
-        });
-        const startIdx = cutoffIdx === -1 ? Math.max(0, rotated.length - 1) : Math.max(0, cutoffIdx - 1);
-        for (const name of rotated.slice(startIdx)) {
-            files.push(path.join(reaDir, name));
-        }
-    }
+    // Both `windowStart === null` and `windowStart` set: walk every
+    // rotated segment. Pre-0.42.0 the `windowStart` branch attempted to
+    // prune rotated files by their filename stamp ("rotated at >=
+    // windowStart minus one buffer file"). That was wrong: the filename
+    // stamp marks the ROTATION event, not the earliest record contained
+    // in the file. A rotated file's records can pre-date its filename
+    // stamp by days when the previous rotation cycle was long. Walking
+    // every rotated file and letting the per-record `timestamp >=
+    // windowStart` filter inside `computeAuditSummary` decide is the
+    // only correct approach: we never falsely drop an in-window record
+    // because of where it happens to live on disk. Reference:
+    // 0.41.0 round-3 P2 + 0.42.0 charter item 3.
+    //
+    // `windowStart === null` (no --since) already walks every rotated
+    // segment — same code path.
+    void windowStart; // intentionally unused — full-walk is correct in both modes
+    for (const name of rotated)
+        files.push(path.join(reaDir, name));
     try {
         const stat = await fs.stat(currentAudit);
         if (stat.isFile())
@@ -261,16 +262,52 @@ export async function computeAuditSummary(options = {}) {
     let latest = null;
     // We only feed in-window records to the chain-sample check.
     const inWindowRecords = [];
+    // 0.42.0 codex round 4 P2 + round 6 P2 (2026-05-16): reserved for
+    // future per-segment time-range metadata that would let us prove a
+    // skipped file is out of scope. Always empty under 0.42.0 — see
+    // the AuditSummaryResult.unreadable_segments docstring.
+    const unreadableSegments = [];
+    // We rebuild the actually-read file list as we go so the summary
+    // never claims to have scanned a file that was silently skipped.
+    // (Currently identical to `files` minus ENOENT entries since every
+    // other read error throws — kept as a separate accumulator so the
+    // shape stays correct when the future `unreadable_segments`
+    // soft-skip path lands.)
+    const actuallyScanned = [];
     for (const filePath of files) {
         let raw;
         try {
             raw = await fs.readFile(filePath, 'utf8');
         }
         catch (e) {
-            if (e.code === 'ENOENT')
+            const errno = e.code;
+            if (errno === 'ENOENT')
                 continue;
-            throw e;
+            // 0.42.0 codex round 4 P2 + round 5 P2 + round 6 P2 (2026-05-16):
+            // earlier rounds attempted to soft-skip unreadable rotations to
+            // accommodate backup-restore artifacts. Round 6 caught that the
+            // soft-skip is unsound: `resolveSummaryFileWalk` now enqueues
+            // every rotated segment under `--since` (filename-stamp pruning
+            // was correctly removed because the stamp marks the rotation
+            // event, not the earliest record contained), so we CANNOT prove
+            // an unreadable file is out of scope without reading it. A
+            // silent skip would mean `rea audit summary` could exit 0 with
+            // an undercount AND `chain_integrity: ok` while real in-window
+            // records went uncounted.
+            //
+            // Throwing with a precise, actionable error is the right call:
+            // the operator can chmod the file, move it out of .rea/, or
+            // delete it. `unreadable_segments` in the result is reserved
+            // for the never-reached future case where we can prove a file
+            // is genuinely out of scope (we'd need rotation start/end
+            // metadata for that — out of scope here).
+            throw new Error(`rea audit summary: cannot read ${filePath} (${errno ?? 'unknown errno'}). ` +
+                `An unreadable audit segment may contain in-window records, so the summary ` +
+                `would be silently incomplete. Fix permissions (e.g. \`chmod u+r ${filePath}\`), ` +
+                `or move the file out of \`.rea/\` if you no longer need it. The current ` +
+                `audit.jsonl is always required.`);
         }
+        actuallyScanned.push(filePath);
         for (const line of raw.split('\n')) {
             if (line.length === 0)
                 continue;
@@ -322,7 +359,12 @@ export async function computeAuditSummary(options = {}) {
         window_seconds: windowSeconds,
         window_start: windowStart !== null ? windowStart.toISOString() : null,
         window_end: windowEnd !== null ? windowEnd.toISOString() : null,
-        files_scanned: files,
+        // 0.42.0 codex round 4 P2: report only the files actually read.
+        // Unreadable rotations are reported separately under
+        // `unreadable_segments` so consumers can tell the difference
+        // between "scanned and empty" and "skipped because permissions".
+        files_scanned: actuallyScanned,
+        unreadable_segments: unreadableSegments,
         total_events: totalEvents,
         by_tool_name: byToolName,
         by_tier: byTier,
@@ -387,6 +429,13 @@ export function renderAuditSummary(result) {
             lines.push('(no audit files found — has `rea serve` ever run?)');
             lines.push('');
         }
+        // 0.42.0 codex round 4 P2: even in the zero-events early-return,
+        // surface unreadable segments so the operator sees the gap.
+        if (result.unreadable_segments.length > 0) {
+            lines.push(`unreadable rotated segments: ${String(result.unreadable_segments.length)} ` +
+                `(see stderr for paths; fix permissions and re-run to include them)`);
+            lines.push('');
+        }
         return lines.join('\n');
     }
     const total = result.total_events;
@@ -409,6 +458,13 @@ export function renderAuditSummary(result) {
             : 'unsampled (no records in window)';
     lines.push(`chain integrity: ${chainLabel}`);
     lines.push(`files scanned:   ${String(result.files_scanned.length)}`);
+    // 0.42.0 codex round 4 P2 (2026-05-16): surface unreadable rotated
+    // segments so an operator scanning the rendered summary doesn't
+    // miss a skipped archive that the JSON consumers can see.
+    if (result.unreadable_segments.length > 0) {
+        lines.push(`unreadable rotated segments: ${String(result.unreadable_segments.length)} ` +
+            `(see stderr for paths; fix permissions and re-run to include them)`);
+    }
     lines.push('');
     return lines.join('\n');
 }

package/dist/cli/doctor.d.ts

@@ -138,6 +138,26 @@ export interface PolicyReaderProbes {
      * ignore the argument; the default production probe uses it.
      */
     python3PyYamlReachable?: (baseDir: string) => boolean;
+    /**
+     * 0.42.0 codex round 5 P2 (2026-05-16) — execution probe for the
+     * python3 list-walker branch in `policy_reader_get_list`. That
+     * branch needs to spawn `python3 -c "..."` with `import json` from
+     * stdlib; PyYAML is irrelevant. The check is execution-based (not
+     * PATH-only) because a `python3` symlink can resolve on PATH but
+     * fail to start in the current sandbox (dangling pyenv/asdf stub,
+     * permission-denied interpreter, missing dynamic libs). A PATH-only
+     * check would let the doctor declare `warn` on a box where the
+     * shim will actually fall through to Tier 3 — masking a real
+     * enforcement gap for list-valued policy keys.
+     *
+     * The probe runs `python3 -c "import json; print('ok')"` with the
+     * same env scrub as the PyYAML probe (PYTHONPATH/PYTHONHOME/
+     * PYTHONSTARTUP unset, PYTHONSAFEPATH=1, sys.path scrubbed) so a
+     * malicious repo cannot plant a `./json.py` that shadows stdlib
+     * and falsely report `true` while the real loader (which scrubs)
+     * fails.
+     */
+    python3ListWalkerReachable?: (baseDir: string) => boolean;
     awkOnPath?: () => string | null;
     jqOnPath?: () => string | null;
 }
@@ -183,12 +203,12 @@ export declare function checkPolicyReaderTier2(baseDir: string, probes?: PolicyR
  * Practically always present (POSIX requirement).
  *
  * 0.40.0 charter item 2 — conditional verdict, refined by codex
- * round 1 P2:
+ * round 1 P2 (0.40.0) and round 2 P2 (0.42.0):
  *   - awk present                                            → `pass`
  *   - awk absent AND Tier 2 reachable                        → `warn`
  *     (Tier 2 implies python3, which is a list-walker)
  *   - awk absent AND Tier 1 reachable AND a list walker
- *     (jq OR python3) is on PATH                             → `warn`
+ *     (jq OR full Tier-2 reachable) is usable                → `warn`
  *   - awk absent AND Tier 1 reachable BUT no list walker     → `fail`
  *     (codex round 1 P2 — list-valued policy reads silently
  *     fail-closed even though scalar reads work, so the
@@ -207,9 +227,29 @@ export declare function checkPolicyReaderTier2(baseDir: string, probes?: PolicyR
  * functional box that has python3 + jq + the rea CLI all wired but
  * happens to lack awk.
  *
+ * List-iteration semantic (clarifying note for codex round 2 P2,
+ * 2026-05-16): `policy_reader_get_list` in
+ * `hooks/_lib/policy-reader.sh` walks the cached subtree JSON via
+ * `jq` OR `python3` (stdlib-only — `json` module, no PyYAML import).
+ * PyYAML is only needed for Tier 2 itself (YAML PARSING into JSON),
+ * NOT for iterating the already-parsed JSON arrays at list-read time.
+ *
+ * Codex round 5 P2 (2026-05-16): the "list walker" predicate uses
+ * `python3ListWalkerReachable` — an EXECUTION probe that actually
+ * spawns `python3 -c "import json"` — instead of `python3OnPath`. A
+ * PATH-only check passes for broken pyenv/asdf shims, dangling
+ * symlinks, and sandboxed environments where the interpreter cannot
+ * start; in those cases the shim's list-walker branch would actually
+ * fail and `blocked_paths`/`protected_writes` enforcement would
+ * silently break while doctor reported `warn`. The execution probe
+ * mirrors `defaultPython3PyYamlReachable` exactly but swaps the
+ * `import yaml` for `import json` so it's not gated on PyYAML
+ * availability (which is irrelevant to list iteration).
+ *
  * Takes `baseDir` so it can evaluate Tier 1's two-stage check (dist
- * present + CLI invokable) and Tier 2's reachability. Probes are
- * threaded through identically.
+ * present + CLI invokable), Tier 2's reachability, and the
+ * list-walker execution probe. All probes are threaded through
+ * identically.
  */
 export declare function checkPolicyReaderTier3(baseDir: string, probes?: PolicyReaderProbes): CheckResult;
 /**