@tekyzinc/gsd-t 3.18.13 → 3.19.0

package/CHANGELOG.md

@@ -2,6 +2,120 @@
 
 All notable changes to GSD-T are documented here. Updated with each release.
 
+## [3.19.00] - 2026-04-23
+
+### Added — M46 Milestone: Unattended Iter-Parallel + Worker Fan-Out Completion
+
+Closes the two biggest gaps from the 2026-04-23 five-surface parallelism audit: (2A) unattended multi-iteration parallelism and (2B) worker-side sub-fan-out.
+
+**D1 — Iteration-parallel supervisor scaffold (`bin/gsd-t-unattended.cjs`):**
+- `_runOneIter(state, opts) → IterResult` — extracted from the while-loop body at line 969 (68-line delta, zero behavior change when called sequentially)
+- `_computeIterBatchSize(state, opts) → number` — mode-safety gates: `verify-needed → 1`, `milestone-boundary → 1`, `complete-milestone → 1`. Production default returns 1 (serial) unless caller passes `opts.maxIterParallel` — opt-in gate on the iter-parallel engagement pending full concurrent-state-safety rewrite (backlog #24).
+- `_runIterParallel(state, opts, iterFn, batchSize) → Promise<IterResult[]>` — uses `Promise.allSettled` for per-iter error isolation; one rejection does not cancel siblings.
+- `_reconcile(state, results)` — deduped union on `completedTasks`, OR on `verifyNeeded`, append-only `artifacts`, last-writer-wins `status`, writes `lastBatch` metadata. **Does NOT advance `state.iter`** — that invariant is owned by the main while loop via `_runOneIter`.
+- `module.exports.__test__` exposes all four helpers to unit tests.
+
+**D2 — Worker sub-dispatch production path (`bin/gsd-t-worker-dispatch.cjs`):**
+- `dispatchWorkerTasks({projectDir, parentSessionId, tasks, maxParallel=4}) → Promise<{parallel, taskResults, wallClockMs, reason}>` — deterministic probe + delegation to `bin/gsd-t-parallel.cjs::runDispatch` when the task graph is file-disjoint and `tasks.length > 1`.
+- Short-circuits with reason strings: `no-tasks`, `single-task`, `file-overlap`, `dispatch-error:*`, `dispatched`.
+- CLI entry: `node bin/gsd-t-worker-dispatch.cjs --parent-session <id> --tasks <path> --max-parallel <n>` — emits JSON result to stdout.
+- `commands/gsd-t-resume.md` Step 0 `GSD_T_UNATTENDED_WORKER=1` branch gains an additive sub-dispatch block (no deletion from existing prose).
+- `bin/spawn-plan-writer.cjs` kind enum extended with `unattended-worker-sub`.
+
+### Contracts
+
+- **`.gsd-t/contracts/iter-parallel-contract.md`** — NEW v1.0.0. Batch semantics, `IterResult` shape, reconciliation rule, stop-check batch-boundary invariant (stop-check is never polled mid-batch).
+- **`.gsd-t/contracts/headless-default-contract.md`** — v2.0.0 → v2.1.0 (additive §Worker Sub-Dispatch documenting the `unattended-worker-sub` kind and the resume-path hand-off).
+
+### Measurements — both proofs passed thresholds
+
+- **D1 iter-proof (`bin/m46-iter-proof.cjs`)**: 10-iter synthetic workload, 200ms work per iter, batch=4 vs serial. Result: `T_serial=2022.6ms`, `T_par=602.9ms`, **speedup=3.35×**, `T_par/T_serial=0.298` — passes thresholds `speedup ≥ 3.0` and `T_par/T_serial ≤ 0.35`.
+- **D2 worker-proof (`bin/m46-worker-proof.cjs`)**: 6-task file-disjoint synthetic workload, serial vs fan-out via `runDispatch`. Result: `T_serial=12134ms`, `T_par=2034ms`, **speedup=5.96×** — passes threshold `speedup ≥ 2.5`.
+- Reports: `.gsd-t/metrics/m46-iter-proof.json` + `.gsd-t/metrics/m46-worker-proof.json`.
+
+### Tests
+
+- **`test/m46-d1-iter-parallel.test.js`** — 12 unit tests covering serial fallback, 3-way parallel batch concurrency (<200ms for three 100ms iters), mode-safety gates (verify-needed / complete-milestone / milestoneBoundary), error isolation (one rejection, two siblings succeed), stop-check batch-boundary invariant, and `_reconcile` semantics.
+- **`test/m46-d2-worker-subdispatch.test.js`** — 6 unit tests covering disjoint fan-out, single-task short-circuit, file-overlap detection, dispatcher error surfacing, and CLI JSON-stdout contract.
+- Full suite: **1946/1946 pass**, zero regressions (M43 heartbeat-watchdog + M44 planner-wire + M45 suites all green).
+
+### Regression caught and fixed mid-milestone
+
+- **Double-increment of `state.iter`** between `_reconcile` and `_runOneIter` tripped 4 tests (m43-heartbeat-watchdog `staleHeartbeat res → exitCode 125`, m44-wire-unattended-to-planner iter-count / fallback / sequential). Root cause: `_reconcile` was advancing `state.iter` by `results.length` while `_runOneIter` already advanced it by 1. Fix: `_reconcile` leaves `state.iter` untouched (main loop owns the invariant); two m46-d1 tests updated to match new contract.
+
+### Follow-up backlog
+
+- **#24 — Dynamic work-stealing rewrite** (full concurrent-safe state isolation for `_runIterParallel` >1 batch). Covers the `state.iter` / heartbeat / writeState shared-mutation issue that keeps the iter-parallel engagement opt-in rather than default-on.
+- **D2-T11 integration smoke** deferred — unit tests + proof harness cover the surface.
+
+## [3.18.18] - 2026-04-23
+
+### Added — Model-aware worker spawn in `runDispatch`
+
+- **`bin/gsd-t-parallel.cjs::runDispatch`**: fan-out workers now default to `claude-sonnet-4-6` via new constant `DEFAULT_WORKER_MODEL` (was: inherit the orchestrator's `ANTHROPIC_MODEL`, which is `claude-opus-4-7` in this user's global settings). Caller overrides via `opts.workerModel`: alias strings (`"opus"` / `"sonnet"` / `"haiku"`) resolve to full model IDs; explicit full IDs pass through; `workerModel: false` opts out of the override entirely and inherits parent. **Why**: the 2026-04-23 M46 Wave 1 dispatch rate-limited all 8 Opus workers (Max 20x subscription concurrent-session throttle). Sonnet lives in a separate rate bucket, so orchestrator Opus + worker Sonnet lifts the concurrency ceiling. Per-task Opus opt-in via `[opus]` marker on tasks.md lines is future work (surfaces in planner metadata).
+- **`bin/headless-auto-spawn.cjs::autoSpawnHeadless`**: accepts `workerModel?: string` and sets `ANTHROPIC_MODEL` in the child env after the caller's `envOverride` merge (so caller always wins if they explicitly set `ANTHROPIC_MODEL` in `env`).
+
+### Added — Spawn stagger to avoid burst spikes
+
+- **`bin/gsd-t-parallel.cjs::runDispatch`**: new `opts.spawnStaggerMs` (default **3000** ms) delays each spawn after the first. Implemented via `Atomics.wait` on a `SharedArrayBuffer` so the blocking wait releases the CPU (no spin loop). 2026-04-23 observation: 8 concurrent `claude -p` spawned within 700 ms → all 429 rate-limited; 3 s stagger avoids the burst. Set `spawnStaggerMs: 0` for pre-v3.18.18 behavior.
+
+### Added — Cache-warming probe (opt-in)
+
+- **`bin/gsd-t-parallel.cjs::_runCacheWarmProbe`** + opt-in flag `opts.cacheWarm` / env `GSD_T_CACHE_WARM=1`. Before fan-out, fires a short `claude -p` that reads CLAUDE.md + progress.md + top-level contracts using the **same model** the workers will run, so Anthropic's 5-minute prompt cache is populated and the workers' identical initial reads return cache-read tokens (free for ITPM budget, lower rate-limit pressure). Probe is synchronous (workers land inside the warm window, not racing it), 60 s timeout, failure does not block fan-out. Dependency-injection hook `opts.cacheWarmProbeImpl` for tests. Gated behind opt-in until backlog #23 (mitmproxy header instrumentation) measures the actual delta; flips to default-on if measurement confirms the ITPM savings are real.
+
+### Tests
+
+- **`test/m44-run-dispatch.test.js`**: 4 new tests for model selection (default Sonnet / alias resolution / explicit opt-out / stagger timing) + 3 new tests for cache-warming (opt-in gating / probe model matches workers / probe failure does not block fan-out). Full suite **2023/2023** pass (baseline 2016 + 7 new).
+
+### Incident — 2026-04-23 M46 Wave 1 rate-limit
+
+- Root cause: all 8 headless workers inherited `ANTHROPIC_MODEL=claude-opus-4-7` from `~/.claude/settings.json` (this user runs Max 20x on Opus globally) and spawned in a 700 ms burst. Max subscription's concurrent-session throttle fired ~1 s into each worker's first tool call. Anthropic Console dashboards showed flat-zero API usage — confirmed the throttle is subscription-side, not API-key-side. Mitigations shipped in this release (model-mix + stagger + opt-in cache-warm) + scoped backlog items #22 (coord-gate runtime coordination) and #23 (mitmproxy header instrumentation for calibration).
+
+## [3.18.17] - 2026-04-23
+
+### Fixed — `npm test` picks up `worker-sim.js` fixture
+
+- **`test/fixtures/m44-proof/worker-sim.js`** was being globbed by `node --test`'s default test-directory matcher (anything under `test/` with a `.js` extension), and failed because the fixture requires `OUT_DIR` to be set. The fixture now exits `0` when env vars are absent instead of `2` — it's a worker fixture, not a test. Full suite back to 2016/2016 green. Required to unblock the v3.18.16 publish.
+
+## [3.18.16] - 2026-04-23
+
+### Added — Proof measurement `--visualize` flag
+
+- **`bin/m44-proof-measure.cjs --visualize`** writes synthetic spawn-plan files into the project's `.gsd-t/spawns/` directory as each simulated worker launches and calls `markTaskDone` + `markSpawnEnded` when they finish, so the M44 D9 parallelism panel (`scripts/gsd-t-transcript.html`, endpoint `/api/parallelism`) renders the fan-out live. Off by default — the unflagged measurement still writes spawn-plans only under the temp fixture root. Enables end-to-end visualizer observation of the dispatcher without burning API tokens.
+- **Reproducibility**: three consecutive 20s-worker runs (13:08, 13:09, 13:27 local) produced identical `T_par / T_seq ≈ 0.251`, `speedup ≈ 3.98×`, `parallelism_factor ≈ 3.97`, `parallelism_factor_mode: "live"` with `activeWorkers: 4` for the full 20s parallel window. Panel transitions IDLE → live → IDLE confirmed by `/api/parallelism` polling.
+
+## [Unreleased] — v3.19.00 pending
+
+### Measured — Dispatcher T/2 criterion (backlog #15, leg 1 of 2)
+
+- **`bin/m44-proof-measure.cjs`** runs a falsifiable measurement of the v3.19.00 parallel dispatcher using a synthetic spawner (`test/fixtures/m44-proof/worker-sim.js`) injected into `runDispatch` via `opts.spawnHeadlessImpl`. Fixture (`test/fixtures/m44-proof/fixture.tasks.md`): 4 file-disjoint tasks with explicit `- touches:` sub-bullets (D5 disjointness requirement). Each worker sleeps `WORKER_DURATION_MS` (default 8000ms) then writes a JSON `.done` marker — zero LLM calls, zero network, zero side effects outside `OUT_DIR`. **Result**: T_par = **8111.1 ms**, T_seq = **32146.1 ms**, speedup **3.96×**, parallelism_factor **3.95** (ideal = 4), dispatch overhead **8.2 ms**. Criterion `T_par ≤ T_seq/2` → **MET ✓**. Report JSON at `.gsd-t/m44-proof-report.json`. This proves the dispatcher fans out concurrently; it does NOT prove N Claude workers produce correct code in T/N (a separate experiment, deferred to a follow-up backlog item).
+
+### Pending — Zero-compaction criterion (backlog #15, leg 2 of 2)
+
+- **NOT YET MEASURED.** Requires an unattended supervisor run over a workload that historically would have triggered mid-run `/compact`, producing zero `type:"compaction_post_spawn"` rows in `.gsd-t/metrics/compactions.jsonl` under the fully-wired v3.19.00 surface (`ca20477` supervisor→planner + `799a8af` single-instrument + `19eb3eb` D9 observability panel). Existing 81 rows in the compactions log contain 0 `compaction_post_spawn` entries, but the only post-19eb3eb unattended state predates the D9 landing and is therefore not a valid sample.
+- **`v3.19.00` tag deferred** until the zero-compaction leg completes. Per user standing directive `feedback_measure_dont_claim.md`: "milestones with measurable success criteria are not complete until measurement is run AND reported."
+
+## [3.18.15] - 2026-04-23
+
+### Fixed — Supervisor false-failed marker (M45 follow-up)
+
+- **`bin/headless-exit-codes.cjs::mapHeadlessExitCode` polarity discipline** — the pre-fix matcher did `lower.includes("tests failed")` / `"verification failed"` / `"context budget exceeded"`, which fired on free-form narration like `"0 tests failed"`, `"no tests failed"`, and quoted phrases inside worker output. During the M45 run the worker's clean output contained `"tests failed"` 6× in healthy prose, flipping its mapped exit code 0 → 1 and causing the supervisor to finalize `status=failed` despite the milestone having been completed and archived. The matchers now require either a non-zero numeric count (`/([1-9]\d*)\s+(?:tests?|specs?|assertions?|examples?|suites?)\s+failed\b/i`), a structured terminal marker (`/^FAIL[:\s]/m`, Jest-style `/^Tests:\s+\d+\s+failed/m`), or a line-boundary / sentence-start anchor for free-form verification/context-budget phrases. 27 new polarity regression tests in `test/m45-fix-headless-exit-polarity.test.js`; all existing `headless.test.js` assertions preserved.
+- **`commands/gsd-t-unattended-watch.md` Step 3 reconciliation** — when the supervisor PID file is absent AND `state.status=failed` AND a fresh milestone archive exists under `.gsd-t/milestones/` (mtime ≥ supervisor `startedAt`), the watch tick now renders a reconciled success report noting the archive as the source of truth instead of the contradictory ✅-cleanly-finalized + failed-status block the previous logic would emit. Raw final report preserved for genuinely failed runs with no archive.
+
+## [3.18.14] - 2026-04-23
+
+### Added — M45 Conversation-Stream Observability
+
+- **Orchestrator dialog visible in the transcript viewer** — new hook `scripts/hooks/gsd-t-conversation-capture.js` (SessionStart + UserPromptSubmit + Stop + opt-in PostToolUse via `GSD_T_CAPTURE_TOOL_USES=1`) writes typed NDJSON frames to `.gsd-t/transcripts/in-session-{sessionId}.ndjson` for every human↔Claude turn. The visualizer's left rail now lists those entries with a `💬 conversation` badge alongside the `▶ spawn` entries, so users can watch their own dialog in the same surface as spawned work.
+- **Compact marker fallback target-selection** — `scripts/gsd-t-compact-detector.js::findActiveTranscript` now prefers a fresh spawn NDJSON when one has been modified within 30s, and falls back to the most recent `in-session-*.ndjson` otherwise. Mid-conversation `/compact` events land in the correct transcript instead of a random stale spawn file.
+- **New contract** `.gsd-t/contracts/conversation-capture-contract.md` v1.0.0 — documents the frame schema (`session_start` / `user_turn` / `assistant_turn` / `tool_use`), file-naming (`in-session-` prefix as the viewer + compact-detector discriminator), hook entry points, session-id source + fallback, and 16 KB content cap.
+- **Settings.json hook wiring documented** — `templates/CLAUDE-global.md` gains an "In-Session Conversation Capture (M45 D2)" section so users who install/update pick up the hook alongside the existing in-session token-usage hook.
+
+### Fixed — M45 D1 Viewer Route
+
+- **`GET /transcripts` now serves the real transcript viewer** — reverts the standalone `renderTranscriptsHtml` index page shipped in v3.18.13. The route now reads `scripts/gsd-t-transcript.html` with `__SPAWN_ID__` → `""`, giving users the same left-rail + main + right-panel surface they get at `/transcript/:id`. JSON back-compat preserved: `Accept: application/json` and `*/*` continue to return `{spawns: [...]}`.
+- **Session-id path-separator sanitization (Red Team BUG-1)** — `_resolveSessionId` in the new conversation-capture hook now rejects session_ids containing `/`, `\`, `\0`, or `..` and falls through to the pid-hash fallback. Prior behavior let `session_id="a/../b"` lexically collapse via `path.join` to produce `transcripts/b.ndjson` without the `in-session-` prefix, breaking the filename-prefix discriminator contract with the viewer + compact-detector.
+
 ## [3.18.13] - 2026-04-23
 
 ### Fixed

package/bin/gsd-t-parallel-probe.cjs

@@ -0,0 +1,132 @@
+"use strict";
+
+/**
+ * gsd-t-parallel-probe — M44 D9 Step 3
+ *
+ * Deterministic in-session probe that runs the parallel planner in dry-run mode
+ * and emits a single JSON line to stdout. Command files (execute / quick /
+ * debug / integrate) invoke this as a bash shim to get a mechanical answer —
+ * replacing LLM prose judgment with a JSON shape that branching logic can read.
+ *
+ * Per user memory `feedback_deterministic_orchestration.md`:
+ *   "prompt-based blocking doesn't work; use JS orchestrators for gates/waits"
+ *
+ * Contract: wave-join-contract.md v1.1.0; unattended-supervisor-contract.md v1.5.0
+ *
+ * Output shape (stdout, one line):
+ *   {"workerCount":N,"parallelTasks":["M44-D9-T1",...],"mode":"in-session",
+ *    "reducedCount":null|N,"warnings":[...],"ok":true}
+ *
+ * On unexpected error (planner throw, missing repo state, etc.), emits a safe
+ * fallback shape so shell callers never have to parse stderr:
+ *   {"workerCount":1,"parallelTasks":[],"mode":"in-session","ok":false,"error":"..."}
+ *
+ * Exit code 0 on both success and safe-fallback. This is intentional: the shim
+ * is a decision probe, not a command; non-zero would force command files to
+ * defensively wrap it and add LLM noise.
+ *
+ * Hard rules:
+ *   - Zero external runtime deps
+ *   - Never writes to stderr by default (shell shim relies on quiet)
+ *   - Never writes to `.gsd-t/events/*` — that's runParallel's job
+ */
+
+const path = require("node:path");
+
+function parseArgv(argv) {
+  const out = { milestone: null, domain: null, mode: null };
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--milestone" && argv[i + 1]) {
+      out.milestone = argv[++i];
+    } else if (a === "--domain" && argv[i + 1]) {
+      out.domain = argv[++i];
+    } else if (a === "--mode" && argv[i + 1]) {
+      out.mode = argv[++i];
+    } else if (a === "--help" || a === "-h") {
+      out.help = true;
+    }
+  }
+  return out;
+}
+
+const HELP = `gsd-t-parallel-probe — deterministic planner probe (JSON out)
+
+Usage:
+  node bin/gsd-t-parallel-probe.cjs [--milestone Mxx] [--domain name] [--mode in-session|unattended]
+
+Output: one JSON line to stdout with keys workerCount, parallelTasks, mode,
+reducedCount, warnings, ok. Exit 0 always. Use with in-session command files
+(execute, quick, debug, integrate) to replace prose-based parallel dispatch
+decisions with a mechanical branch on workerCount.
+`;
+
+function probe(argv, env) {
+  const args = parseArgv(argv || []);
+  if (args.help) {
+    process.stdout.write(HELP);
+    return 0;
+  }
+
+  let runParallel;
+  try {
+    ({ runParallel } = require(path.join(__dirname, "gsd-t-parallel.cjs")));
+  } catch (e) {
+    process.stdout.write(
+      JSON.stringify({
+        workerCount: 1,
+        parallelTasks: [],
+        mode: args.mode || "in-session",
+        reducedCount: null,
+        warnings: [],
+        ok: false,
+        error: `planner_load:${(e && e.message) || "unknown"}`,
+      }) + "\n",
+    );
+    return 0;
+  }
+
+  let result;
+  try {
+    result = runParallel({
+      projectDir: process.cwd(),
+      mode: args.mode || undefined,
+      milestone: args.milestone || undefined,
+      domain: args.domain || undefined,
+      dryRun: true,
+      env: env || process.env,
+    });
+  } catch (e) {
+    process.stdout.write(
+      JSON.stringify({
+        workerCount: 1,
+        parallelTasks: [],
+        mode: args.mode || "in-session",
+        reducedCount: null,
+        warnings: [],
+        ok: false,
+        error: `planner_error:${(e && e.message) || "unknown"}`,
+      }) + "\n",
+    );
+    return 0;
+  }
+
+  process.stdout.write(
+    JSON.stringify({
+      workerCount: Number(result.workerCount) || 0,
+      parallelTasks: Array.isArray(result.parallelTasks) ? result.parallelTasks : [],
+      mode: result.mode || args.mode || "in-session",
+      reducedCount: typeof result.reducedCount === "number" ? result.reducedCount : null,
+      warnings: Array.isArray(result.warnings) ? result.warnings : [],
+      ok: true,
+    }) + "\n",
+  );
+  return 0;
+}
+
+module.exports = { probe, _parseArgv: parseArgv, _HELP: HELP };
+
+if (require.main === module) {
+  const code = probe(process.argv.slice(2), process.env);
+  process.exit(code);
+}