@tekyzinc/gsd-t 4.0.19 → 4.0.21

package/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable changes to GSD-T are documented here. Updated with each release.
 
+## [4.0.21] - 2026-06-02 (M74 Adaptive Rate-Limit Throttle — patch)
+
+### Added — the scan throttle now self-lowers on a rate limit instead of failing
+
+M73's fixed 10-permit gate prevents the all-at-once stampede, but couldn't react if a rate limit still occurred. M74 makes the gate ADAPTIVE: on a rate-limit error (`isRateLimit` matches "temporarily limiting requests", 429, overloaded, etc.) `gatedAgent` lowers the global ceiling by 1 (10→9→8…, floor `MIN_CONCURRENT=4`), backs off (2s/4s/6s), and RETRIES the same agent (up to 4 attempts) — so a transient rate limit throttles the run down rather than failing it. After 8 clean completions the ceiling nudges back up toward 10. A non-rate-limit error is not retried (bubbles up normally).
+
+- `templates/workflows/gsd-t-scan.workflow.js`: `makeAdaptiveSemaphore` (shrinkable/recoverable ceiling, never yanks in-flight permits), rate-limit-aware `gatedAgent` with backoff+retry, `isRateLimit`, runtime `sleep`.
+- `test/m74-adaptive-throttle.test.js`: +5 tests (rate-limit detection incl. the real server message; floor/recovery bounds; lowered-ceiling stops granting until in-use drops; **all work completes despite 5 injected rate limits, zero failures**).
+
+Verified by 3 real sandbox diagnostics: `setTimeout` resolves in the sandbox (backoff is real); the adaptive gate lowered 10→5 under injected rate limits and completed all 12 items with 0 errors.
+
+Suite: 1302 pass / 0 fail / 4 skip — zero regressions.
+
+## [4.0.20] - 2026-06-02 (M73 Scan Concurrency Throttle — patch)
+
+### Fixed — unthrottled fan-out triggered an API rate limit that wiped a whole scan
+
+A v4.0.19 Hilo run produced an EMPTY register (0 findings) — not a workflow-logic bug: the scan fanned out 29 finders + their verifiers ALL AT ONCE (~58 concurrent Sonnet agents), hit a server-side API rate limit ("temporarily limiting requests · Rate limited"), and all 58 agents errored out empty. (M72's coverage-honesty correctly flagged it: 0/29, pointed to the prior 133-item register — no false "complete".)
+
+- `templates/workflows/gsd-t-scan.workflow.js`: added a single GLOBAL counting semaphore (`makeSemaphore`) of 10 permits. EVERY finder + verify agent acquires a permit before running and releases after (`gatedAgent`); freed permits are handed FIFO to the next waiter. All slices + findings still fan out at once, but total in-flight never exceeds 10 — a shared worker-pool: the instant any agent finishes, the next queued one starts, so throughput stays maximal at a safe ceiling. (Finders + verifiers are Sonnet, fine at 10; the lone Opus synthesis runs after, ungated.)
+- This replaces the initial batched approach (which left worker slots idle while a slice serialized its verifies) with a global queue — better throughput AND simpler.
+
+Verified by two real sandbox diagnostics: a 30-agent `mapLimit` probe and a 56-agent (8 finders × 6 verifies, fanned out at once) semaphore probe — both measured peakConcurrency = 10, never exceeded.
+
+Suite: 1297 pass / 0 fail / 4 skip — zero regressions.
+
 ## [4.0.19] - 2026-06-02 (M72 Scan Dropped-Slice Recovery + Coverage Honesty — patch)
 
 ### Fixed — a scan that under-covers no longer presents partial results as complete

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tekyzinc/gsd-t",
-  "version": "4.0.19",
+  "version": "4.0.21",
   "description": "GSD-T: Contract-Driven Development for Claude Code — 54 slash commands with headless-by-default workflow spawning, unattended supervisor relay with event stream, graph-powered code analysis, real-time agent dashboard, task telemetry, doc-ripple enforcement, backlog management, impact analysis, test sync, milestone archival, and PRD generation",
   "author": "Tekyz, Inc.",
   "license": "MIT",

package/templates/workflows/gsd-t-scan.workflow.js

@@ -281,11 +281,96 @@ function finderPrompt(slice) {
     `CRITICAL: you MUST return a JSON object matching the schema (slice + findings array) as your FINAL output — even if findings is empty. Do not end without the structured result.`,
   ].filter(Boolean).join("\n");
 }
+// M73: GLOBAL CONCURRENCY GATE (shared-worker-pool model). The v4.0.19 Hilo run
+// fanned out 29 finders + their verifiers ALL AT ONCE → ~58 concurrent Sonnet agents
+// → server-side API rate limit → all errored empty → 0 findings. Rather than batch
+// per-slice (which leaves slots idle while a slice serializes its verifies), use ONE
+// global semaphore of MAX_CONCURRENT permits that EVERY agent call (finder OR verify,
+// from any slice) must acquire. Work fans out naturally; the gate alone enforces the
+// cap. The instant any agent finishes, the next queued one starts → always ≈10 in
+// flight while work remains = max throughput at a safe ceiling. (All gated agents are
+// Sonnet; the lone Opus synthesis runs after, ungated.)
+const MAX_CONCURRENT = 10;
+
+// Minimal counting semaphore: acquire() resolves when a permit is free; release()
+// hands the permit to the next waiter (FIFO).
+// M74: ADAPTIVE semaphore — the ceiling can shrink (on a rate limit) and recover.
+// `inUse` = permits currently held; a permit is grantable only while inUse < ceiling.
+// Lowering the ceiling doesn't yank in-flight permits; it just stops granting new
+// ones until enough release that inUse drops below the new ceiling.
+const MIN_CONCURRENT = 4; // never throttle below this — keeps the run moving
+function makeAdaptiveSemaphore(initial) {
+  let ceiling = initial;
+  let inUse = 0;
+  const waiters = [];
+  function pump() {
+    while (inUse < ceiling && waiters.length) { inUse++; waiters.shift()(); }
+  }
+  return {
+    async acquire() {
+      if (inUse < ceiling) { inUse++; return; }
+      await new Promise((res) => waiters.push(res)); // pump() increments inUse on grant
+    },
+    release() { inUse--; pump(); },
+    lower() { // shrink ceiling by 1 on a rate limit (floor at MIN_CONCURRENT)
+      if (ceiling > MIN_CONCURRENT) { ceiling--; return true; }
+      return false;
+    },
+    raise() { // gentle recovery toward the initial ceiling after sustained success
+      if (ceiling < initial) { ceiling++; pump(); return true; }
+      return false;
+    },
+    get ceiling() { return ceiling; },
+  };
+}
+const gate = makeAdaptiveSemaphore(MAX_CONCURRENT);
+
+function isRateLimit(err) {
+  const s = String((err && (err.message || err)) || "").toLowerCase();
+  return /rate.?limit|temporarily limiting|429|overloaded|too many requests|capacity/.test(s);
+}
+
+// M74: gatedAgent now reacts to rate limits in REAL TIME — on a rate-limit error it
+// lowers the global ceiling (10→9→8…, floor MIN_CONCURRENT), backs off, and retries
+// the SAME agent (up to a few times) instead of letting it fail. After a streak of
+// clean completions it nudges the ceiling back up. This means a transient rate limit
+// throttles the run down automatically rather than failing it (the v4.0.19 wipeout).
+let _cleanStreak = 0;
+async function gatedAgent(prompt, opts) {
+  await gate.acquire();
+  try {
+    for (let attempt = 1; attempt <= 4; attempt++) {
+      try {
+        const r = await agent(prompt, opts);
+        // success: nudge the ceiling back up after every 8 clean completions.
+        if (++_cleanStreak >= 8) { _cleanStreak = 0; if (gate.raise()) log(`↑ throttle recovered to ${gate.ceiling} concurrent (clean streak)`); }
+        return r;
+      } catch (e) {
+        if (isRateLimit(e) && attempt < 4) {
+          _cleanStreak = 0;
+          const lowered = gate.lower();
+          const backoffMs = 2000 * attempt; // 2s, 4s, 6s
+          log(`⚠ rate limit hit — ${lowered ? `throttling down to ${gate.ceiling} concurrent` : `already at floor ${gate.ceiling}`}; backing off ${backoffMs}ms then retry ${attempt + 1}/4 (${(opts && opts.label) || "agent"})`);
+          await sleep(backoffMs);
+          continue;
+        }
+        throw e; // non-rate-limit error, or out of retries → bubble up
+      }
+    }
+  } finally {
+    gate.release();
+  }
+}
+// Backoff sleep. setTimeout is available in the Workflow sandbox (verified by a
+// real sandbox probe, run wf_7e90a974 — NOT assumed; Date.now/Math.random ARE banned
+// but setTimeout is not). Used only for rate-limit backoff between retries.
+function sleep(ms) { return new Promise((res) => setTimeout(res, ms)); }
+
 async function runFinder(slice) {
   // up to 2 attempts; a null/invalid (non-array findings) result counts as a drop.
   for (let attempt = 1; attempt <= 2; attempt++) {
     try {
-      const r = await agent(finderPrompt(slice), {
+      const r = await gatedAgent(finderPrompt(slice), {
         label: attempt === 1 ? `find:${slice.key}` : `find:${slice.key} (retry)`,
         phase: "Deep Scan", schema: FINDER_SCHEMA, model: "sonnet",
       });
@@ -298,40 +383,43 @@ async function runFinder(slice) {
   return null; // both attempts failed → dropped slice
 }
 
-const sliceResults = await pipeline(
-  slices,
-  (slice) => runFinder(slice),
-  async (finderResult, originalItem) => {
-    const slice = originalItem || {};
-    const sliceKey = slice.key || (finderResult && finderResult.slice) || "unknown-slice";
-    // M72: distinguish a FAILED finder (null after retries) from a genuinely-clean slice.
-    if (!finderResult || !Array.isArray(finderResult.findings)) {
-      return { slice: sliceKey, findings: [], failed: true };
-    }
-    if (verifyMode === "none" || finderResult.findings.length === 0) {
-      return { slice: sliceKey, findings: finderResult.findings || [], failed: false };
-    }
-    const verified = await parallel(
-      finderResult.findings.map((f) => async () => {
-        try {
-          const v = await agent(
-            [
-              `You are a VERIFIER for one tech-debt finding in \`${projectDir}\`. Confirm it against the ACTUAL code (open the referenced files with Read) — do not trust the finder.`,
-              `Finding: ${JSON.stringify(f)}`,
-              `confirmed=true only if the defect genuinely exists. If misread → verdict="false-positive". If real but wrong severity → set correctedSeverity. If real but underspecified → verdict="needs-detail" (kept). Return JSON per the schema.`,
-            ].join("\n"),
-            { label: `verify:${sliceKey}`, phase: "Deep Scan", schema: VERIFY_SCHEMA, model: "sonnet" }
-          );
-          if (!v || v.verdict === "false-positive" || v.confirmed === false) return null;
-          return { ...f, severity: v.correctedSeverity || f.severity, _verify: v.verdict };
-        } catch (e) {
-          return { ...f, _verify: "verify-errored" };
-        }
-      })
-    );
-    return { slice: sliceKey, findings: verified.filter(Boolean), failed: false };
+async function scanSlice(slice) {
+  const sliceKey = slice.key || "unknown-slice";
+  const finderResult = await runFinder(slice);
+  // M72: distinguish a FAILED finder (null after retries) from a genuinely-clean slice.
+  if (!finderResult || !Array.isArray(finderResult.findings)) {
+    return { slice: sliceKey, findings: [], failed: true };
   }
-);
+  if (verifyMode === "none" || finderResult.findings.length === 0) {
+    return { slice: sliceKey, findings: finderResult.findings || [], failed: false };
+  }
+  // Fan out ALL verifies for this slice — the global gate (not a per-slice limit)
+  // bounds total in-flight, so this is safe AND keeps every worker slot busy.
+  const verified = await parallel(
+    finderResult.findings.map((f) => async () => {
+      try {
+        const v = await gatedAgent(
+          [
+            `You are a VERIFIER for one tech-debt finding in \`${projectDir}\`. Confirm it against the ACTUAL code (open the referenced files with Read) — do not trust the finder.`,
+            `Finding: ${JSON.stringify(f)}`,
+            `confirmed=true only if the defect genuinely exists. If misread → verdict="false-positive". If real but wrong severity → set correctedSeverity. If real but underspecified → verdict="needs-detail" (kept). Return JSON per the schema.`,
+          ].join("\n"),
+          { label: `verify:${sliceKey}`, phase: "Deep Scan", schema: VERIFY_SCHEMA, model: "sonnet" }
+        );
+        if (!v || v.verdict === "false-positive" || v.confirmed === false) return null;
+        return { ...f, severity: v.correctedSeverity || f.severity, _verify: v.verdict };
+      } catch (e) {
+        return { ...f, _verify: "verify-errored" };
+      }
+    })
+  );
+  return { slice: sliceKey, findings: verified.filter(Boolean), failed: false };
+}
+
+// Fan out ALL slices at once — every finder + verify acquires the shared gate, so
+// total in-flight never exceeds MAX_CONCURRENT regardless of slice/finding counts.
+log(`deep scan: ${slices.length} slices via a shared ${MAX_CONCURRENT}-permit gate (Sonnet finders+verifiers; max throughput at a safe ceiling)`);
+const sliceResults = await parallel(slices.map((slice) => () => scanSlice(slice)));
 
 // M72: coverage accounting — a dropped pipeline result (null) OR a failed:true slice
 // is a COVERAGE GAP. Surface it deterministically; never present partial as complete.