@tekyzinc/gsd-t 4.0.18 → 4.0.20

package/CHANGELOG.md

@@ -2,6 +2,36 @@
 
 All notable changes to GSD-T are documented here. Updated with each release.
 
+## [4.0.20] - 2026-06-02 (M73 Scan Concurrency Throttle — patch)
+
+### Fixed — unthrottled fan-out triggered an API rate limit that wiped a whole scan
+
+A v4.0.19 Hilo run produced an EMPTY register (0 findings) — not a workflow-logic bug: the scan fanned out 29 finders + their verifiers ALL AT ONCE (~58 concurrent Sonnet agents), hit a server-side API rate limit ("temporarily limiting requests · Rate limited"), and all 58 agents errored out empty. (M72's coverage-honesty correctly flagged it: 0/29, pointed to the prior 133-item register — no false "complete".)
+
+- `templates/workflows/gsd-t-scan.workflow.js`: added a single GLOBAL counting semaphore (`makeSemaphore`) of 10 permits. EVERY finder + verify agent acquires a permit before running and releases after (`gatedAgent`); freed permits are handed FIFO to the next waiter. All slices + findings still fan out at once, but total in-flight never exceeds 10 — a shared worker-pool: the instant any agent finishes, the next queued one starts, so throughput stays maximal at a safe ceiling. (Finders + verifiers are Sonnet, fine at 10; the lone Opus synthesis runs after, ungated.)
+- This replaces the initial batched approach (which left worker slots idle while a slice serialized its verifies) with a global queue — better throughput AND simpler.
+
+Verified by two real sandbox diagnostics: a 30-agent `mapLimit` probe and a 56-agent (8 finders × 6 verifies, fanned out at once) semaphore probe — both measured peakConcurrency = 10, never exceeded.
+
+Suite: 1297 pass / 0 fail / 4 skip — zero regressions.
+
+## [4.0.19] - 2026-06-02 (M72 Scan Dropped-Slice Recovery + Coverage Honesty — patch)
+
+### Fixed — a scan that under-covers no longer presents partial results as complete
+
+The first real Hilo run (v4.0.18) surfaced 133 findings but 7 of 19 deep-finder slices had silently FAILED — they returned no schema-valid output (the runtime nudged them twice then dropped them), and the workflow treated a dropped slice identically to a genuinely-clean one (`findings: []`). The register confidently presented ~⅔ coverage as complete. For a quality tool, silent partial coverage is worse than a shallow scan.
+
+- `templates/workflows/gsd-t-scan.workflow.js`:
+  - **Retry**: each finder is now retried once on a null/invalid result (`runFinder`).
+  - **Detect, don't conflate**: a slice that still fails is flagged `failed:true` — never merged with an empty-but-successful slice. Coverage accounting computes `failedSlices`, `slicesSucceeded`, `coverageComplete` deterministically; dropped slices' (absent) findings are excluded from the count.
+  - **Surface loudly**: synthesis is REQUIRED (not relying on the agent's notice) to put a "⚠ PARTIAL COVERAGE — N of M areas not scanned" banner at the top of the register and list the un-scanned slices. The workflow return status downgrades to `"complete-partial-coverage"` (not `"complete"`) and includes `slicesFailed[]`.
+  - **Synthesis robustness**: instructed to write the register INCREMENTALLY (header+summary, then append each severity section) so a multi-hundred-item register can't stall on one giant Write; synthesis-input truncation raised 200KB→500KB.
+- `test/m72-coverage-accounting.test.js`: +4 tests (dropped slice excluded + flagged; clean slice not a gap; null pipeline result = failed; full coverage = complete). Coverage logic also verified by a real sandbox diagnostic run.
+
+Effect: a scan with any failed slice is clearly marked incomplete; resuming the run re-scans only the failed slices (cached successes are reused).
+
+Suite: 1297 pass / 0 fail / 4 skip — zero regressions.
+
 ## [4.0.18] - 2026-06-02 (M71 Runtime-Native Scan Workflow — patch)
 
 ### Fixed — the scan Workflow now actually RUNS in the Workflow sandbox (it never did)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tekyzinc/gsd-t",
-  "version": "4.0.18",
+  "version": "4.0.20",
   "description": "GSD-T: Contract-Driven Development for Claude Code — 54 slash commands with headless-by-default workflow spawning, unattended supervisor relay with event stream, graph-powered code analysis, real-time agent dashboard, task telemetry, doc-ripple enforcement, backlog management, impact analysis, test sync, milestone archival, and PRD generation",
   "author": "Tekyz, Inc.",
   "license": "MIT",

package/templates/workflows/gsd-t-scan.workflow.js

@@ -261,52 +261,132 @@ const deep = budget && budget.total && budget.total > 300000 ? "MAXIMUM" : "thor
 
 // Deep scan — pipeline: per-slice deep finder → single verify (no barrier).
 phase("Deep Scan");
-const sliceResults = await pipeline(
-  slices,
-  (slice) => agent(
-    [
-      `⛔ Scan ONLY files under the absolute project path \`${projectDir}\`. \`cd ${projectDir}\` first; never read outside this tree.`,
-      `You are a DEEP tech-debt finder for ONE slice of a scan of \`${projectDir}\`: \`${slice.key}\` (dimension: ${slice.dimension}).`,
-      `Owned paths (relative to \`${projectDir}\`): ${JSON.stringify(slice.paths)}.`,
-      slice.why ? `Why this slice matters: ${slice.why}` : ``,
-      ``,
-      `MANDATE: ENUMERATE, do NOT sample. Read EVERY file under your owned paths (use Read/Grep). The legacy scan failed because one agent sampled ~5 issues across the whole repo and stopped — you own only this slice, so go to the bottom of it.`,
-      `Depth = ${deep}. "thorough" = every file, every non-trivial real defect (high+medium confidence). "MAXIMUM" = also lower-confidence/speculative items worth review.`,
-      `Surface: bugs, security holes, missing validation, broken invariants, race conditions, dead/duplicated code, N+1s, untested critical paths, contract drift, domain-specific correctness (money math, state-machine gaps, timezone bugs, idempotency holes).`,
-      `For each finding: title, severity (CRITICAL/HIGH/MEDIUM/LOW), human area label, concrete file:line refs, detail, impact, remediation, honest confidence. If a substantial slice yields only 1-2 findings, re-check before concluding it's clean. Empty findings array if genuinely clean. Return JSON per the schema.`,
-    ].filter(Boolean).join("\n"),
-    { label: `find:${slice.key}`, phase: "Deep Scan", schema: FINDER_SCHEMA, model: "sonnet" }
-  ),
-  async (finderResult, originalItem) => {
-    const slice = originalItem || {};
-    const sliceKey = slice.key || (finderResult && finderResult.slice) || "unknown-slice";
-    if (!finderResult || !Array.isArray(finderResult.findings)) return { slice: sliceKey, findings: [] };
-    if (verifyMode === "none" || finderResult.findings.length === 0) {
-      return { slice: sliceKey, findings: finderResult.findings || [] };
+
+// M72: a finder that returns no schema-valid output (runtime nudged it twice then
+// dropped it) must NOT be silently treated as a clean slice — that presents PARTIAL
+// coverage as complete (7/19 slices dropped this way on the Hilo run). So: (1) the
+// finder call is RETRIED once on a null/invalid result; (2) a slice that still fails
+// is flagged `failed:true` and tracked — never conflated with a genuinely-empty slice.
+function finderPrompt(slice) {
+  return [
+    `⛔ Scan ONLY files under the absolute project path \`${projectDir}\`. \`cd ${projectDir}\` first; never read outside this tree.`,
+    `You are a DEEP tech-debt finder for ONE slice of a scan of \`${projectDir}\`: \`${slice.key}\` (dimension: ${slice.dimension}).`,
+    `Owned paths (relative to \`${projectDir}\`): ${JSON.stringify(slice.paths)}.`,
+    slice.why ? `Why this slice matters: ${slice.why}` : ``,
+    ``,
+    `MANDATE: ENUMERATE, do NOT sample. Read EVERY file under your owned paths (use Read/Grep). You own only this slice, so go to the bottom of it.`,
+    `Depth = ${deep}. "thorough" = every file, every non-trivial real defect (high+medium confidence). "MAXIMUM" = also lower-confidence/speculative items worth review.`,
+    `Surface: bugs, security holes, missing validation, broken invariants, race conditions, dead/duplicated code, N+1s, untested critical paths, contract drift, domain-specific correctness (money math, state-machine gaps, timezone bugs, idempotency holes).`,
+    `For each finding: title, severity (CRITICAL/HIGH/MEDIUM/LOW), human area label, concrete file:line refs, detail, impact, remediation, honest confidence. If a substantial slice yields only 1-2 findings, re-check before concluding it's clean. Empty findings array ONLY if genuinely clean.`,
+    `CRITICAL: you MUST return a JSON object matching the schema (slice + findings array) as your FINAL output — even if findings is empty. Do not end without the structured result.`,
+  ].filter(Boolean).join("\n");
+}
+// M73: GLOBAL CONCURRENCY GATE (shared-worker-pool model). The v4.0.19 Hilo run
+// fanned out 29 finders + their verifiers ALL AT ONCE → ~58 concurrent Sonnet agents
+// → server-side API rate limit → all errored empty → 0 findings. Rather than batch
+// per-slice (which leaves slots idle while a slice serializes its verifies), use ONE
+// global semaphore of MAX_CONCURRENT permits that EVERY agent call (finder OR verify,
+// from any slice) must acquire. Work fans out naturally; the gate alone enforces the
+// cap. The instant any agent finishes, the next queued one starts → always ≈10 in
+// flight while work remains = max throughput at a safe ceiling. (All gated agents are
+// Sonnet; the lone Opus synthesis runs after, ungated.)
+const MAX_CONCURRENT = 10;
+
+// Minimal counting semaphore: acquire() resolves when a permit is free; release()
+// hands the permit to the next waiter (FIFO).
+function makeSemaphore(permits) {
+  let avail = permits;
+  const waiters = [];
+  return {
+    async acquire() {
+      if (avail > 0) { avail--; return; }
+      await new Promise((res) => waiters.push(res));
+      // resumed: a release() handed us the permit (avail already decremented there).
+    },
+    release() {
+      const next = waiters.shift();
+      if (next) next();           // hand permit directly to the next waiter
+      else avail++;               // no waiter: return permit to the pool
+    },
+  };
+}
+const gate = makeSemaphore(MAX_CONCURRENT);
+async function gatedAgent(prompt, opts) {
+  await gate.acquire();
+  try { return await agent(prompt, opts); }
+  finally { gate.release(); }
+}
+
+async function runFinder(slice) {
+  // up to 2 attempts; a null/invalid (non-array findings) result counts as a drop.
+  for (let attempt = 1; attempt <= 2; attempt++) {
+    try {
+      const r = await gatedAgent(finderPrompt(slice), {
+        label: attempt === 1 ? `find:${slice.key}` : `find:${slice.key} (retry)`,
+        phase: "Deep Scan", schema: FINDER_SCHEMA, model: "sonnet",
+      });
+      if (r && Array.isArray(r.findings)) return r; // valid (incl. empty)
+      log(`⚠ finder slice "${slice.key}" attempt ${attempt} returned no valid output${attempt < 2 ? " — retrying" : ""}`);
+    } catch (e) {
+      log(`⚠ finder slice "${slice.key}" attempt ${attempt} threw: ${e && e.message}${attempt < 2 ? " — retrying" : ""}`);
     }
-    const verified = await parallel(
-      finderResult.findings.map((f) => async () => {
-        try {
-          const v = await agent(
-            [
-              `You are a VERIFIER for one tech-debt finding in \`${projectDir}\`. Confirm it against the ACTUAL code (open the referenced files with Read) — do not trust the finder.`,
-              `Finding: ${JSON.stringify(f)}`,
-              `confirmed=true only if the defect genuinely exists. If misread → verdict="false-positive". If real but wrong severity → set correctedSeverity. If real but underspecified → verdict="needs-detail" (kept). Return JSON per the schema.`,
-            ].join("\n"),
-            { label: `verify:${sliceKey}`, phase: "Deep Scan", schema: VERIFY_SCHEMA, model: "sonnet" }
-          );
-          if (!v || v.verdict === "false-positive" || v.confirmed === false) return null;
-          return { ...f, severity: v.correctedSeverity || f.severity, _verify: v.verdict };
-        } catch (e) {
-          return { ...f, _verify: "verify-errored" };
-        }
-      })
-    );
-    return { slice: sliceKey, findings: verified.filter(Boolean) };
   }
-);
-const allFindings = sliceResults.filter(Boolean).flatMap((r) => (r.findings || []).map((f) => ({ ...f, slice: r.slice })));
-log(`deep scan complete: ${allFindings.length} verified findings across ${sliceResults.filter(Boolean).length} slices`);
+  return null; // both attempts failed → dropped slice
+}
+
+async function scanSlice(slice) {
+  const sliceKey = slice.key || "unknown-slice";
+  const finderResult = await runFinder(slice);
+  // M72: distinguish a FAILED finder (null after retries) from a genuinely-clean slice.
+  if (!finderResult || !Array.isArray(finderResult.findings)) {
+    return { slice: sliceKey, findings: [], failed: true };
+  }
+  if (verifyMode === "none" || finderResult.findings.length === 0) {
+    return { slice: sliceKey, findings: finderResult.findings || [], failed: false };
+  }
+  // Fan out ALL verifies for this slice — the global gate (not a per-slice limit)
+  // bounds total in-flight, so this is safe AND keeps every worker slot busy.
+  const verified = await parallel(
+    finderResult.findings.map((f) => async () => {
+      try {
+        const v = await gatedAgent(
+          [
+            `You are a VERIFIER for one tech-debt finding in \`${projectDir}\`. Confirm it against the ACTUAL code (open the referenced files with Read) — do not trust the finder.`,
+            `Finding: ${JSON.stringify(f)}`,
+            `confirmed=true only if the defect genuinely exists. If misread → verdict="false-positive". If real but wrong severity → set correctedSeverity. If real but underspecified → verdict="needs-detail" (kept). Return JSON per the schema.`,
+          ].join("\n"),
+          { label: `verify:${sliceKey}`, phase: "Deep Scan", schema: VERIFY_SCHEMA, model: "sonnet" }
+        );
+        if (!v || v.verdict === "false-positive" || v.confirmed === false) return null;
+        return { ...f, severity: v.correctedSeverity || f.severity, _verify: v.verdict };
+      } catch (e) {
+        return { ...f, _verify: "verify-errored" };
+      }
+    })
+  );
+  return { slice: sliceKey, findings: verified.filter(Boolean), failed: false };
+}
+
+// Fan out ALL slices at once — every finder + verify acquires the shared gate, so
+// total in-flight never exceeds MAX_CONCURRENT regardless of slice/finding counts.
+log(`deep scan: ${slices.length} slices via a shared ${MAX_CONCURRENT}-permit gate (Sonnet finders+verifiers; max throughput at a safe ceiling)`);
+const sliceResults = await parallel(slices.map((slice) => () => scanSlice(slice)));
+
+// M72: coverage accounting — a dropped pipeline result (null) OR a failed:true slice
+// is a COVERAGE GAP. Surface it deterministically; never present partial as complete.
+const resultsByIndex = sliceResults; // pipeline preserves order
+const failedSlices = [];
+slices.forEach((s, i) => {
+  const r = resultsByIndex[i];
+  if (!r || r.failed) failedSlices.push(s.key);
+});
+const succeededCount = slices.length - failedSlices.length;
+const coverageComplete = failedSlices.length === 0;
+const allFindings = sliceResults.filter(Boolean).filter((r) => !r.failed).flatMap((r) => (r.findings || []).map((f) => ({ ...f, slice: r.slice })));
+if (!coverageComplete) {
+  log(`⚠ PARTIAL COVERAGE — ${failedSlices.length}/${slices.length} slices failed after retry and produced NO findings: ${failedSlices.join(", ")}. The register will be flagged INCOMPLETE. Resume the run to re-scan only the failed slices.`);
+}
+log(`deep scan complete: ${allFindings.length} verified findings across ${succeededCount}/${slices.length} slices${coverageComplete ? " (full coverage)" : " (PARTIAL)"}`);
 
 // Synthesis — an agent does the ARCHIVE + REGISTER WRITE + GIT entirely via its
 // own Bash/Write tools. The orchestrator does NOT touch fs. The agent is given the
@@ -320,13 +400,19 @@ const synthesis = await agent(
     ``,
     `STEP 1 — ARCHIVE (do this FIRST, deterministically, via Bash): if \`${projectDir}/.gsd-t/techdebt.md\` exists, rename it to \`${projectDir}/.gsd-t/techdebt_YYYY-MM-DD.md\` using its header date (fallback: today's date from \`date +%F\`); on same-day collision append \`_2\`, \`_3\`. Use \`git mv\` if it's a git repo, else \`mv\`. Capture the archive path. ${pre.priorRegisterExists ? "(A prior register EXISTS — you MUST archive it.)" : "(No prior register — skip archiving.)"}`,
     ``,
-    `STEP 2 — WRITE the fresh \`${projectDir}/.gsd-t/techdebt.md\` (Write tool). Start TD numbering at TD-${tdStart} (computed deterministically — do NOT renumber or restart). Structure: a Summary table (CRITICAL/HIGH/MEDIUM/LOW counts), then sections Critical→High→Medium→Low, each finding as \`### TD-NNN — {title}\` with Area / Severity / Status: OPEN / Location (file:line) / Description / Impact / Remediation / Milestone candidate. Re-rank globally by true severity; de-duplicate findings multiple slices surfaced into one item listing all locations. GSD-T effort units only (domain/wave/spawn/token) — never human-hours.`,
+    // M72: MANDATORY coverage banner — enforced here, not left to the agent's notice.
+    coverageComplete
+      ? `COVERAGE: all ${slices.length} slices succeeded — full coverage. Note this in the header.`
+      : `⚠ COVERAGE IS INCOMPLETE — ${failedSlices.length} of ${slices.length} slices FAILED to return findings and were NOT scanned: ${failedSlices.join(", ")}. You MUST put a prominent "> ⚠ PARTIAL COVERAGE — N of M codebase areas were not scanned this pass (listed below); findings UNDER-COUNT the real debt. Re-run for full coverage." banner at the TOP of the register, AND list the un-scanned slice names. Do NOT present this as a complete picture.`,
+    ``,
+    `STEP 2 — WRITE the fresh \`${projectDir}/.gsd-t/techdebt.md\` (Write tool). Start TD numbering at TD-${tdStart} (computed deterministically — do NOT renumber or restart). Structure: the coverage banner (above), a Summary table (CRITICAL/HIGH/MEDIUM/LOW counts), then sections Critical→High→Medium→Low, each finding as \`### TD-NNN — {title}\` with Area / Severity / Status: OPEN / Location (file:line) / Description / Impact / Remediation / Milestone candidate. Re-rank globally by true severity; de-duplicate findings multiple slices surfaced into one item listing all locations. GSD-T effort units only (domain/wave/spawn/token) — never human-hours.`,
+    `If the findings set is large, WRITE INCREMENTALLY — create the file with the header+summary first (Write), then APPEND each severity section with Edit. Do NOT attempt to emit the entire multi-hundred-item register in a single Write call (it can stall); build it up section by section so progress is durable.`,
     ``,
     `STEP 3 — COMMIT via Bash if it's a git repo (\`git add .gsd-t/techdebt.md\` + the archive; commit). Do NOT push.`,
     ``,
-    `Verified findings:`,
+    `Verified findings (${allFindings.length} total):`,
     "```json",
-    findingsJson.length > 200000 ? findingsJson.slice(0, 200000) + "\n…(truncated)" : findingsJson,
+    findingsJson.length > 500000 ? findingsJson.slice(0, 500000) + "\n…(TRUNCATED — too many findings to inline; the above is the first portion. Note in the register that some lower-severity items may be omitted due to volume.)" : findingsJson,
     "```",
     ``,
     `Return JSON per the schema: status "written" once the register file is on disk, the counts, the archivePath (or ""), and tdRange (e.g. "TD-${tdStart}..TD-NNN").`,
@@ -429,8 +515,12 @@ log(`docs commit: ${commitAgent && commitAgent.status}`);
 // projectDir. The fragile, wrong-tree HTML report is not worth that risk; dropped.
 
 return {
-  status: "complete",
-  slices: slices.length,
+  // M72: status reflects coverage — a partial scan is NOT "complete".
+  status: coverageComplete ? "complete" : "complete-partial-coverage",
+  coverageComplete,
+  slicesTotal: slices.length,
+  slicesSucceeded: succeededCount,
+  slicesFailed: failedSlices,           // names of un-scanned areas (empty if full coverage)
   findings: allFindings.length,
   counts: synthesis.counts,
   tdRange: synthesis.tdRange,