testguard-cli 0.1.3 → 0.2.0

package/CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-09-17
+
+### Added
+
+- **`testguard scaffold <file>`** — mechanical fault producer (#6). Proposes
+  the five shapes both field reports found behind ~80% of hand-written
+  faults: guard forced false, single-line guard or state change removed,
+  `return <check>` → `return true`, security literal weakened, check call
+  removed. Every proposal is an exact-line anchor with `expectHits` and
+  `occurrence` computed from the file (verifiable by construction; anything
+  `locate()` would reject is never emitted), `producedBy: derived`,
+  `defendedBy` prefilled from the tests that import the module, grouped
+  under a preceding `@claim` annotation or by enclosing function.
+  Statements are `TODO:` placeholders; the output is a draft under
+  `.testguard/`, never the claims file. `--claim <ID>` puts everything under
+  one claim and copies it if it exists; `--json` prints instead.
+- Self-claim `TG-SCAFFOLD-ANCHORS-HIT`; install smoke exercises `scaffold`.
+
 ## [0.1.3] - 2026-09-17
 
 From a second field report on a real codebase (456 tests, 27 claims, 35

package/README.md

@@ -52,7 +52,7 @@ tests were written against the survivors, 39/39 were killed.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.1.3` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.2.0` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
@@ -66,6 +66,7 @@ npx testguard-cli claims      # what does this project claim, and is every claim
 npx testguard-cli probe       # try to falsify each claim; report what the tests missed
 npx testguard-cli baseline    # freeze today's unproven findings; from now on only new ones gate
 npx testguard-cli brief       # tell the agent where the suite is blind, before it writes
+npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to keep or drop
 ```
 
 1. **Claims** live in `testguard.claims.json` (editors validate it against
@@ -135,6 +136,34 @@ npx testguard-cli brief       # tell the agent where the suite is blind, before
    `--text` prints only, and exits 0 silently when there is no evidence yet,
    so the hook can never break a session.
 
+### Authoring faults mechanically
+
+Writing faults by hand means reading the code to find exact anchors. Two
+field reports found that ~80% of hand-written faults are one of five shapes,
+so `scaffold` proposes them for you:
+
+```bash
+npx testguard-cli scaffold src/auth.ts          # → .testguard/scaffold-auth.json (a draft, never your claims file)
+npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN   # every proposal under one claim; copies it if it exists
+```
+
+| Shape | What it proposes |
+|---|---|
+| `condition-forced` | `if (<guard>) {` → `if (false) {` — a guard is a `!…` condition or one whose body returns, throws or 4xx-es |
+| `statement-deleted` | a single-line guard (`if (…) return …;`) or a state change (`x = …;`) removed |
+| `return-altered` | `return <check>;` (`===`, `.includes(`, `&&`, …) → `return true;` |
+| `literal-changed` | `httpOnly`/`secure` flipped, `sameSite` → `none`, a cost/rounds → `1`, a ttl/tolerance/window/limit ×1000 |
+| `call-removed` | a bare `verify…()` / `validate…()` / `check…()` / `authorize…()` call removed |
+
+Every proposal's `find` is the exact line with `expectHits`/`occurrence`
+computed from the file, so it is verifiable by construction; provenance is
+`producer: derived`; `defendedBy` is prefilled from the tests that import
+the module; proposals are grouped under a preceding `@claim <ID>` annotation
+or by enclosing function. Statements are `TODO:` placeholders — a proposal
+becomes a claim only when a human states what it defends. Deterministic
+heuristics, no AST, no LLM; a proposal the tool cannot anchor is never
+emitted.
+
 **Commit `.testguard/baseline.json`; ignore `evidence.json` and `brief.json`.**
 The baseline is the frozen contract; the other two are regenerated per run.
 
@@ -160,13 +189,14 @@ through every verdict.
 
 ## Status
 
-**v0.1.** Four commands, vitest runner, hand-authored faults. The contract
+**v0.2.** Five commands, vitest runner, hand-authored faults plus a
+mechanical scaffold for the five common shapes. The contract
 spine — six JSON Schemas shared with the other Guard tools — is under
 [`spec/`](spec/). One exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20.
 
 Not yet: test generation (the two-gate acceptance loop), other runners,
-mechanical fault producers, and calibration of fault classes against real
-escaped bugs. Each is designed for; none is claimed.
+AST-aware producers, and calibration of fault classes against real escaped
+bugs. Each is designed for; none is claimed.
 
 ## Licence
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testguard-cli",
-  "version": "0.1.3",
+  "version": "0.2.0",
   "description": "Proves a test suite defends the claims a project makes: injects the faults those claims forbid and reports every one the tests fail to detect.",
   "type": "module",
   "license": "MIT",

package/src/cli.mjs

@@ -10,6 +10,7 @@ import { probeCommand } from './commands/probe.mjs';
 import { claimsCommand } from './commands/claims.mjs';
 import { baselineCommand } from './commands/baseline.mjs';
 import { briefCommand } from './commands/brief.mjs';
+import { scaffoldCommand } from './commands/scaffold.mjs';
 
 const VERSION = JSON.parse(readFileSync(join(fileURLToPath(import.meta.url), '..', '..', 'package.json'), 'utf8')).version;
 
@@ -19,6 +20,7 @@ const USAGE = `testguard ${VERSION} — proves a test suite defends the claims a
   testguard probe [dir]      inject each claim's faults, run its defenders, report what survived
   testguard baseline [dir]   freeze today's unproven findings so only new ones gate
   testguard brief [dir]      emit the blind-spot block for an agent's session-start context
+  testguard scaffold <file>  propose faults mechanically for one source file, as a draft claims document
 
 probe
   --claims <path>      claims file             (default: <dir>/testguard.claims.json)
@@ -38,6 +40,9 @@ probe
   --no-reuse           re-probe claims whose inputs have not changed
   --quiet              suppress the per-fault stream and ranked block; print only the summary and evidence path
 
+scaffold   --claim <ID> (put every proposal under this claim; copies it if it exists)  --out <path>  --json
+           shapes: if-guard → if (false) · single-line guard/mutation removed · return <check> → return true
+                   · security flag/window/cost literal weakened · verify/validate/check call removed
 claims     --json
 baseline   --evidence <path>  --out <path>
 brief      --evidence <path>  --baseline <path>  --max <n>  --text (print only; safe for hooks)
@@ -45,7 +50,7 @@ brief      --evidence <path>  --baseline <path>  --max <n>  --text (print only;
 exit codes: 0 nothing new to prove · 1 unproven claims (or claim drift) · 2 precondition failed · 3 usage
 `;
 
-const COMMANDS = { probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand };
+const COMMANDS = { probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand };
 
 export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n'), err: (s) => process.stderr.write(s + '\n') }) {
   let parsed;
@@ -104,7 +109,8 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
     return 3;
   }
   try {
-    return await handler({ projectDir: resolve(dirArg ?? '.'), values, version: VERSION }, io);
+    const projectDir = command === 'scaffold' ? resolve('.') : resolve(dirArg ?? '.');
+    return await handler({ projectDir, file: dirArg, values, version: VERSION }, io);
   } catch (e) {
     if (e instanceof ClaimsError || e instanceof PreconditionError || e instanceof GitError || e instanceof SpecDocError) {
       io.err(`error: ${e.message}`);

package/src/commands/scaffold.mjs

@@ -0,0 +1,34 @@
+import { existsSync } from 'node:fs';
+import { join, resolve, relative, basename, extname } from 'node:path';
+import { scaffoldFile } from '../scaffold/scaffold.mjs';
+import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
+import { writeSpecDoc } from '../evidence/writer.mjs';
+import { PreconditionError } from '../probe/worktree.mjs';
+
+export async function scaffoldCommand({ projectDir, file, values, version }, io) {
+  if (!file) {
+    io.err('usage: testguard scaffold <source-file> [--claim <ID>] [--out <path>] [--json]');
+    return 3;
+  }
+  const abs = resolve(file);
+  if (!existsSync(abs)) throw new PreconditionError(`no such file: ${file}`);
+  const rel = relative(projectDir, abs);
+  if (rel.startsWith('..')) throw new PreconditionError(`${file} is outside the project directory ${projectDir}`);
+
+  const claimsPath = values.claims ? resolve(values.claims) : defaultClaimsPath(projectDir);
+  const existingClaims = existsSync(claimsPath) ? loadClaims(claimsPath) : undefined;
+  const { doc, stats } = scaffoldFile({ projectDir, file: rel, claimId: values.claim, existingClaims, toolVersion: version });
+
+  if (values.json) {
+    io.out(JSON.stringify(doc, null, 2));
+    return 0;
+  }
+  const outPath = values.out ? resolve(values.out) : join(projectDir, '.testguard', `scaffold-${basename(rel, extname(rel))}.json`);
+  writeSpecDoc('claims', outPath, doc);
+  const shapes = Object.entries(stats.byClass).map(([k, v]) => `${v} ${k}`).join(', ');
+  io.out(`${stats.proposals} proposed fault${stats.proposals === 1 ? '' : 's'} in ${stats.claims} draft claim${stats.claims === 1 ? '' : 's'} for ${rel}${shapes ? ` — ${shapes}` : ''}`);
+  io.out(stats.defendedBy.length ? `defendedBy prefilled from imports: ${stats.defendedBy.join(', ')}` : 'no test file imports this module; probing the draft as-is will report NOCOVER');
+  io.out(`draft: ${outPath}`);
+  io.out('Next: replace each TODO statement, drop proposals that are not claims, then move the claims into testguard.claims.json.');
+  return 0;
+}

package/src/scaffold/producers.mjs

@@ -0,0 +1,85 @@
+/**
+ * Mechanical fault producers. Deterministic, line-oriented, no AST, no LLM.
+ *
+ * Two independent field reports found that ~80% of hand-written faults are
+ * one of these shapes. Each producer looks at one line (plus a little
+ * context) and proposes a fault whose `find` is the exact line, so the anchor
+ * hits by construction. A human keeps or drops every proposal.
+ */
+
+const COMMENT = /^\s*(\/\/|\*|\/\*)/;
+const GUARD_BODY = /\b(return|throw)\b|\.status\(\s*4\d\d|\bredirect\(|\bfalse\b|\bnull\b/;
+const IF_LINE = /^(\s*)(?:\}\s*)?(?:else\s+)?if\s*\((.+)\)\s*(\{\s*|(?:return|throw)\b.*;\s*)?$/;
+const RETURN_CHECK = /^\s*return\s+(.+);\s*$/;
+const CHECK_EXPR = /(===|!==|\.includes\(|\.has\(|\.some\(|\.every\(|\.test\(|\.startsWith\(|\.endsWith\(|\binstanceof\b|&&|\|\||^!)/;
+const CHECK_CALL = /^\s*(?:await\s+)?(?:[\w$]+\.)*(verify|validate|assert|check|require|ensure|authoriz|authentic|rateLimit|throttle|enforce|guard)\w*\s*\(.*\)\s*;\s*$/i;
+const MUTATION = /^\s*(?!(?:const|let|var|return|if|for|while|else|switch|case|import|export|throw)\b)[\w$]+(?:[.\[][\w$'"\]]+)*\s*(=|\+=|-=|\|\|=|&&=|\?\?=)\s*(?!=)[^;]*;\s*$/;
+const FLAG_TRUE = /\b(httpOnly|secure|signed|requireTLS|rejectUnauthorized|strict)\s*:\s*true\b/;
+const SAME_SITE = /\bsameSite\s*:\s*(['"])(strict|lax)\1/i;
+const COST_NUM = /\b(\w*(?:cost|rounds|iterations)\w*)\s*([:=])\s*(\d+)\b/i;
+const WINDOW_NUM = /\b(\w*(?:ttl|expir|tolerance|window|maxAge|max_age|timeout|limit|attempts|length|minLength|clockSkew|skew)\w*)\s*([:=])\s*(\d+)\b/i;
+const FUNCTION_HEAD = [
+  /^\s*(?:export\s+)?(?:default\s+)?(?:async\s+)?function\s*\*?\s*([\w$]+)\s*\(/,
+  /^\s*(?:export\s+)?(?:const|let|var)\s+([\w$]+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|[\w$]+)\s*=>/,
+  /^\s*(?:export\s+)?(?:const|let|var)\s+([\w$]+)\s*=\s*(?:async\s+)?function\b/,
+  /^\s*(?:public\s+|private\s+|protected\s+)?(?:static\s+)?(?:async\s+)?([\w$]+)\s*\([^)]*\)\s*(?::\s*[\w<>\[\]| ]+)?\s*\{\s*$/,
+];
+const NOT_A_FUNCTION = new Set(['if', 'for', 'while', 'switch', 'catch', 'else', 'return', 'function', 'constructor']);
+
+/** Name of the function whose head is on this line, or null. */
+export function functionHead(line) {
+  for (const re of FUNCTION_HEAD) {
+    const m = re.exec(line);
+    if (m && !NOT_A_FUNCTION.has(m[1])) return m[1];
+  }
+  return null;
+}
+
+function isGuard(cond, lines, i) {
+  if (/^\s*!/.test(cond)) return true;
+  const tail = lines.slice(i, i + 4).join('\n');
+  return GUARD_BODY.test(tail);
+}
+
+/** Proposals for one line. Each: { faultClass, description, replace } where find is the line itself. */
+export function proposalsForLine(lines, i) {
+  const line = lines[i];
+  const out = [];
+  if (!line.trim() || COMMENT.test(line)) return out;
+
+  const ifm = IF_LINE.exec(line);
+  if (ifm && isGuard(ifm[2], lines, i)) {
+    const singleLine = ifm[3] && /^(return|throw)\b/.test(ifm[3].trim());
+    if (singleLine) {
+      out.push({ faultClass: 'statement-deleted', description: `Guard removed: \`${line.trim()}\` no longer runs.`, replace: '' });
+    } else {
+      out.push({ faultClass: 'condition-forced', description: `Guard never triggers: \`if (${ifm[2].trim()})\` becomes \`if (false)\`.`, replace: line.replace(`(${ifm[2]})`, '(false)') });
+    }
+    return out; // an `if` line is not also a mutation/return/call
+  }
+
+  const rm = RETURN_CHECK.exec(line);
+  if (rm && CHECK_EXPR.test(rm[1]) && !/^(new|await)\b/.test(rm[1].trim())) {
+    out.push({ faultClass: 'return-altered', description: `Check always passes: \`return ${rm[1].trim()}\` becomes \`return true\`.`, replace: line.replace(/return\s+.+;/, 'return true;') });
+  }
+
+  if (CHECK_CALL.test(line)) {
+    out.push({ faultClass: 'call-removed', description: `Check call removed: \`${line.trim()}\` no longer runs.`, replace: '' });
+  } else if (MUTATION.test(line)) {
+    out.push({ faultClass: 'statement-deleted', description: `State change removed: \`${line.trim()}\` no longer runs.`, replace: '' });
+  }
+
+  let m;
+  if ((m = FLAG_TRUE.exec(line))) {
+    out.push({ faultClass: 'literal-changed', description: `Security flag flipped: \`${m[1]}: true\` becomes \`${m[1]}: false\`.`, replace: line.replace(m[0], `${m[1]}: false`) });
+  }
+  if ((m = SAME_SITE.exec(line))) {
+    out.push({ faultClass: 'literal-changed', description: `sameSite weakened: \`${m[2]}\` becomes \`none\`.`, replace: line.replace(m[0], `sameSite: ${m[1]}none${m[1]}`) });
+  }
+  if ((m = COST_NUM.exec(line))) {
+    out.push({ faultClass: 'literal-changed', description: `Work factor collapsed: \`${m[1]}\` ${m[3]} becomes 1.`, replace: line.replace(m[0], `${m[1]}${m[2] === ':' ? ': ' : ' = '}1`) });
+  } else if ((m = WINDOW_NUM.exec(line))) {
+    out.push({ faultClass: 'literal-changed', description: `Window widened ×1000: \`${m[1]}\` ${m[3]} becomes ${Number(m[3]) * 1000}.`, replace: line.replace(m[0], `${m[1]}${m[2] === ':' ? ': ' : ' = '}${Number(m[3]) * 1000}`) });
+  }
+  return out;
+}

package/src/scaffold/scaffold.mjs

@@ -0,0 +1,118 @@
+import { readFileSync } from 'node:fs';
+import { basename, extname, join } from 'node:path';
+import { proposalsForLine, functionHead } from './producers.mjs';
+import { locate } from '../probe/inject.mjs';
+import { discoverDefenders } from '../probe/discover.mjs';
+import { validate } from '../../spec/lib/validate.mjs';
+
+const ANNOTATION = /@claim\s+([A-Za-z0-9]+(?:[._]?[A-Za-z0-9]+)*-[A-Za-z0-9._-]*[A-Za-z0-9])\b/;
+
+const idPart = (s) => s.replace(/[^A-Za-z0-9]+/g, '-').replace(/^-+|-+$/g, '').toUpperCase();
+
+/** How many times `find` occurs, and which occurrence the line at `offset` is. */
+function anchorFor(source, find, offset) {
+  let hits = 0;
+  let occurrence = 0;
+  let i = -1;
+  while ((i = source.indexOf(find, i + 1)) !== -1) {
+    hits++;
+    if (i === offset) occurrence = hits;
+  }
+  return { hits, occurrence };
+}
+
+/**
+ * Scan one source file and produce a DRAFT claims document: every proposal is
+ * an exact-line anchor that `locate()` accepts, grouped by enclosing function
+ * (or by a preceding `@claim <ID>` annotation, or entirely under `claimId`),
+ * with `producedBy: { producer: "derived" }` and TODO statements a human must
+ * replace. Never touches the real claims file.
+ */
+export function scaffoldFile({ projectDir, file, claimId, existingClaims, toolVersion = '0.0.0' }) {
+  const source = readFileSync(join(projectDir, file), 'utf8');
+  const lines = source.split('\n');
+  const stem = idPart(basename(file, extname(file)));
+  const producedBy = { producer: 'derived', by: `testguard scaffold ${toolVersion}` };
+
+  const groups = new Map(); // key → { id, proposals[] , annotated }
+  const groupFor = (key, id) => {
+    if (!groups.has(key)) groups.set(key, { id, proposals: [] });
+    return groups.get(key);
+  };
+
+  let offset = 0;
+  let fn = null;
+  let fnDepth = 0;
+  let depth = 0;
+  let pendingAnnotation = null;
+  const proposals = [];
+  lines.forEach((line, i) => {
+    const ann = ANNOTATION.exec(line);
+    if (ann && /^\s*(\/\/|\*|\/\*)/.test(line)) pendingAnnotation = ann[1];
+
+    const head = functionHead(line);
+    if (head) {
+      fn = head;
+      fnDepth = depth;
+    }
+
+    for (const p of proposalsForLine(lines, i)) {
+      const find = line;
+      const { hits, occurrence } = anchorFor(source, find, offset);
+      const fault = { ...p, find, replace: p.replace, expectHits: hits, occurrence, line: i + 1, fn, annotation: pendingAnnotation };
+      if (locate(source, fault).status !== 'ok') continue; // never propose an anchor that would be unverifiable
+      proposals.push(fault);
+    }
+
+    depth += (line.match(/\{/g) ?? []).length - (line.match(/\}/g) ?? []).length;
+    if (fn && depth <= fnDepth && !head) fn = null;
+    if (pendingAnnotation && !ann && !head && line.trim() && !/^\s*(\/\/|\*|\/\*)/.test(line)) pendingAnnotation = null;
+    offset += line.length + 1;
+  });
+
+  const existing = new Map((existingClaims?.claims ?? []).map((c) => [c.id, c]));
+  const usedIds = new Set();
+  for (const p of proposals) {
+    const key = claimId ? '__all__' : p.annotation ?? p.fn ?? '__file__';
+    let id = claimId ?? p.annotation ?? `${stem}-${p.fn ? idPart(p.fn) : 'FILE'}`;
+    groupFor(key, id).proposals.push(p);
+  }
+
+  const defendedBy = discoverDefenders(projectDir, file);
+  const claims = [];
+  for (const g of groups.values()) {
+    let id = g.id;
+    while (usedIds.has(id)) id += '-2';
+    usedIds.add(id);
+    const base = existing.get(id);
+    const fnLabel = g.proposals[0].fn ? `\`${g.proposals[0].fn}\`` : 'this file';
+    const claim = {
+      id,
+      statement: base?.statement ?? `TODO: state what ${fnLabel} in ${file} guarantees (${g.proposals.length} proposed fault${g.proposals.length === 1 ? '' : 's'}; keep or drop each)`,
+      source: base?.source ?? { kind: 'manual', ref: `testguard scaffold ${file}` },
+      severity: base?.severity ?? 'medium',
+      producedBy: base?.producedBy ?? producedBy,
+      ...(base?.defendedBy?.length ? { defendedBy: base.defendedBy } : defendedBy.length ? { defendedBy } : {}),
+      faults: g.proposals.map((p, n) => ({
+        id: `S${n + 1}`,
+        description: `[line ${p.line}] ${p.description}`,
+        faultClass: p.faultClass,
+        file,
+        find: p.find,
+        replace: p.replace,
+        ...(p.expectHits > 1 ? { expectHits: p.expectHits, occurrence: p.occurrence } : {}),
+        producedBy,
+      })),
+      ...(base?.tags ? { tags: base.tags } : {}),
+    };
+    claims.push(claim);
+  }
+
+  const doc = { $schema: './node_modules/testguard-cli/spec/schemas/claims.schema.json', schemaVersion: 1, claims };
+  const result = validate('claims', doc);
+  if (!result.ok) throw new Error(`scaffold produced a non-conforming draft:\n${result.errors.map((e) => `  ${e.path}: ${e.message}`).join('\n')}`);
+
+  const byClass = {};
+  for (const p of proposals) byClass[p.faultClass] = (byClass[p.faultClass] ?? 0) + 1;
+  return { doc, stats: { proposals: proposals.length, claims: claims.length, byClass, defendedBy } };
+}