@dombaras/agent-harness 0.1.2 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -2
- package/bin/agent-harness.js +247 -8
- package/package.json +2 -2
- package/templates/.agents/AGENTS.md +15 -4
- package/templates/.agents/memory/flow-map.md +23 -0
- package/templates/.agents/memory/model-routing.md +5 -1
- package/templates/.agents/memory/qa-plan.md +49 -0
- package/templates/.agents/rules/00-operating.md +2 -1
- package/templates/.agents/skills/planner/SKILL.md +5 -0
- package/templates/.agents/skills/qa-architect/SKILL.md +37 -5
- package/templates/.agents/skills/qa-runner/SKILL.md +1 -0
- package/templates/.opencode/agents/qa-architect.md +2 -1
- package/templates/AGENTS.md +1 -0
- package/templates/scripts/qa/check-qa-plan.js +321 -0
- package/templates/scripts/qa/check-qa-scripts.js +1 -0
package/README.md
CHANGED
|
@@ -39,8 +39,8 @@ npx @dombaras/agent-harness init --target . --dry-run
|
|
|
39
39
|
| `.agents/rules/00-operating.md` | always-loaded rules summary (wired into `opencode.json` `instructions`) | harness (overwrite) |
|
|
40
40
|
| `.agents/skills/*/SKILL.md` | persona instruction skills | harness (overwrite) |
|
|
41
41
|
| `opencode.json` | main/small model routing + `instructions` | harness (**merged**, see below) |
|
|
42
|
-
| `scripts/qa/*` | `test:dispatch` / `test:governance` gates + QA-script wiring check | harness (overwrite) |
|
|
43
|
-
| `.agents/memory/*` | project data (domain-map, stack-versions, handoff, locations, model-routing, history) | **project** (create-if-missing) |
|
|
42
|
+
| `scripts/qa/*` | `test:dispatch` / `test:governance` / `test:qa-plan` gates + QA-script wiring check | harness (overwrite) |
|
|
43
|
+
| `.agents/memory/*` | project data (domain-map, stack-versions, handoff, locations, model-routing, history, flow-map, qa-plan) | **project** (create-if-missing) |
|
|
44
44
|
| `.harness.json` | deployed version + project profile + per-file checksums | harness |
|
|
45
45
|
|
|
46
46
|
### `opencode.json` is merged, not clobbered
|
|
@@ -59,6 +59,13 @@ tiers, trust tiers) live in `.agents/memory/domain-map.md`, which `init`
|
|
|
59
59
|
scaffolds for you to fill in — the skills and rules reference memory instead of
|
|
60
60
|
hardcoding domain assumptions.
|
|
61
61
|
|
|
62
|
+
**Flow map (`.agents/memory/flow-map.md`)** — the project-owned registry powering the
|
|
63
|
+
flow-closure half of `test:qa-plan`. Each user-facing flow lists the code surfaces that
|
|
64
|
+
implement the same behavior across codebases/layers (e.g. web component, `mobile/` sheet,
|
|
65
|
+
API route) plus its optional variants (e.g. `condition-selector-present` vs `-absent`). Fill
|
|
66
|
+
it in so a change to one surface is forced to account for all the others. If empty, the gate
|
|
67
|
+
warns but does not block; the guarantee only applies to registered flows.
|
|
68
|
+
|
|
62
69
|
## Model routing & enforcement
|
|
63
70
|
|
|
64
71
|
- Concrete models are set in `.opencode/agents/<name>.md` (`model:` field) and
|
|
@@ -84,6 +91,16 @@ hardcoding domain assumptions.
|
|
|
84
91
|
|
|
85
92
|
- `npm run test:dispatch` — persona model-pin preflight + persona↔skill parity + model-routing↔pin drift check.
|
|
86
93
|
- `npm run test:governance` — session wrap-up dispatch-log enforcement.
|
|
94
|
+
- `npm run test:qa-plan` — **diff-coverage + flow-closure gate**:
|
|
95
|
+
every changed CODE path must have a covering assertion (or a waivered reason) in
|
|
96
|
+
`.agents/memory/qa-plan.md` before a change is verified — and every touched flow
|
|
97
|
+
(`.agents/memory/flow-map.md`) must have every sibling surface and declared optional
|
|
98
|
+
variant addressed in the plan's `## Parallel-surface & variant audit`. Kills the
|
|
99
|
+
"verified by a suite that never touched the change" failure mode: a tier label never
|
|
100
|
+
proves coverage. It also makes **"Fix One, Fix All" mechanical**: a change to one
|
|
101
|
+
surface (e.g. web) cannot ship while the same flow in another codebase (e.g. `mobile/`)
|
|
102
|
+
silently keeps old behavior — the plan must name each sibling as covered (`->`) or
|
|
103
|
+
out-of-scope (`audited: reason`, e.g. `deferred — logged in handoff.md`).
|
|
87
104
|
- `node scripts/qa/check-qa-scripts.js` — warns (or `--strict` fails) when the
|
|
88
105
|
DoD-referenced runtime QA tiers aren't wired into `package.json`.
|
|
89
106
|
|
|
@@ -99,6 +116,22 @@ npx @dombaras/agent-harness update --target /path/to/project
|
|
|
99
116
|
```
|
|
100
117
|
|
|
101
118
|
- Overwrites harness-owned files, preserves `.agents/memory/*`.
|
|
119
|
+
- Auto-wires the harness gate scripts (`test:dispatch`, `test:governance`,
|
|
120
|
+
`test:qa-plan`) into the target's `package.json` (merged, add-only).
|
|
121
|
+
- **Auto-commits** only the harness files it changed (`chore(harness): @dombaras/agent-harness
|
|
122
|
+
<old> -> <new>`) and **pushes** to origin, so the next session never sees unexplained
|
|
123
|
+
modified harness files. Your unrelated uncommitted work is never staged.
|
|
124
|
+
- `--no-commit` to skip commit+push, `--no-push` to commit but not push.
|
|
125
|
+
- `init` never commits.
|
|
126
|
+
- **`[Nothing-to-Update]` confirmation**: if the target is already at this CLI's version and no
|
|
127
|
+
harness-owned file changed, `update` says so explicitly, writes nothing, and makes no commit —
|
|
128
|
+
a no-op is never mistaken for a sync and never churns `.harness.json`.
|
|
129
|
+
- **Downgrade guard**: `update` refuses to run on a target whose deployed harness is NEWER than
|
|
130
|
+
this CLI (e.g. deployed from a local checkout that was never published) and tells you to publish
|
|
131
|
+
that version first.
|
|
132
|
+
- **Published-version check**: if this CLI's version is not the latest `dist-tags.latest` on npm,
|
|
133
|
+
`update` warns you — catches the "bumped the version and updated a project from a local
|
|
134
|
+
checkout, but never published" gap before it silently forks the projects.
|
|
102
135
|
- **Non-destructive**: if a harness-owned file was modified locally since the
|
|
103
136
|
last deploy (tracked by checksum in `.harness.json`), it is backed up to
|
|
104
137
|
`.harness-backup/<timestamp>/` before overwrite.
|
package/bin/agent-harness.js
CHANGED
|
@@ -27,6 +27,7 @@ const fs = require("fs");
|
|
|
27
27
|
const path = require("path");
|
|
28
28
|
const crypto = require("crypto");
|
|
29
29
|
const readline = require("readline");
|
|
30
|
+
const { spawnSync } = require("child_process");
|
|
30
31
|
|
|
31
32
|
const PKG = require("../package.json");
|
|
32
33
|
const TEMPLATES_DIR = path.resolve(__dirname, "..", "templates");
|
|
@@ -35,6 +36,36 @@ const BACKUP_DIR = ".harness-backup";
|
|
|
35
36
|
|
|
36
37
|
// ---------------------------------------------------------------- helpers
|
|
37
38
|
|
|
39
|
+
/* Numeric semver compare (major.minor.patch; prerelease tags ignored).
|
|
40
|
+
* Returns <0 / 0 / >0 like a normal comparator. */
|
|
41
|
+
function compareVersions(a, b) {
|
|
42
|
+
const pa = String(a || "0").split(".").map((n) => parseInt(n, 10) || 0);
|
|
43
|
+
const pb = String(b || "0").split(".").map((n) => parseInt(n, 10) || 0);
|
|
44
|
+
for (let i = 0; i < 3; i++) {
|
|
45
|
+
if ((pa[i] || 0) !== (pb[i] || 0)) return (pa[i] || 0) > (pb[i] || 0) ? 1 : -1;
|
|
46
|
+
}
|
|
47
|
+
return 0;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/* Best-effort: the latest `dist-tags.latest` for PKG.name on the configured
|
|
51
|
+
* registry. Returns null when the registry is unreachable / package unknown,
|
|
52
|
+
* so the check never blocks a deploy. */
|
|
53
|
+
function publishedLatestVersion() {
|
|
54
|
+
const npmCli = process.env.npm_execpath
|
|
55
|
+
? path.join(path.dirname(process.env.npm_execpath), "npm-cli.js")
|
|
56
|
+
: null;
|
|
57
|
+
const args = ["view", PKG.name, "version", "--json", "--fetch-timeout=5000", "--fetch-retries=0"];
|
|
58
|
+
const r = npmCli
|
|
59
|
+
? spawnSync(process.execPath, [npmCli, ...args], { encoding: "utf8", timeout: 10000 })
|
|
60
|
+
: spawnSync("npm", args, { encoding: "utf8", timeout: 10000 });
|
|
61
|
+
if (r.status !== 0) return null;
|
|
62
|
+
try {
|
|
63
|
+
return JSON.parse(r.stdout.trim());
|
|
64
|
+
} catch (_) {
|
|
65
|
+
return r.stdout.trim() || null;
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
|
|
38
69
|
function listFiles(dir, base = dir) {
|
|
39
70
|
const out = [];
|
|
40
71
|
if (!fs.existsSync(dir)) return out;
|
|
@@ -135,6 +166,95 @@ function mergeOpencodeJson(target, variables) {
|
|
|
135
166
|
return JSON.stringify(merged, null, 2) + "\n";
|
|
136
167
|
}
|
|
137
168
|
|
|
169
|
+
// Harness-shipped npm scripts auto-wired into the target project's package.json
|
|
170
|
+
// scripts (merged, not clobbered). Every other script the project has is preserved.
|
|
171
|
+
// These gates ship with the harness, so a deploy wires them up like opencode.json.
|
|
172
|
+
const HARNESS_SCRIPTS = {
|
|
173
|
+
"test:dispatch": "node scripts/qa/check-dispatch-config.js",
|
|
174
|
+
"test:governance": "node scripts/qa/governance.js",
|
|
175
|
+
"test:qa-plan": "node scripts/qa/check-qa-plan.js",
|
|
176
|
+
};
|
|
177
|
+
|
|
178
|
+
// Merge harness gate scripts into the project's existing package.json, preserving
|
|
179
|
+
// every other script. ADD-ONLY: an existing script is never overwritten, so a
|
|
180
|
+
// project's deliberate customization of a gate survives re-deploys. Returns
|
|
181
|
+
// { text, hadPkg }.
|
|
182
|
+
function mergePackageJson(target) {
|
|
183
|
+
const p = path.join(target, "package.json");
|
|
184
|
+
const hadPkg = fs.existsSync(p);
|
|
185
|
+
const existing = hadPkg ? safeJson(fs.readFileSync(p, "utf8"), {}) : {};
|
|
186
|
+
const scripts = Object.assign({}, existing.scripts || {});
|
|
187
|
+
for (const [name, cmd] of Object.entries(HARNESS_SCRIPTS)) {
|
|
188
|
+
if (!(name in scripts)) scripts[name] = cmd;
|
|
189
|
+
}
|
|
190
|
+
const merged = { ...existing, scripts };
|
|
191
|
+
return { text: JSON.stringify(merged, null, 2) + "\n", hadPkg };
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
// ---------------------------------------------------------------- git helpers
|
|
195
|
+
|
|
196
|
+
function runGit(target, args) {
|
|
197
|
+
const result = require("child_process").spawnSync("git", args, {
|
|
198
|
+
cwd: target,
|
|
199
|
+
encoding: "utf8",
|
|
200
|
+
});
|
|
201
|
+
return { ok: result.status === 0, status: result.status, out: (result.stdout || "").trim(), err: (result.stderr || "").trim() };
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
function isGitRepo(target) {
|
|
205
|
+
const r = runGit(target, ["rev-parse", "--is-inside-work-tree"]);
|
|
206
|
+
return r.ok;
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
// Auto-commit ONLY the harness-owned files an update just changed, so the target
|
|
210
|
+
// project's next session never sees "unexplained" modified harness files. Scoped
|
|
211
|
+
// to this run's changed harness files (never `git add -A`) so unrelated uncommitted
|
|
212
|
+
// work is NEVER swept in. Memory/backup files are excluded (project data / rollback).
|
|
213
|
+
// Returns { committed:boolean, staged:string[], skipped:string[], pushed:boolean|null }.
|
|
214
|
+
function commitHarnessChanges(target, changedRels, opts) {
|
|
215
|
+
const out = { committed: false, staged: [], skipped: [], pushed: null };
|
|
216
|
+
if (!opts.commit) return out;
|
|
217
|
+
if (!isGitRepo(target)) {
|
|
218
|
+
out.skipped.push("not a git repo");
|
|
219
|
+
return out;
|
|
220
|
+
}
|
|
221
|
+
if (changedRels.length === 0) {
|
|
222
|
+
out.skipped.push("no harness files changed");
|
|
223
|
+
return out;
|
|
224
|
+
}
|
|
225
|
+
const addR = runGit(target, ["add", "--", ...changedRels]);
|
|
226
|
+
if (!addR.ok) {
|
|
227
|
+
out.skipped.push("git add failed: " + addR.err);
|
|
228
|
+
return out;
|
|
229
|
+
}
|
|
230
|
+
const diffCached = runGit(target, ["diff", "--cached", "--name-only"]);
|
|
231
|
+
const staged = (diffCached.out || "").split(/\r?\n/).filter(Boolean).map(relKey);
|
|
232
|
+
if (staged.length === 0) {
|
|
233
|
+
out.skipped.push("no harness changes staged");
|
|
234
|
+
return out;
|
|
235
|
+
}
|
|
236
|
+
out.staged = staged;
|
|
237
|
+
const prior = opts.priorVersion ? opts.priorVersion + " -> " : "";
|
|
238
|
+
const message = `chore(harness): @dombaras/agent-harness ${prior}${PKG.version}`;
|
|
239
|
+
const commitR = runGit(target, ["commit", "-m", message]);
|
|
240
|
+
if (!commitR.ok) {
|
|
241
|
+
out.skipped.push("commit failed: " + commitR.err);
|
|
242
|
+
return out;
|
|
243
|
+
}
|
|
244
|
+
out.committed = true;
|
|
245
|
+
if (opts.push) {
|
|
246
|
+
const hasRemote = runGit(target, ["remote"]).ok && runGit(target, ["remote"]).out;
|
|
247
|
+
if (hasRemote) {
|
|
248
|
+
const pushR = runGit(target, ["push", "origin", "HEAD"]);
|
|
249
|
+
out.pushed = pushR.ok;
|
|
250
|
+
if (!pushR.ok) out.skipped.push("push failed: " + pushR.err);
|
|
251
|
+
} else {
|
|
252
|
+
out.skipped.push("no git remote to push");
|
|
253
|
+
}
|
|
254
|
+
}
|
|
255
|
+
return out;
|
|
256
|
+
}
|
|
257
|
+
|
|
138
258
|
// ---------------------------------------------------------------- deploy
|
|
139
259
|
|
|
140
260
|
function backup(target, rel, content) {
|
|
@@ -145,7 +265,7 @@ function backup(target, rel, content) {
|
|
|
145
265
|
}
|
|
146
266
|
|
|
147
267
|
async function deploy(target, opts) {
|
|
148
|
-
const { name, domain, yes, dryRun, isUpdate } = opts || {};
|
|
268
|
+
const { name, domain, yes, dryRun, isUpdate, commit, push } = opts || {};
|
|
149
269
|
const existing = readConfig(target);
|
|
150
270
|
const projectName = name || existing.projectName || path.basename(target);
|
|
151
271
|
let projectDomain = domain || existing.projectDomain || "";
|
|
@@ -154,10 +274,37 @@ async function deploy(target, opts) {
|
|
|
154
274
|
}
|
|
155
275
|
const VARIABLES = { PROJECT_NAME: projectName, PROJECT_DOMAIN: projectDomain };
|
|
156
276
|
|
|
277
|
+
// Refuse to downgrade: the target's deployed harness is newer than this CLI.
|
|
278
|
+
// This is how the "I updated from a local checkout that I never published"
|
|
279
|
+
// mistake surfaces instead of silently writing older templates over a newer
|
|
280
|
+
// deployment (and auto-committing that regression).
|
|
281
|
+
if (isUpdate && existing.version && compareVersions(existing.version, PKG.version) > 0) {
|
|
282
|
+
throw new Error(
|
|
283
|
+
`"${projectName}" has harness v${existing.version} — NEWER than this CLI (v${PKG.version}). ` +
|
|
284
|
+
`Not downgrading. Publish v${existing.version} (or this CLI) so \`update\` can re-sync from the registry.`
|
|
285
|
+
);
|
|
286
|
+
}
|
|
287
|
+
|
|
157
288
|
const previousFiles = existing.files || {};
|
|
158
289
|
const actions = [];
|
|
159
290
|
const manifest = {};
|
|
160
291
|
|
|
292
|
+
// Auto-wire harness gate scripts into the target package.json (merged, add-only).
|
|
293
|
+
{
|
|
294
|
+
const { text, hadPkg } = mergePackageJson(target);
|
|
295
|
+
const key = "package.json";
|
|
296
|
+
const targetAbs = path.join(target, "package.json");
|
|
297
|
+
const current = hadPkg ? fs.readFileSync(targetAbs, "utf8") : null;
|
|
298
|
+
if (!hadPkg) actions.push({ rel: key, kind: "create" });
|
|
299
|
+
else if (current !== text) actions.push({ rel: key, kind: "merge" });
|
|
300
|
+
else actions.push({ rel: key, kind: "unchanged" });
|
|
301
|
+
manifest[key] = sha256(text);
|
|
302
|
+
if (!dryRun) {
|
|
303
|
+
fs.mkdirSync(path.dirname(targetAbs), { recursive: true });
|
|
304
|
+
fs.writeFileSync(targetAbs, text, "utf8");
|
|
305
|
+
}
|
|
306
|
+
}
|
|
307
|
+
|
|
161
308
|
for (const { abs, rel } of listFiles(TEMPLATES_DIR)) {
|
|
162
309
|
const key = relKey(rel);
|
|
163
310
|
const targetAbs = path.join(target, rel);
|
|
@@ -213,6 +360,16 @@ async function deploy(target, opts) {
|
|
|
213
360
|
}
|
|
214
361
|
}
|
|
215
362
|
|
|
363
|
+
// Nothing-to-update detection: an update that made no effective change to any
|
|
364
|
+
// harness-owned file and was already running this exact version is a no-op —
|
|
365
|
+
// report it clearly instead of printing a zero-delta "Updated", and leave the
|
|
366
|
+
// target byte-for-byte untouched (no .harness.json timestamp churn, no commit).
|
|
367
|
+
const changedRelsAll = actions
|
|
368
|
+
.filter((a) => ["create", "overwrite", "overwrite (backup)", "merge"].includes(a.kind))
|
|
369
|
+
.map((a) => a.rel);
|
|
370
|
+
const nothingToUpdate =
|
|
371
|
+
!!isUpdate && changedRelsAll.length === 0 && compareVersions(existing.version || "0", PKG.version) === 0;
|
|
372
|
+
|
|
216
373
|
const config = {
|
|
217
374
|
version: PKG.version,
|
|
218
375
|
projectName,
|
|
@@ -221,11 +378,34 @@ async function deploy(target, opts) {
|
|
|
221
378
|
updatedAt: new Date().toISOString(),
|
|
222
379
|
files: manifest,
|
|
223
380
|
};
|
|
224
|
-
if (!dryRun) {
|
|
381
|
+
if (!dryRun && !nothingToUpdate) {
|
|
225
382
|
fs.writeFileSync(path.join(target, CONFIG_FILE), JSON.stringify(config, null, 2) + "\n", "utf8");
|
|
226
383
|
}
|
|
227
384
|
|
|
228
|
-
|
|
385
|
+
// Auto-commit only this run's changed harness-owned files (update only).
|
|
386
|
+
let commitResult = null;
|
|
387
|
+
if (isUpdate && !dryRun && !nothingToUpdate) {
|
|
388
|
+
const changedRels = changedRelsAll.filter(
|
|
389
|
+
(r) => !r.startsWith(".agents/memory/") && !r.startsWith(".harness-backup/")
|
|
390
|
+
);
|
|
391
|
+
changedRels.push(CONFIG_FILE); // .harness.json reflects the new deployed version
|
|
392
|
+
commitResult = commitHarnessChanges(target, changedRels, {
|
|
393
|
+
commit,
|
|
394
|
+
push,
|
|
395
|
+
priorVersion: existing.version || null,
|
|
396
|
+
});
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
return {
|
|
400
|
+
projectName,
|
|
401
|
+
projectDomain,
|
|
402
|
+
actions,
|
|
403
|
+
dryRun: !!dryRun,
|
|
404
|
+
commitResult,
|
|
405
|
+
isUpdate: !!isUpdate,
|
|
406
|
+
nothingToUpdate,
|
|
407
|
+
priorVersion: existing.version || null,
|
|
408
|
+
};
|
|
229
409
|
}
|
|
230
410
|
|
|
231
411
|
function summarize(actions) {
|
|
@@ -236,10 +416,22 @@ function summarize(actions) {
|
|
|
236
416
|
|
|
237
417
|
function printSummary(result, isUpdate) {
|
|
238
418
|
const { projectName, actions, dryRun } = result;
|
|
419
|
+
|
|
420
|
+
if (isUpdate && result.nothingToUpdate) {
|
|
421
|
+
console.log(
|
|
422
|
+
`\n[Nothing-to-Update] agent-harness v${PKG.version}: "${projectName}" is already at ` +
|
|
423
|
+
`v${result.priorVersion || PKG.version} and no harness-owned file changed.`
|
|
424
|
+
);
|
|
425
|
+
return;
|
|
426
|
+
}
|
|
427
|
+
|
|
239
428
|
const verb = isUpdate
|
|
240
429
|
? (dryRun ? "Would update" : "Updated")
|
|
241
430
|
: (dryRun ? "Would initialize" : "Initialized");
|
|
242
431
|
console.log(`\nagent-harness v${PKG.version} ${verb.toLowerCase()} "${projectName}"` + (dryRun ? " (dry run)" : ""));
|
|
432
|
+
if (isUpdate && result.priorVersion && result.priorVersion !== PKG.version) {
|
|
433
|
+
console.log(` harness: v${result.priorVersion} -> v${PKG.version}`);
|
|
434
|
+
}
|
|
243
435
|
for (const a of actions) console.log(` [${a.kind}] ${a.rel}`);
|
|
244
436
|
const counts = summarize(actions);
|
|
245
437
|
console.log(
|
|
@@ -249,12 +441,29 @@ function printSummary(result, isUpdate) {
|
|
|
249
441
|
);
|
|
250
442
|
}
|
|
251
443
|
|
|
444
|
+
/* Best-effort warning: when this CLI's version is not the latest published on
|
|
445
|
+
* the registry, flag it. This catches the "bumped + ran update from a local
|
|
446
|
+
* checkout, but never published" silent gap. */
|
|
447
|
+
function printUnpublishedWarning() {
|
|
448
|
+
const published = publishedLatestVersion();
|
|
449
|
+
if (published == null || published === "" || published === PKG.version) return;
|
|
450
|
+
const cmp = compareVersions(PKG.version, published);
|
|
451
|
+
if (cmp <= 0) return;
|
|
452
|
+
console.log(
|
|
453
|
+
`\n WARNING: this CLI is v${PKG.version}, but the latest version published on ` +
|
|
454
|
+
`npm is v${published}. If you ran this from a local checkout, the changes were ` +
|
|
455
|
+
`NOT published — Publish first, then \`update\` other projects, or they will ` +
|
|
456
|
+
`branch off an older harness.`
|
|
457
|
+
);
|
|
458
|
+
}
|
|
459
|
+
|
|
252
460
|
function printNextSteps() {
|
|
253
461
|
console.log(" Next steps:");
|
|
254
|
-
console.log(" 1. Fill in `.agents/memory/domain-map.md
|
|
255
|
-
console.log("
|
|
256
|
-
console.log(
|
|
257
|
-
console.log(
|
|
462
|
+
console.log(" 1. Fill in `.agents/memory/domain-map.md`, `.agents/memory/stack-versions.md`,");
|
|
463
|
+
console.log(" and `.agents/memory/flow-map.md` (register cross-surface flows so");
|
|
464
|
+
console.log(" test:qa-plan enforces their sibling-surface/variant closure).");
|
|
465
|
+
console.log(" 2. Harness gate scripts (`test:dispatch`, `test:governance`, `test:qa-plan`) were");
|
|
466
|
+
console.log(' auto-wired into package.json "scripts".');
|
|
258
467
|
console.log(" 3. Restart your agent CLI (config is read once at startup).\n");
|
|
259
468
|
}
|
|
260
469
|
|
|
@@ -263,7 +472,12 @@ function printUsage() {
|
|
|
263
472
|
`agent-harness v${PKG.version}\n\n` +
|
|
264
473
|
`Usage:\n` +
|
|
265
474
|
` agent-harness init [--target <dir>] [--name <project>] [--domain <desc>] [--yes] [--dry-run]\n` +
|
|
266
|
-
` agent-harness update [--target <dir>] [--dry-run]\n` +
|
|
475
|
+
` agent-harness update [--target <dir>] [--dry-run] [--no-commit] [--no-push]\n` +
|
|
476
|
+
`\n` +
|
|
477
|
+
` update auto-commits only the harness files it changes (chore(harness): ...) and\n` +
|
|
478
|
+
` pushes to origin by default. Use --no-commit to skip commit+push, or --no-push\n` +
|
|
479
|
+
` to commit but not push. Your unrelated uncommitted work is never staged.\n` +
|
|
480
|
+
`\n` +
|
|
267
481
|
` agent-harness list\n` +
|
|
268
482
|
` agent-harness --version | -v\n` +
|
|
269
483
|
` agent-harness --help | -h\n`
|
|
@@ -293,6 +507,8 @@ async function init(target, flags) {
|
|
|
293
507
|
yes: !!flags["--yes"],
|
|
294
508
|
dryRun: !!flags["--dry-run"],
|
|
295
509
|
isUpdate: false,
|
|
510
|
+
commit: false,
|
|
511
|
+
push: false,
|
|
296
512
|
});
|
|
297
513
|
printSummary(result, false);
|
|
298
514
|
if (!result.dryRun) printNextSteps();
|
|
@@ -307,8 +523,30 @@ async function update(target, flags) {
|
|
|
307
523
|
yes: true,
|
|
308
524
|
dryRun: !!flags["--dry-run"],
|
|
309
525
|
isUpdate: true,
|
|
526
|
+
commit: !flags["--no-commit"],
|
|
527
|
+
push: !flags["--no-commit"] && !flags["--no-push"],
|
|
310
528
|
});
|
|
311
529
|
printSummary(result, true);
|
|
530
|
+
printUnpublishedWarning();
|
|
531
|
+
printCommitResult(result);
|
|
532
|
+
}
|
|
533
|
+
|
|
534
|
+
function printCommitResult(result) {
|
|
535
|
+
const c = result.commitResult;
|
|
536
|
+
if (!c) return;
|
|
537
|
+
console.log("");
|
|
538
|
+
if (c.committed) {
|
|
539
|
+
console.log(" committed harness files: " + c.staged.join(", "));
|
|
540
|
+
console.log(
|
|
541
|
+
c.pushed === true
|
|
542
|
+
? " pushed to origin."
|
|
543
|
+
: c.pushed === null
|
|
544
|
+
? " push skipped (no --push / no remote)."
|
|
545
|
+
: " WARNING: commit made but push FAILED."
|
|
546
|
+
);
|
|
547
|
+
} else {
|
|
548
|
+
console.log(" no harness auto-commit: " + (c.skipped.join("; ") || "no-op"));
|
|
549
|
+
}
|
|
312
550
|
}
|
|
313
551
|
|
|
314
552
|
function parseArgs(argv) {
|
|
@@ -317,6 +555,7 @@ function parseArgs(argv) {
|
|
|
317
555
|
const a = argv[i];
|
|
318
556
|
if (a === "--target" || a === "--name" || a === "--domain") flags[a] = argv[++i];
|
|
319
557
|
else if (a === "--dry-run" || a === "--yes" || a === "-y") flags[a] = true;
|
|
558
|
+
else if (a === "--no-commit" || a === "--no-push") flags[a] = true;
|
|
320
559
|
}
|
|
321
560
|
return flags;
|
|
322
561
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@dombaras/agent-harness",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.6",
|
|
4
4
|
"description": "Reusable multi-agent harness for AI-assisted development: personas, skills, operating rules, model routing, and QA gates. Deploy into any project with `npx @dombaras/agent-harness init`.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"agent-harness": "bin/agent-harness.js"
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
],
|
|
17
17
|
"scripts": {
|
|
18
18
|
"test": "node --test",
|
|
19
|
-
"lint": "node --check bin/agent-harness.js && node --check templates/scripts/qa/check-dispatch-config.js && node --check templates/scripts/qa/governance.js && node --check templates/scripts/qa/check-qa-scripts.js"
|
|
19
|
+
"lint": "node --check bin/agent-harness.js && node --check templates/scripts/qa/check-dispatch-config.js && node --check templates/scripts/qa/governance.js && node --check templates/scripts/qa/check-qa-plan.js && node --check templates/scripts/qa/check-qa-scripts.js"
|
|
20
20
|
},
|
|
21
21
|
"keywords": [
|
|
22
22
|
"opencode",
|
|
@@ -20,7 +20,7 @@ The canonical rulebook. The always-loaded summary is `.agents/rules/00-operating
|
|
|
20
20
|
- **Progressive Hydration & Cold-Start**: Never render empty states while waiting on async auth/sync. Hydrate from local cache so the UI renders immediately.
|
|
21
21
|
- **The 5 Essential UI States**: Every screen/list component must handle Ideal, Empty (with CTA), Loading (skeleton), Error (retry), and Partial/single-item states.
|
|
22
22
|
- **Error-Handling Contract (Mobile)**: Every `catch` block in a native screen must (1) fire an error haptic, (2) set a user-visible error state that renders inline — never `console.warn` alone, and (3) provide a retry or dismissal path. Follow the reference pattern in `.agents/skills/mobile-engineer/SKILL.md`.
|
|
23
|
-
- **Cross-Screen Consistency ("Fix One, Fix All")**: When you implement a UX pattern in one screen, grep for analogous screens and apply it globally; if scope is too large, log a deferred item in the handoff with exact files/patterns. A local-only fix is a regression-in-waiting.
|
|
23
|
+
- **Cross-Screen Consistency ("Fix One, Fix All")**: When you implement a UX pattern in one screen, grep for analogous screens and apply it globally; if scope is too large, log a deferred item in the handoff with exact files/patterns. A local-only fix is a regression-in-waiting. Mechanically enforced by `test:qa-plan` via `.agents/memory/flow-map.md`: a change that touches a registered flow must address EVERY sibling surface (other codebases like `mobile/`, other layers) and every declared optional variant in the QA-plan audit — covered `->`, or `audited:` with a reason (including `deferred — logged in handoff`). A sibling the plan does not name fails the gate.
|
|
24
24
|
- **Bidirectional & RTL**: Use logical properties (`ps-*`, `pe-*`, `ms-*`, `me-*`, `text-start`, `text-end`) never physical ones (`pl-*`, `left-*`); use inverted horizontal lists / direction-aware layout in native RTL; pass dynamic `dir` to UI primitives.
|
|
25
25
|
- **Legacy Pruning & Physical Walkthrough**: When introducing a new flow/paradigm, explicitly audit and PRUNE obsolete mechanisms that contradict it — never bolt new steps on top of removed ones. Trace every physical/interaction flow step-by-step from each actor's perspective to eliminate unnecessary friction.
|
|
26
26
|
|
|
@@ -55,6 +55,15 @@ The canonical rulebook. The always-loaded summary is `.agents/rules/00-operating
|
|
|
55
55
|
| Data ingestion / entity resolution | `data-engineer` |
|
|
56
56
|
| Deployment / cron / secrets / build+release | `devops-engineer` |
|
|
57
57
|
| Session wrap-up / handoff | `handoff` |
|
|
58
|
+
- **QA planning is never optional (no silent skip)**: every change that touches
|
|
59
|
+
code paths MUST get a QA plan before it is verified. The plan is authored by
|
|
60
|
+
`qa-architect` (thinker). If `qa-architect` is not dispatched — for ANY reason,
|
|
61
|
+
including dispatch failure or simply not invoking it — the **main orchestrator
|
|
62
|
+
performs the qa-architect role itself** and writes the same coverage plan into
|
|
63
|
+
`.agents/memory/qa-plan.md`. The QA plan/copy escalation ladder NEVER leaves a
|
|
64
|
+
code change unplanned: `qa-architect` -> orchestrator fallback -> (never a silent
|
|
65
|
+
skip). `npm run test:qa-plan` enforces that every changed code path has a covering
|
|
66
|
+
assertion or a waivered reason.
|
|
58
67
|
- **Waivers** (the only way to skip a persona): a persona may be skipped only when (a) the change is fully covered by an automated gate on push or in the QA tiers, AND (b) the skip is pre-audited in `.agents/memory/` with a cited pointer. Log every waiver as `waived: <persona>` with `reason: <gate|pointer>`.
|
|
59
68
|
- **Dispatch failure ladder** (never silent): retry once (resume the same `task_id`); if it still fails, do the work inline and log `degraded: <persona> model: <reason>`; never silently skip.
|
|
60
69
|
- **Bound every dispatch (task spec)** — never hand a persona a bare metric ("get under N lines", "type everything"). Each dispatch prompt carries: `Objective` (one deliverable) → `Owned files` (exact paths) → `Read-only files` → `Shared contracts (owns|consumes)` → `Done = <gate command + observable metric>` → `Out-of-scope`.
|
|
@@ -70,7 +79,8 @@ The canonical rulebook. The always-loaded summary is `.agents/rules/00-operating
|
|
|
70
79
|
- Never run heavy regression suites blindly for small changes.
|
|
71
80
|
- Split QA personas: `qa-architect` (thinks — inspects the diff, audits gaps, selects the minimal tier, authors progression tests) vs `qa-runner` (does — executes the chosen tier, reports pass/fail verbatim). The thinker never runs; the doer never designs.
|
|
72
81
|
- **Static ≠ Runtime rule**: never declare UI/mobile/API changes verified from static typing alone — execute a real runtime path.
|
|
73
|
-
- **
|
|
82
|
+
- **Coverage ≠ label rule**: a tier command verifies a change ONLY when its backing suite actually exercises the changed path. A suite that touches unrelated code proves nothing about this change. `npm run test:qa-plan` mechanically asserts every changed code path is covered by the QA plan (or waivered) before verification — and, via `flow-map.md`, that every touched flow's sibling surfaces and declared optional variants are addressed in the plan's audit (the "fix one, fix all" gate: a web-only diff must still say what the mobile twin does).
|
|
83
|
+
- **Tiers** (commands are **project-provided** — the target project must define them; the harness ships only `test:dispatch`, `test:governance`, and `test:qa-plan`):
|
|
74
84
|
| Tier | Scope | Command |
|
|
75
85
|
|---|---|---|
|
|
76
86
|
| 1 | UI/CSS/mobile/copy | `npm run test:quick` |
|
|
@@ -79,7 +89,7 @@ The canonical rulebook. The always-loaded summary is `.agents/rules/00-operating
|
|
|
79
89
|
| 4 | Data ingestion | the project's data E2E script |
|
|
80
90
|
| 5 | Major release / breaking refactor | `npm run test:verify` |
|
|
81
91
|
| 6 | Security / deps | `npm run test:security` |
|
|
82
|
-
- **Progression Workflow**: for new routes/endpoints/states, (1) author a dedicated assertion path, (2) run the targeted progression test, (3) graduate it into the permanent regression suite.
|
|
92
|
+
- **Progression Workflow**: for new routes/endpoints/states, (1) author a dedicated assertion path, (2) run the targeted progression test, (3) graduate it into the permanent regression suite. A change is NOT verified if the executed tier's backing suite never exercised the modified path — author a progression test for that path first (this is the "GAP" rule).
|
|
83
93
|
|
|
84
94
|
## 7. Definition of Done & Continuous Backup
|
|
85
95
|
|
|
@@ -93,7 +103,7 @@ Every completed task passes this gate, in order:
|
|
|
93
103
|
- [ ] Cross-screen audit if a new UX pattern was introduced.
|
|
94
104
|
- [ ] No modified file over ~500 lines without extracting components.
|
|
95
105
|
- [ ] No blanket file-level `eslint-disable`/`@ts-nocheck`/`@ts-ignore` suppressions (line-level only, each with a reason).
|
|
96
|
-
1. **QA tier** — `qa-architect` picks the tier (and authors progression tests
|
|
106
|
+
1. **QA tier** — `qa-architect` inspects the diff, picks the tier BY COVERAGE (not path-label), records the coverage map in `.agents/memory/qa-plan.md`, and authors progression tests for any GAP; if `qa-architect` is not dispatched the orchestrator plans in its place. `qa-runner` executes and makes it pass (§6). A change is not verified until `npm run test:qa-plan` passes: every changed code path covered or waivered, **and** every touched flow (`.agents/memory/flow-map.md`) has every sibling surface and declared variant addressed in the plan's `## Parallel-surface & variant audit`, **AND** the executed tier exercised the modified path.
|
|
97
107
|
2. **Security check** — apply `security-engineer` when the change touches data/auth/input/secrets/deps.
|
|
98
108
|
3. **Commit** — concise `feat:` / `fix:` / `refactor:` message.
|
|
99
109
|
4. **Push** — `git push origin main`.
|
|
@@ -104,6 +114,7 @@ Every completed task passes this gate, in order:
|
|
|
104
114
|
## 8. Session & Traceability Discipline
|
|
105
115
|
|
|
106
116
|
- **One session = one coherent task**; keep a short checklist as the single source of truth.
|
|
117
|
+
- **Harness-update awareness**: `npx @dombaras/agent-harness update` auto-commits the harness files it changes as `chore(harness): @dombaras/agent-harness <old> -> <new>`. If you see such a commit (or modified `.agents/**`, `scripts/qa/**`, `AGENTS.md`, `opencode.json` you didn't touch), do NOT treat it as unexplained — run `git log -1 --stat` / `git show` to see exactly which harness files changed and why, then read the rulebook. Never guess at the cause.
|
|
107
118
|
- **Planned vs. Shipped**: at session end, record a "planned → shipped → deferred" delta so aspirational docs are never mistaken for reality. Always verify against live code and schema.
|
|
108
119
|
- **Dispatch Log (mandatory)**: every wrap-up opens with `subagent → model → shipped/deferred`. An empty log is non-compliant.
|
|
109
120
|
- **Persist Continuity**: use the `handoff` skill to update `.agents/memory/handoff.md`. Project history lives in `.agents/memory/history.md`; stack facts in `.agents/memory/stack-versions.md`.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Flow map — every user-facing flow and where it lives
|
|
2
|
+
|
|
3
|
+
Project-owned (like all `.agents/memory/*`). This is the registry that turns the
|
|
4
|
+
"Fix One, Fix All" rule (`AGENTS.md` §3) into a **gate**:
|
|
5
|
+
`npm run test:qa-plan` reads this file and, for every flow your diff touches,
|
|
6
|
+
requires the QA plan to explicitly address EVERY sibling surface — the same
|
|
7
|
+
behavior implemented anywhere else: another codebase (web vs `mobile/`), another
|
|
8
|
+
layer (API route), another page — and every OPTIONAL variant (state a surface can
|
|
9
|
+
be in, e.g. "condition selector present" vs "absent") before the change may be
|
|
10
|
+
verified.
|
|
11
|
+
|
|
12
|
+
The failure mode this kills: a change to one surface (e.g. web) ships while its
|
|
13
|
+
twin in another codebase silently keeps the old behavior. The diff is web-only so
|
|
14
|
+
per-file coverage looks green, but the flow is now inconsistent — the exact
|
|
15
|
+
"web/mobile return flow" miss.
|
|
16
|
+
|
|
17
|
+
Keep it current: every time a flow gains/removes a surface or variant, update its
|
|
18
|
+
entry here in the same change.
|
|
19
|
+
|
|
20
|
+
## flow: return
|
|
21
|
+
behavior: "Mark as Returned" — return POST with an optional condition payload (condition selector)
|
|
22
|
+
surfaces: components/LendingActionDrawers.tsx, mobile/features/return/ReturnStatusSheet.tsx, app/api/transactions/[id]/return/route.ts
|
|
23
|
+
variants: condition-selector-present, condition-selector-absent
|
|
@@ -59,10 +59,14 @@ Every subagent must return a single final message with, in order:
|
|
|
59
59
|
not name a concrete model no longer pinned anywhere (doc↔config drift).
|
|
60
60
|
- `npm run test:governance` — `scripts/qa/governance.js` enforces the wrap-up
|
|
61
61
|
dispatch log.
|
|
62
|
+
- `npm run test:qa-plan` — `scripts/qa/check-qa-plan.js` enforces diff coverage:
|
|
63
|
+
every changed CODE path must have a covering assertion (or a waivered reason)
|
|
64
|
+
in `.agents/memory/qa-plan.md` before the change is verified.
|
|
62
65
|
- `scripts/qa/check-qa-scripts.js` — verifies the DoD-referenced QA tier scripts
|
|
63
66
|
(project-provided) are wired into `package.json`.
|
|
64
67
|
|
|
65
|
-
Run `test:dispatch` after editing any agent/skill path
|
|
68
|
+
Run `test:dispatch` after editing any agent/skill path, and `test:qa-plan` on any
|
|
69
|
+
change that touches code paths.
|
|
66
70
|
|
|
67
71
|
## Gotchas
|
|
68
72
|
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# QA plan — coverage-driven
|
|
2
|
+
|
|
3
|
+
One entry per SHIPPED change. Authored by `qa-architect` (thinker), or by the
|
|
4
|
+
main orchestrator as the mandatory fallback when qa-architect is not dispatched.
|
|
5
|
+
Gated mechanically by `npm run test:qa-plan` — a changed code path that is neither
|
|
6
|
+
covered nor waivered here fails, so the change cannot be declared verified.
|
|
7
|
+
|
|
8
|
+
Keep the LAST change on top. Remove the illustrative examples below; never leave
|
|
9
|
+
a real-looking path in Coverage map without the actual runtime assertion backing it.
|
|
10
|
+
|
|
11
|
+
## Change intent
|
|
12
|
+
|
|
13
|
+
One sentence naming the SEMANTIC behavior change (what the user experiences), not
|
|
14
|
+
the file list. The flow-closure half of the gate judges sibling surfaces and
|
|
15
|
+
optional variant states against this intent.
|
|
16
|
+
|
|
17
|
+
- change: return flow no longer offers a condition selector — every return POSTs without a condition payload
|
|
18
|
+
|
|
19
|
+
## Parallel-surface & variant audit
|
|
20
|
+
|
|
21
|
+
For every flow in `.agents/memory/flow-map.md` whose `surfaces:` intersect this
|
|
22
|
+
diff, address EVERY sibling surface and EVERY declared `variant` here — either
|
|
23
|
+
`->` a covering assertion or `audited:` with a reason. `test:qa-plan` fails on any
|
|
24
|
+
sibling surface or variant the plan does not NAME. A sibling that intentionally
|
|
25
|
+
keeps old behavior must say so (`audited: deferred — logged in handoff`), never be
|
|
26
|
+
silently omitted.
|
|
27
|
+
|
|
28
|
+
- components/LendingActionDrawers.tsx -> test:quick::return (click-through: Mark as Returned, no condition selector)
|
|
29
|
+
- mobile/features/return/ReturnStatusSheet.tsx audited: deferred — mobile still renders the selector; same simplification pending, logged in handoff.md
|
|
30
|
+
- app/api/transactions/[id]/return/route.ts -> test:api::return (POST without condition payload succeeds)
|
|
31
|
+
- flow: return variant: condition-selector-present audited: removed by this change — asserting absence only
|
|
32
|
+
- flow: return variant: condition-selector-absent -> test:quick::return (no-selector click-through renders no condition UI)
|
|
33
|
+
|
|
34
|
+
## Coverage map
|
|
35
|
+
|
|
36
|
+
A `Changed path -> executing test/assertion` list. Every changed CODE path needs a
|
|
37
|
+
runtime assertion that ACTUALLY exercises it (never trust a tier label — verify the
|
|
38
|
+
backing suite reaches the path). Paths may be files or directories. Lines starting
|
|
39
|
+
with `#` are ignored.
|
|
40
|
+
|
|
41
|
+
# components/return/ReturnConfirmDrawer.tsx -> test:quick::progression (click-through: Mark as Returned -> confirm -> /api/transactions/[id]/return)
|
|
42
|
+
# app/api/transactions/[id]/return/route.ts -> test:quick::progression (return POST succeeds with no condition payload)
|
|
43
|
+
|
|
44
|
+
## Waivers
|
|
45
|
+
|
|
46
|
+
Trivial diffs only (pure copy/docs, or a path whose behavior is already covered
|
|
47
|
+
elsewhere). Every waiver MUST carry a `reason:` on the same line.
|
|
48
|
+
|
|
49
|
+
# waived: src/some-screen.tsx reason: copy-only string swap, no runtime path changed
|
|
@@ -9,6 +9,7 @@ Before reading or editing any file for a task, dispatch the relevant personas vi
|
|
|
9
9
|
- **Bound every task before dispatching.** No persona gets a bare metric ("get under N lines", "type everything"). Each dispatch prompt carries a task spec: `Objective` (one deliverable) → `Owned files` (exact paths) → `Read-only files` → `Shared contracts (owns|consumes)` → `Done = <gate command + observable metric>` → `Out-of-scope`.
|
|
10
10
|
- **Parallel = disjoint.** Dispatch two personas in parallel only when their owned files AND shared contracts are disjoint; otherwise serialize and put the shared-contract owner first.
|
|
11
11
|
- **Accept, don't trust.** After a code persona reports done, the orchestrator re-runs the gate itself (`tsc --noEmit`, `lint:hooks`, `test:quick`) and greps the metric before integrating — a subagent's `Evidence` is a claim, not proof.
|
|
12
|
+
- **QA planning is never optional.** Every change touching code paths gets a QA plan (coverage map in `.agents/memory/qa-plan.md`) from `qa-architect`; if it isn't dispatched, the main orchestrator plans in its place. `npm run test:qa-plan` fails on any changed code path with no covering assertion or waiver, and on any touched flow (`.agents/memory/flow-map.md`) whose sibling surfaces / declared variants the plan doesn't address.
|
|
12
13
|
- Persona map, waivers, and the dispatch-failure ladder: `.agents/AGENTS.md` §5 and `.agents/memory/model-routing.md`.
|
|
13
14
|
- Every subagent returns the output contract (`Result` → `Evidence` → `Deferred & risks`).
|
|
14
15
|
- Wrap up with a **dispatch log** (`subagent → model → shipped/deferred`) in `.agents/memory/handoff.md`.
|
|
@@ -20,7 +21,7 @@ Before reading or editing any file for a task, dispatch the relevant personas vi
|
|
|
20
21
|
3. **5 UI states** (ideal/empty/loading/error/partial), user-visible errors (never `console.warn`-only), and RTL-safe logical props.
|
|
21
22
|
4. **Verify stack versions** before writing framework code (`.agents/memory/stack-versions.md`).
|
|
22
23
|
5. **Token discipline** — search before read, targeted reads, batch reads, don't re-read unchanged files, right-size QA.
|
|
23
|
-
6. **Static ≠ runtime** — never declare verified from `tsc` alone; execute a real runtime/API path.
|
|
24
|
+
6. **Static ≠ runtime** — never declare verified from `tsc` alone; execute a real runtime/API path that exercises the MODIFIED code — a tier label never proves coverage. `test:qa-plan` gates every changed code path to a covering assertion or waiver, and every touched flow's sibling surfaces / variants to an audit entry (`.agents/memory/flow-map.md` — "fix one, fix all").
|
|
24
25
|
7. **Commit** with `feat:`/`fix:`/`refactor:` then push to main.
|
|
25
26
|
|
|
26
27
|
## Definition of Done
|
|
@@ -29,6 +29,11 @@ dispatch it broad.
|
|
|
29
29
|
## Rules
|
|
30
30
|
|
|
31
31
|
- Read `.agents/memory/model-routing.md` for the persona → model tiers.
|
|
32
|
+
- Read `.agents/memory/flow-map.md`. If the task's behavior maps to a flow whose `surfaces:` cross
|
|
33
|
+
codebases/layers (e.g. web + `mobile/` + an API route), the dispatch plan MUST include the sibling
|
|
34
|
+
surface's persona (e.g. `mobile-engineer`) — or explicitly mark the sibling out-of-scope so
|
|
35
|
+
`qa-architect` records it as `audited: deferred` in the QA plan's flow-closure audit. A change to
|
|
36
|
+
one surface that never mentions the twin is a plan defect, not scope.
|
|
32
37
|
- Never dispatch a persona for work the main model should just do (reads, commits, integration).
|
|
33
38
|
- Prefer the most specific persona; if none clearly fits, ask the user rather than guess.
|
|
34
39
|
- Dispatch independent subagents in parallel; serialize only when one depends on another's output.
|
|
@@ -11,9 +11,28 @@ You own QA **strategy** — risk assessment, tier selection, and progression tes
|
|
|
11
11
|
## Output contract (always return)
|
|
12
12
|
|
|
13
13
|
1. **Risk assessment** — blast radius (files/subsystems touched) + risk level `LOW | MEDIUM | HIGH`.
|
|
14
|
-
2. **Selected tier + command(s)** — from the matrix below.
|
|
15
|
-
3. **
|
|
16
|
-
|
|
14
|
+
2. **Selected tier + command(s)** — from the matrix below, chosen by **coverage, not path-label** (see Diff coverage map).
|
|
15
|
+
3. **Diff coverage map (MANDATORY)** — enumerate every changed path, one entry per line, as a
|
|
16
|
+
`Changed path -> executing test/assertion` table. A tier command matches a change only when its
|
|
17
|
+
backing suite ACTUALLY exercises that path/state. Any change with no runtime assertion is a **GAP**
|
|
18
|
+
— you must author a progression test for it (Progression vs regression below). Persist the same map
|
|
19
|
+
into `.agents/memory/qa-plan.md` under `## Coverage map` (this is what `npm run test:qa-plan` gates on).
|
|
20
|
+
3b. **Flow closure audit (MANDATORY — "fix one, fix all")** — read `.agents/memory/flow-map.md`. For
|
|
21
|
+
every flow whose `surfaces:` intersect this diff, address in the plan's
|
|
22
|
+
`## Parallel-surface & variant audit` EVERY sibling surface (the same behavior living in another
|
|
23
|
+
codebase/layer — web vs `mobile/`, API route, another page) and EVERY declared `variant`. Each
|
|
24
|
+
entry is `- <surface> -> <assertion>` or `- <surface> audited: <reason>`; each variant is
|
|
25
|
+
`- flow: <name> variant: <label> -> <assertion>` or `... audited: <reason>`. A sibling that is
|
|
26
|
+
legitimately out of scope is NOT left unnamed — it is `audited: deferred — logged in handoff.md`.
|
|
27
|
+
`test:qa-plan` fails on any sibling surface or declared variant the plan does not name.
|
|
28
|
+
If `flow-map.md` has no flows, note it and proceed (the gate warns, does not block; register the
|
|
29
|
+
flow in flow-map.md when you touch its behavior).
|
|
30
|
+
4. **Progression plan** — any NEW tests to author (and where) before the run, or a statement that no new tests are needed. For every **GAP** you MUST author a progression test that renders/executes the real path (real route/API — never a shell mock) and drives it click-through from the user's perspective.
|
|
31
|
+
5. **Handoff to qa-runner** — exact commands + pass criteria.
|
|
32
|
+
|
|
33
|
+
> The rule the whole plan rests on: a tier label never proves coverage. `test:quick` at label "Tier 1
|
|
34
|
+
> components" does NOT cover a component unless its script actually exercises it. If the project's
|
|
35
|
+
> backing script for a tier doesn't reach a changed path, that's a **coverage gap**, not a passing run.
|
|
17
36
|
|
|
18
37
|
## Dynamic Test Selection Matrix
|
|
19
38
|
|
|
@@ -29,7 +48,7 @@ You own QA **strategy** — risk assessment, tier selection, and progression tes
|
|
|
29
48
|
Diff-aware planning helper: `npm run qa:plan [-- --json] [-- --base main]`.
|
|
30
49
|
|
|
31
50
|
> The tier commands above are **project-provided** — the harness ships only
|
|
32
|
-
> `test:dispatch` and `test:
|
|
51
|
+
> `test:dispatch`, `test:governance`, and `test:qa-plan`. If a referenced script is missing,
|
|
33
52
|
> route that finding to the orchestrator (`scripts/qa/check-qa-scripts.js`
|
|
34
53
|
> verifies wiring mechanically).
|
|
35
54
|
|
|
@@ -40,6 +59,19 @@ Diff-aware planning helper: `npm run qa:plan [-- --json] [-- --base main]`.
|
|
|
40
59
|
|
|
41
60
|
Author new assertion pathways in `scripts/verify-all.js` (API), `scripts/qa/routes.js` (routes), or `scripts/qa/tests/*.test.ts` (mobile/smoke), then hand the run to `qa-runner`. Once passing, the test graduates permanently into the regression baseline.
|
|
42
61
|
|
|
62
|
+
## Coverage gate (`npm run test:qa-plan`)
|
|
63
|
+
|
|
64
|
+
Every change that touches code paths MUST have a coverage map persisted in
|
|
65
|
+
`.agents/memory/qa-plan.md` (or a scoped `waived:` entry per path) before the run can be declared
|
|
66
|
+
verified — AND a flow-closure audit of every touched flow's sibling surfaces and declared variants
|
|
67
|
+
(`.agents/memory/flow-map.md`). The mechanical gate checks the live `git diff` against both and
|
|
68
|
+
fails on any uncovered code path or unaddressed sibling/variant. If you are not dispatched for a
|
|
69
|
+
change, the main orchestrator performs your role and writes the same plan itself — a change with
|
|
70
|
+
code-path edits and no QA plan is a rules violation.
|
|
71
|
+
|
|
43
72
|
## Iron law
|
|
44
73
|
|
|
45
|
-
Never declare a change verified from static typing alone — every tier must execute a real runtime path
|
|
74
|
+
Never declare a change verified from static typing alone — every tier must execute a real runtime path
|
|
75
|
+
that exercises the MODIFIED code. A suite that touches unrelated code proves nothing about this change.
|
|
76
|
+
If the suite does not exercise the modified path, author a progression test first and cover the GAP in
|
|
77
|
+
the coverage map.
|
|
@@ -17,6 +17,7 @@ You **execute**. A `qa-architect` (or the orchestrator) tells you which tier to
|
|
|
17
17
|
## Responsibilities
|
|
18
18
|
|
|
19
19
|
- Run the tier you are given: `npm run test:quick` / `test:routes` / `test:api` / `test:verify` / `test:security`.
|
|
20
|
+
- Run the diff-coverage gate: `npm run test:qa-plan` and report uncovered paths verbatim.
|
|
20
21
|
- Data/catalog E2E journey (100 records): `npx ts-node scripts/test-catalog.ts`.
|
|
21
22
|
- Catalog test-data cleanup when asked (via the catalog reset endpoint/scripts).
|
|
22
23
|
|
|
@@ -12,7 +12,8 @@ You are the {{PROJECT_NAME}} QA Architect. Read and follow the complete persona
|
|
|
12
12
|
|
|
13
13
|
## Scope & integrity (non-negotiable)
|
|
14
14
|
|
|
15
|
-
- Edit ONLY files your role owns (progression-test authoring). `opencode.json`, `.agents/memory/*`, `.agents/rules/*`, other personas' files, and unrelated code are READ-ONLY absent an explicit orchestrator grant.
|
|
15
|
+
- Edit ONLY files your role owns (progression-test authoring + `.agents/memory/qa-plan.md`). `opencode.json`, other `.agents/memory/*`, `.agents/rules/*`, other personas' files, and unrelated code are READ-ONLY absent an explicit orchestrator grant.
|
|
16
16
|
- You design and author tests only — route execution to `qa-runner`. Never fabricate dispatch-log or handoff entries that did not actually occur.
|
|
17
|
+
- Persist your plan's `## Coverage map` into `.agents/memory/qa-plan.md` — `npm run test:qa-plan` gates every changed code path against it. A change with code-path edits and no coverage map is not verifiable.
|
|
17
18
|
|
|
18
19
|
Return your final message in this exact order: **Result** (what shipped / decided) -> **Evidence** (files changed, commands run, observed output) -> **Deferred & risks** (follow-ups the orchestrator must handle). Keep it under ~15 lines.
|
package/templates/AGENTS.md
CHANGED
|
@@ -5,6 +5,7 @@ The always-loaded rules summary lives in [`.agents/rules/00-operating.md`](.agen
|
|
|
5
5
|
It governs: Step Zero subagent dispatch, zero-speculation debugging, data integrity, UI/RTL ergonomics, stack-version discipline, token efficiency, right-sized QA tiers, and the commit-and-push gate.
|
|
6
6
|
|
|
7
7
|
- **Dispatch personas** via the `task` tool (see `.agents/rules/00-operating.md` §Step Zero); consult the relevant `.agents/skills/<persona>/SKILL.md`.
|
|
8
|
+
- **Every change touching code paths gets a QA plan** (`.agents/memory/qa-plan.md`) from `qa-architect` — if it isn't dispatched, the main orchestrator plans in its place. `npm run test:qa-plan` fails on any changed code path with no covering assertion or waiver, and on any touched flow (`.agents/memory/flow-map.md`) whose sibling surfaces / declared variants the plan doesn't address.
|
|
8
9
|
- **At session start**, read `.agents/memory/locations.md` and `.agents/memory/model-routing.md`.
|
|
9
10
|
|
|
10
11
|
> Deployed and maintained by `@dombaras/agent-harness` (`npx @dombaras/agent-harness init` / `update`). Do not hand-edit harness-owned files — regenerate them and commit the deltas.
|
|
@@ -0,0 +1,321 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
"use strict";
|
|
3
|
+
/*
|
|
4
|
+
* test:qa-plan — diff-coverage gate (harness-shipped).
|
|
5
|
+
*
|
|
6
|
+
* The anti-verification-from-unrelated-suites gate. A tier label never proves
|
|
7
|
+
* coverage; this gate forces a change's QA plan to actually cover the code it
|
|
8
|
+
* modified — otherwise "verified" is meaningless.
|
|
9
|
+
*
|
|
10
|
+
* Two stages:
|
|
11
|
+
*
|
|
12
|
+
* 1. Coverage map — inspects the live git diff and requires every changed CODE
|
|
13
|
+
* path to appear in the QA plan's `## Coverage map` (persisted by
|
|
14
|
+
* qa-architect, or by the main orchestrator as the qa-architect fallback)
|
|
15
|
+
* BEFORE a change may be declared verified. Trivial diffs (copy/docs/not
|
|
16
|
+
* code) and waivered paths are exempt.
|
|
17
|
+
*
|
|
18
|
+
* 2. Flow closure — reads `.agents/memory/flow-map.md` (project-owned): a
|
|
19
|
+
* registry of user-facing flows, each listing the SURFACES that implement the
|
|
20
|
+
* same behavior (web / mobile / API — any codebase/layer) and its optional
|
|
21
|
+
* VARIANTS. Any flow whose surfaces intersect this diff must have every
|
|
22
|
+
* sibling surface and every declared variant addressed in the plan's
|
|
23
|
+
* `## Parallel-surface & variant audit` (covered `->` or `audited:` with a
|
|
24
|
+
* reason). This is the gate that makes "Fix One, Fix All" mechanical: a
|
|
25
|
+
* web-only diff can no longer silently leave the mobile twin on old behavior.
|
|
26
|
+
* If flow-map.md is absent or lists no flows, the stage warns but never fails
|
|
27
|
+
* (backward compatible — projects opt into flow closure by filling it in).
|
|
28
|
+
*
|
|
29
|
+
* node scripts/qa/check-qa-plan.js # diff vs HEAD
|
|
30
|
+
* node scripts/qa/check-qa-plan.js --base main # diff vs a ref
|
|
31
|
+
* node scripts/qa/check-qa-plan.js --strict # non-code changes too
|
|
32
|
+
*
|
|
33
|
+
* Exit 0 on: no code-path changes, full coverage, valid waivers, full flow closure.
|
|
34
|
+
* Exit 1 on: a changed code path with no covering assertion/waiver, or a flow
|
|
35
|
+
* sibling surface / declared variant the QA plan does not address.
|
|
36
|
+
*/
|
|
37
|
+
const fs = require("fs");
|
|
38
|
+
const path = require("path");
|
|
39
|
+
const { execFileSync } = require("child_process");
|
|
40
|
+
|
|
41
|
+
const root = path.resolve(__dirname, "..", "..");
|
|
42
|
+
|
|
43
|
+
const argv = process.argv.slice(2);
|
|
44
|
+
const base = (argv.find((a) => a.startsWith("--base=")) || "").split("=")[1] || "HEAD";
|
|
45
|
+
const strict = argv.includes("--strict");
|
|
46
|
+
|
|
47
|
+
// Paths that are tooling/harness, not application code under test. A change
|
|
48
|
+
// that only touches these needs no runtime QA plan (they are guarded by
|
|
49
|
+
// test:dispatch / test:governance themselves).
|
|
50
|
+
const TOOLING = [
|
|
51
|
+
"opencode.json",
|
|
52
|
+
".harness.json",
|
|
53
|
+
".opencode/",
|
|
54
|
+
".agents/",
|
|
55
|
+
"scripts/qa/",
|
|
56
|
+
".github/",
|
|
57
|
+
"package.json",
|
|
58
|
+
];
|
|
59
|
+
// Non-code files never need a runtime path (copy/docs/styling/data-flat).
|
|
60
|
+
const NON_CODE = /\.(md|mdx|png|jpg|jpeg|gif|svg|webp|ico|json|lock|txt|env|yml|yaml|toml|csv)$/i;
|
|
61
|
+
|
|
62
|
+
const CODE = /\.(ts|tsx|js|jsx|css|scss|sass|prisma|sql)$/i;
|
|
63
|
+
|
|
64
|
+
function norm(p) {
|
|
65
|
+
return p.replace(/\\/g, "/");
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
function gitChangedFiles() {
|
|
69
|
+
let files = [];
|
|
70
|
+
try {
|
|
71
|
+
const out = execFileSync("git", ["diff", "--name-only", base, "--"], {
|
|
72
|
+
cwd: root,
|
|
73
|
+
encoding: "utf8",
|
|
74
|
+
});
|
|
75
|
+
files = out.split(/\r?\n/).filter(Boolean);
|
|
76
|
+
} catch (e) {
|
|
77
|
+
// 128 = not a git repo / bad ref / no base history (e.g. empty repo).
|
|
78
|
+
if (e.status === 128) return null;
|
|
79
|
+
throw e;
|
|
80
|
+
}
|
|
81
|
+
// include untracked files so brand-new sources can't silently skip the gate
|
|
82
|
+
try {
|
|
83
|
+
const untracked = execFileSync(
|
|
84
|
+
"git",
|
|
85
|
+
["ls-files", "--others", "--exclude-standard", "--", root],
|
|
86
|
+
{ cwd: root, encoding: "utf8" }
|
|
87
|
+
)
|
|
88
|
+
.split(/\r?\n/)
|
|
89
|
+
.filter(Boolean);
|
|
90
|
+
files = [...new Set([...files, ...untracked])];
|
|
91
|
+
} catch (_) {
|
|
92
|
+
/* ls-files is best-effort */
|
|
93
|
+
}
|
|
94
|
+
return files.map(norm);
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
function isTooling(p) {
|
|
98
|
+
const n = norm(p);
|
|
99
|
+
return TOOLING.some((t) => n === t || n.startsWith(t));
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
/* Parse the QA plan file into { covered:Set, waivers:Set } of normalized paths.
|
|
103
|
+
* Coverage map format (one entry per changed path, under `## Coverage map`):
|
|
104
|
+
* - components/return/ReturnConfirmDrawer.tsx -> test:quick::progression (click-through)
|
|
105
|
+
* Waivers live anywhere and carry a reason on the same line:
|
|
106
|
+
* waived: components/nav.tsx reason: pure-copy change, no runtime path
|
|
107
|
+
*/
|
|
108
|
+
function parsePlan(file) {
|
|
109
|
+
const out = { covered: new Set(), waivers: new Set() };
|
|
110
|
+
if (!fs.existsSync(file)) return out;
|
|
111
|
+
const src = fs.readFileSync(file, "utf8");
|
|
112
|
+
const lines = src.split(/\r?\n/);
|
|
113
|
+
|
|
114
|
+
// Coverage map section only
|
|
115
|
+
let inMap = false;
|
|
116
|
+
for (const raw of lines) {
|
|
117
|
+
const line = raw.trim();
|
|
118
|
+
if (!line) continue;
|
|
119
|
+
if (/^#{1,6}\s*Coverage map\s*$/i.test(line)) {
|
|
120
|
+
inMap = true;
|
|
121
|
+
continue;
|
|
122
|
+
}
|
|
123
|
+
if (/^[#;/]/.test(line)) continue;
|
|
124
|
+
if (inMap && /^#{1,6}\s/.test(line)) inMap = false;
|
|
125
|
+
if (!inMap) continue;
|
|
126
|
+
|
|
127
|
+
const m = line.match(/^[-*]\s*["']?([^\s"'>]+)/);
|
|
128
|
+
if (m) out.covered.add(norm(m[1]));
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
// Waivers (anywhere) — path then reason
|
|
132
|
+
for (const raw of lines) {
|
|
133
|
+
const line = raw.trim();
|
|
134
|
+
if (!/\bwaived:\s*\S/i.test(line)) continue;
|
|
135
|
+
const p = (line.match(/\bwaived:\s*["']?([^\s"',]+)/i) || [])[1];
|
|
136
|
+
if (p && /\breason:\s*\S/i.test(line)) out.waivers.add(norm(p));
|
|
137
|
+
}
|
|
138
|
+
return out;
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
function isCovered(pathi, covered, waivers) {
|
|
142
|
+
if (covered.has(pathi) || waivers.has(pathi)) return true;
|
|
143
|
+
// directory-prefix coverage (a plan may cover a whole folder)
|
|
144
|
+
for (const c of covered) if (pathi.startsWith(c.replace(/\/?$/, "/"))) return true;
|
|
145
|
+
for (const w of waivers) if (pathi.startsWith(w.replace(/\/?$/, "/"))) return true;
|
|
146
|
+
return false;
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/* Parse `.agents/memory/flow-map.md` into [{ name, surfaces:[], variants:[] }].
|
|
150
|
+
* Format (one flow per `## flow:` header):
|
|
151
|
+
* ## flow: return
|
|
152
|
+
* behavior: "Mark as Returned" — optional condition payload (condition selector)
|
|
153
|
+
* surfaces: components/LendingActionDrawers.tsx, mobile/features/return/ReturnStatusSheet.tsx, app/api/transactions/[id]/return/route.ts
|
|
154
|
+
* variants: condition-selector-present, condition-selector-absent
|
|
155
|
+
*/
|
|
156
|
+
function parseFlowMap(file) {
|
|
157
|
+
const flows = [];
|
|
158
|
+
if (!fs.existsSync(file)) return flows;
|
|
159
|
+
let cur = null;
|
|
160
|
+
for (const raw of fs.readFileSync(file, "utf8").split(/\r?\n/)) {
|
|
161
|
+
const line = raw.trim();
|
|
162
|
+
if (!line) continue;
|
|
163
|
+
const h = line.match(/^#{1,6}\s*flow:\s*(.+)$/i);
|
|
164
|
+
if (h) {
|
|
165
|
+
cur = { name: h[1].trim(), surfaces: [], variants: [] };
|
|
166
|
+
flows.push(cur);
|
|
167
|
+
continue;
|
|
168
|
+
}
|
|
169
|
+
if (/^[#;]/.test(line)) continue;
|
|
170
|
+
if (!cur) continue;
|
|
171
|
+
const s = line.match(/^surfaces?:\s*(.+)$/i);
|
|
172
|
+
if (s) cur.surfaces = s[1].split(",").map((x) => norm(x.trim())).filter(Boolean);
|
|
173
|
+
const v = line.match(/^variants?:\s*(.+)$/i);
|
|
174
|
+
if (v) cur.variants = v[1].split(",").map((x) => x.trim()).filter(Boolean);
|
|
175
|
+
}
|
|
176
|
+
return flows.filter((f) => f.name && f.surfaces.length > 0);
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
/* Parse the plan's `## Parallel-surface & variant audit` entries.
|
|
180
|
+
* Entry forms:
|
|
181
|
+
* - <surface> -> <assertion>
|
|
182
|
+
* - <surface> audited: <reason>
|
|
183
|
+
* - flow: <name> variant: <label> -> <assertion>
|
|
184
|
+
* - flow: <name> variant: <label> audited: <reason>
|
|
185
|
+
*/
|
|
186
|
+
function parseAudit(file) {
|
|
187
|
+
const entries = [];
|
|
188
|
+
if (!fs.existsSync(file)) return entries;
|
|
189
|
+
let inAudit = false;
|
|
190
|
+
for (const raw of fs.readFileSync(file, "utf8").split(/\r?\n/)) {
|
|
191
|
+
const line = raw.trim();
|
|
192
|
+
if (!line) continue;
|
|
193
|
+
if (/^#{1,6}\s*Parallel[ -]surface.{0,40}(?:&\s*|and\s+).{0,40}audit\s*$/i.test(line)) {
|
|
194
|
+
inAudit = true;
|
|
195
|
+
continue;
|
|
196
|
+
}
|
|
197
|
+
if (/^[#;]/.test(line)) continue;
|
|
198
|
+
if (inAudit && /^#{1,6}\s/.test(line)) inAudit = false;
|
|
199
|
+
if (!inAudit) continue;
|
|
200
|
+
|
|
201
|
+
const vm = line.match(/^[-*]\s*flow:\s*["']?([^\s"']+)\s+variant:\s*["']?([^\s"']+)\s+(audited:.+|->\s*.+)$/i);
|
|
202
|
+
if (vm) {
|
|
203
|
+
entries.push({ flow: vm[1], variant: vm[2], surface: null, disposition: vm[3].trim() });
|
|
204
|
+
continue;
|
|
205
|
+
}
|
|
206
|
+
const pm = line.match(/^[-*]\s*["']?([^\s"'>]+)\s+(audited:.+|->\s*.+)$/);
|
|
207
|
+
if (pm) entries.push({ surface: norm(pm[1]), disposition: pm[2].trim() });
|
|
208
|
+
}
|
|
209
|
+
return entries;
|
|
210
|
+
}
|
|
211
|
+
|
|
212
|
+
/* True when path a and path b are the same file/dir or one nests under the other
|
|
213
|
+
* as a directory prefix — lenient matching for surfaces vs coverage/audit paths. */
|
|
214
|
+
function matches(a, b) {
|
|
215
|
+
const A = norm(a).replace(/\/$/, "");
|
|
216
|
+
const B = norm(b).replace(/\/$/, "");
|
|
217
|
+
return A === B || A.startsWith(B + "/") || B.startsWith(A + "/");
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
const failures = [];
|
|
221
|
+
const pass = (m) => console.log(" \u2713 " + m);
|
|
222
|
+
const fail = (m) => {
|
|
223
|
+
failures.push(m);
|
|
224
|
+
console.log(" \u2717 " + m);
|
|
225
|
+
};
|
|
226
|
+
|
|
227
|
+
console.log("diff-coverage gate \u2014 QA plan covers every changed code path");
|
|
228
|
+
|
|
229
|
+
const changed = gitChangedFiles();
|
|
230
|
+
if (changed == null) {
|
|
231
|
+
console.log(" \u26a0 not a git repo or no base history \u2014 coverage gate skipped");
|
|
232
|
+
console.log("\nRESULT: skipped (0 failures)");
|
|
233
|
+
process.exit(0);
|
|
234
|
+
}
|
|
235
|
+
if (changed.length === 0) {
|
|
236
|
+
console.log(" \u2713 no diff vs " + base + " \u2014 nothing to cover");
|
|
237
|
+
console.log("\nRESULT: green");
|
|
238
|
+
process.exit(0);
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
const plan = path.join(root, ".agents", "memory", "qa-plan.md");
|
|
242
|
+
const audit = parseAudit(plan);
|
|
243
|
+
const { covered, waivers } = parsePlan(plan);
|
|
244
|
+
|
|
245
|
+
console.log(` checked against: ${path.relative(root, plan)}`);
|
|
246
|
+
|
|
247
|
+
const changedCode = [];
|
|
248
|
+
for (const f of changed) {
|
|
249
|
+
if (isTooling(f)) {
|
|
250
|
+
pass(`${f} (harness/tooling \u2014 self-guarded)`);
|
|
251
|
+
continue;
|
|
252
|
+
}
|
|
253
|
+
if (NON_CODE.test(f) && !strict) {
|
|
254
|
+
pass(`${f} (non-code \u2014 no runtime path required)`);
|
|
255
|
+
continue;
|
|
256
|
+
}
|
|
257
|
+
if (!CODE.test(f)) {
|
|
258
|
+
pass(`${f} (not a code path)`);
|
|
259
|
+
continue;
|
|
260
|
+
}
|
|
261
|
+
changedCode.push(f);
|
|
262
|
+
if (isCovered(f, covered, waivers)) pass(`${f} covered by QA plan`);
|
|
263
|
+
else
|
|
264
|
+
fail(
|
|
265
|
+
`${f} NOT covered \u2014 add it to the \`## Coverage map\` in qa-plan.md (or a waivered entry with a reason)`
|
|
266
|
+
);
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
// ------------------------------------------------------------ flow closure
|
|
270
|
+
const flowMap = parseFlowMap(path.join(root, ".agents", "memory", "flow-map.md"));
|
|
271
|
+
const hitFlows = new Set();
|
|
272
|
+
for (const f of flowMap) {
|
|
273
|
+
if (changedCode.some((c) => f.surfaces.some((s) => matches(c, s)))) hitFlows.add(f);
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
if (!flowMap.length) {
|
|
277
|
+
console.log(
|
|
278
|
+
" \u26a0 flow-map.md has no flows \u2014 sibling/variant closure NOT enforced (add flows to secure \"Fix One, Fix All\")"
|
|
279
|
+
);
|
|
280
|
+
} else if (hitFlows.size) {
|
|
281
|
+
console.log("flow-closure gate \u2014 every sibling surface & variant of a touched flow is addressed");
|
|
282
|
+
for (const f of flowMap) {
|
|
283
|
+
if (!hitFlows.has(f)) {
|
|
284
|
+
pass(`flow "${f.name}" not touched \u2014 no sibling requirement`);
|
|
285
|
+
continue;
|
|
286
|
+
}
|
|
287
|
+
// Sibling surfaces: every declared surface must be addressed by the diff
|
|
288
|
+
// (as a changed path) or by a named audit entry, or audited with a reason.
|
|
289
|
+
for (const s of f.surfaces) {
|
|
290
|
+
if (changedCode.some((c) => matches(c, s))) {
|
|
291
|
+
continue; // the changed path itself is required in the Coverage map above
|
|
292
|
+
}
|
|
293
|
+
const addressed = audit.some((e) => e.surface && matches(e.surface, s));
|
|
294
|
+
if (addressed) pass(`flow "${f.name}": sibling \`${s}\` addressed in audit`);
|
|
295
|
+
else
|
|
296
|
+
fail(
|
|
297
|
+
`flow "${f.name}": sibling surface \`${s}\` not addressed \u2014 add it to \`## Parallel-surface & variant audit\` as \`-> <assertion>\` or \`audited: <reason>\``
|
|
298
|
+
);
|
|
299
|
+
}
|
|
300
|
+
// Declared variants: each must be named with a disposition for this flow.
|
|
301
|
+
for (const v of f.variants) {
|
|
302
|
+
const addressed = audit.some((e) => e.flow && e.variant && e.flow === f.name && e.variant === v);
|
|
303
|
+
if (addressed) pass(`flow "${f.name}": variant \`${v}\` addressed`);
|
|
304
|
+
else
|
|
305
|
+
fail(
|
|
306
|
+
`flow "${f.name}": declared variant \`${v}\` not addressed \u2014 add \`flow: ${f.name} variant: ${v} -> <assertion>\` or \`... audited: <reason>\` to \`## Parallel-surface & variant audit\``
|
|
307
|
+
);
|
|
308
|
+
}
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
|
|
312
|
+
if (!changedCode.length) {
|
|
313
|
+
console.log(" \u2713 no application code paths changed");
|
|
314
|
+
}
|
|
315
|
+
console.log(
|
|
316
|
+
"\nRESULT: " +
|
|
317
|
+
(failures.length
|
|
318
|
+
? `${failures.length} FAILURE(S) \u2014 every changed code path needs a covering assertion or a waivered reason; every touched flow's sibling surfaces & variants must be addressed in the audit`
|
|
319
|
+
: "green \u2014 all changed code paths have a covering QA plan and all touched flows are closed")
|
|
320
|
+
);
|
|
321
|
+
process.exit(failures.length ? 1 : 0);
|
|
@@ -22,6 +22,7 @@ const strict = process.argv.includes("--strict");
|
|
|
22
22
|
const EXPECTED = {
|
|
23
23
|
"test:dispatch": "harness — persona model-pin preflight (shipped)",
|
|
24
24
|
"test:governance": "harness — session-governance gate (shipped)",
|
|
25
|
+
"test:qa-plan": "harness — diff-coverage + flow-closure gate (changed code paths covered; touched flows' sibling surfaces/variants audited)",
|
|
25
26
|
"test:quick": "Tier 1 — types + translations + mobile smoke",
|
|
26
27
|
"test:routes": "Tier 2 — route/page render",
|
|
27
28
|
"test:api": "Tier 3 — API/ORM edge cases",
|