@dombaras/agent-harness 0.1.1 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -4
- package/bin/agent-harness.js +205 -7
- package/package.json +2 -2
- package/templates/.agents/AGENTS.md +21 -5
- package/templates/.agents/memory/flow-map.md +23 -0
- package/templates/.agents/memory/model-routing.md +12 -2
- package/templates/.agents/memory/qa-plan.md +49 -0
- package/templates/.agents/rules/00-operating.md +6 -2
- package/templates/.agents/skills/frontend-engineer/SKILL.md +1 -0
- package/templates/.agents/skills/mobile-engineer/SKILL.md +4 -0
- package/templates/.agents/skills/planner/SKILL.md +19 -3
- package/templates/.agents/skills/qa-architect/SKILL.md +37 -5
- package/templates/.agents/skills/qa-runner/SKILL.md +1 -0
- package/templates/.opencode/agents/data-engineer.md +9 -0
- package/templates/.opencode/agents/devops-engineer.md +9 -0
- package/templates/.opencode/agents/diagnostics-expert.md +9 -0
- package/templates/.opencode/agents/frontend-engineer.md +9 -0
- package/templates/.opencode/agents/mobile-engineer.md +9 -0
- package/templates/.opencode/agents/qa-architect.md +2 -1
- package/templates/.opencode/agents/security-engineer.md +9 -0
- package/templates/.opencode/agents/system-architect.md +9 -0
- package/templates/.opencode/agents/ui-designer.md +9 -0
- package/templates/AGENTS.md +1 -0
- package/templates/scripts/qa/check-dispatch-config.js +32 -0
- package/templates/scripts/qa/check-qa-plan.js +321 -0
- package/templates/scripts/qa/check-qa-scripts.js +1 -0
|
@@ -10,14 +10,30 @@ You decompose the task and produce the dispatch plan the orchestrator executes.
|
|
|
10
10
|
|
|
11
11
|
## Output contract (always return)
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
13
|
+
Never hand a persona a bare metric ("get under N lines", "type everything"). If a
|
|
14
|
+
task cannot be bounded by the spec below, split it further or flag it — do not
|
|
15
|
+
dispatch it broad.
|
|
16
|
+
|
|
17
|
+
1. **Dispatch plan** — ordered list of entries, each with:
|
|
18
|
+
- `persona → model`
|
|
19
|
+
- **Objective** — one sentence naming a single deliverable.
|
|
20
|
+
- **Owned files** — exact paths this persona may edit.
|
|
21
|
+
- **Read-only files** — explicit out-of-scope paths (governance, memory, skills, other screens).
|
|
22
|
+
- **Shared contracts** — for each shared module/type: `owns` or `consumes` (exactly one owner per contract).
|
|
23
|
+
- **Done =** — the exact gate command(s) + observable metric (e.g. `tsc --noEmit` clean, `\bany\b` count 0), so success is verifiable, not self-reported.
|
|
24
|
+
- **Dependencies** — which other dispatch, if any, must complete first.
|
|
25
|
+
2. **Parallel batches** — group only DISJOINT entries (disjoint owned files AND disjoint shared contracts). Serialize personas that would touch the same files or emit the same shared contract; put the contract owner first.
|
|
26
|
+
3. **Orchestrator-only work** — glue/mechanical steps (reads, git, commits, final integration, acceptance re-verification) kept on the main model.
|
|
16
27
|
4. **Risk flags** — any ambiguity that needs the user before proceeding.
|
|
17
28
|
|
|
18
29
|
## Rules
|
|
19
30
|
|
|
20
31
|
- Read `.agents/memory/model-routing.md` for the persona → model tiers.
|
|
32
|
+
- Read `.agents/memory/flow-map.md`. If the task's behavior maps to a flow whose `surfaces:` cross
|
|
33
|
+
codebases/layers (e.g. web + `mobile/` + an API route), the dispatch plan MUST include the sibling
|
|
34
|
+
surface's persona (e.g. `mobile-engineer`) — or explicitly mark the sibling out-of-scope so
|
|
35
|
+
`qa-architect` records it as `audited: deferred` in the QA plan's flow-closure audit. A change to
|
|
36
|
+
one surface that never mentions the twin is a plan defect, not scope.
|
|
21
37
|
- Never dispatch a persona for work the main model should just do (reads, commits, integration).
|
|
22
38
|
- Prefer the most specific persona; if none clearly fits, ask the user rather than guess.
|
|
23
39
|
- Dispatch independent subagents in parallel; serialize only when one depends on another's output.
|
|
@@ -11,9 +11,28 @@ You own QA **strategy** — risk assessment, tier selection, and progression tes
|
|
|
11
11
|
## Output contract (always return)
|
|
12
12
|
|
|
13
13
|
1. **Risk assessment** — blast radius (files/subsystems touched) + risk level `LOW | MEDIUM | HIGH`.
|
|
14
|
-
2. **Selected tier + command(s)** — from the matrix below.
|
|
15
|
-
3. **
|
|
16
|
-
|
|
14
|
+
2. **Selected tier + command(s)** — from the matrix below, chosen by **coverage, not path-label** (see Diff coverage map).
|
|
15
|
+
3. **Diff coverage map (MANDATORY)** — enumerate every changed path, one entry per line, as a
|
|
16
|
+
`Changed path -> executing test/assertion` table. A tier command matches a change only when its
|
|
17
|
+
backing suite ACTUALLY exercises that path/state. Any change with no runtime assertion is a **GAP**
|
|
18
|
+
— you must author a progression test for it (Progression vs regression below). Persist the same map
|
|
19
|
+
into `.agents/memory/qa-plan.md` under `## Coverage map` (this is what `npm run test:qa-plan` gates on).
|
|
20
|
+
3b. **Flow closure audit (MANDATORY — "fix one, fix all")** — read `.agents/memory/flow-map.md`. For
|
|
21
|
+
every flow whose `surfaces:` intersect this diff, address in the plan's
|
|
22
|
+
`## Parallel-surface & variant audit` EVERY sibling surface (the same behavior living in another
|
|
23
|
+
codebase/layer — web vs `mobile/`, API route, another page) and EVERY declared `variant`. Each
|
|
24
|
+
entry is `- <surface> -> <assertion>` or `- <surface> audited: <reason>`; each variant is
|
|
25
|
+
`- flow: <name> variant: <label> -> <assertion>` or `... audited: <reason>`. A sibling that is
|
|
26
|
+
legitimately out of scope is NOT left unnamed — it is `audited: deferred — logged in handoff.md`.
|
|
27
|
+
`test:qa-plan` fails on any sibling surface or declared variant the plan does not name.
|
|
28
|
+
If `flow-map.md` has no flows, note it and proceed (the gate warns, does not block; register the
|
|
29
|
+
flow in flow-map.md when you touch its behavior).
|
|
30
|
+
4. **Progression plan** — any NEW tests to author (and where) before the run, or a statement that no new tests are needed. For every **GAP** you MUST author a progression test that renders/executes the real path (real route/API — never a shell mock) and drives it click-through from the user's perspective.
|
|
31
|
+
5. **Handoff to qa-runner** — exact commands + pass criteria.
|
|
32
|
+
|
|
33
|
+
> The rule the whole plan rests on: a tier label never proves coverage. `test:quick` at label "Tier 1
|
|
34
|
+
> components" does NOT cover a component unless its script actually exercises it. If the project's
|
|
35
|
+
> backing script for a tier doesn't reach a changed path, that's a **coverage gap**, not a passing run.
|
|
17
36
|
|
|
18
37
|
## Dynamic Test Selection Matrix
|
|
19
38
|
|
|
@@ -29,7 +48,7 @@ You own QA **strategy** — risk assessment, tier selection, and progression tes
|
|
|
29
48
|
Diff-aware planning helper: `npm run qa:plan [-- --json] [-- --base main]`.
|
|
30
49
|
|
|
31
50
|
> The tier commands above are **project-provided** — the harness ships only
|
|
32
|
-
> `test:dispatch` and `test:
|
|
51
|
+
> `test:dispatch`, `test:governance`, and `test:qa-plan`. If a referenced script is missing,
|
|
33
52
|
> route that finding to the orchestrator (`scripts/qa/check-qa-scripts.js`
|
|
34
53
|
> verifies wiring mechanically).
|
|
35
54
|
|
|
@@ -40,6 +59,19 @@ Diff-aware planning helper: `npm run qa:plan [-- --json] [-- --base main]`.
|
|
|
40
59
|
|
|
41
60
|
Author new assertion pathways in `scripts/verify-all.js` (API), `scripts/qa/routes.js` (routes), or `scripts/qa/tests/*.test.ts` (mobile/smoke), then hand the run to `qa-runner`. Once passing, the test graduates permanently into the regression baseline.
|
|
42
61
|
|
|
62
|
+
## Coverage gate (`npm run test:qa-plan`)
|
|
63
|
+
|
|
64
|
+
Every change that touches code paths MUST have a coverage map persisted in
|
|
65
|
+
`.agents/memory/qa-plan.md` (or a scoped `waived:` entry per path) before the run can be declared
|
|
66
|
+
verified — AND a flow-closure audit of every touched flow's sibling surfaces and declared variants
|
|
67
|
+
(`.agents/memory/flow-map.md`). The mechanical gate checks the live `git diff` against both and
|
|
68
|
+
fails on any uncovered code path or unaddressed sibling/variant. If you are not dispatched for a
|
|
69
|
+
change, the main orchestrator performs your role and writes the same plan itself — a change with
|
|
70
|
+
code-path edits and no QA plan is a rules violation.
|
|
71
|
+
|
|
43
72
|
## Iron law
|
|
44
73
|
|
|
45
|
-
Never declare a change verified from static typing alone — every tier must execute a real runtime path
|
|
74
|
+
Never declare a change verified from static typing alone — every tier must execute a real runtime path
|
|
75
|
+
that exercises the MODIFIED code. A suite that touches unrelated code proves nothing about this change.
|
|
76
|
+
If the suite does not exercise the modified path, author a progression test first and cover the GAP in
|
|
77
|
+
the coverage map.
|
|
@@ -17,6 +17,7 @@ You **execute**. A `qa-architect` (or the orchestrator) tells you which tier to
|
|
|
17
17
|
## Responsibilities
|
|
18
18
|
|
|
19
19
|
- Run the tier you are given: `npm run test:quick` / `test:routes` / `test:api` / `test:verify` / `test:security`.
|
|
20
|
+
- Run the diff-coverage gate: `npm run test:qa-plan` and report uncovered paths verbatim.
|
|
20
21
|
- Data/catalog E2E journey (100 records): `npx ts-node scripts/test-catalog.ts`.
|
|
21
22
|
- Catalog test-data cleanup when asked (via the catalog reset endpoint/scripts).
|
|
22
23
|
|
|
@@ -3,6 +3,15 @@ description: Use for catalog/data ingestion and entity resolution — external-s
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/gpt-5.6-luna
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} Data Engineer. Read and follow the complete persona instructions in `.agents/skills/data-engineer/SKILL.md`, then carry out the task.
|
|
@@ -3,6 +3,15 @@ description: Use for deployment, CI/CD, cron/scheduling, env & secrets managemen
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/gpt-5.6-luna
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} DevOps Engineer. Read and follow the complete persona instructions in `.agents/skills/devops-engineer/SKILL.md`, then carry out the task.
|
|
@@ -4,6 +4,15 @@ mode: subagent
|
|
|
4
4
|
model: opencode/deepseek-v4-pro
|
|
5
5
|
temperature: 0.1
|
|
6
6
|
steps: 30
|
|
7
|
+
permission:
|
|
8
|
+
edit:
|
|
9
|
+
"*": allow
|
|
10
|
+
"opencode.json": deny
|
|
11
|
+
".harness.json": deny
|
|
12
|
+
".opencode/**": deny
|
|
13
|
+
".agents/**": deny
|
|
14
|
+
bash:
|
|
15
|
+
"*": allow
|
|
7
16
|
---
|
|
8
17
|
|
|
9
18
|
You are the {{PROJECT_NAME}} Diagnostics Expert. Read and follow the complete persona instructions in `.agents/skills/diagnostics-expert/SKILL.md`, then carry out the task.
|
|
@@ -3,6 +3,15 @@ description: Use when writing or fixing web React / Next.js / Tailwind / shadcn
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/gpt-5.6-luna
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} Frontend Engineer. Read and follow the complete persona instructions in `.agents/skills/frontend-engineer/SKILL.md`, then carry out the task.
|
|
@@ -3,6 +3,15 @@ description: Use when writing or fixing Expo / React Native native code — nati
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/gpt-5.6-luna
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} Mobile Engineer. Read and follow the complete persona instructions in `.agents/skills/mobile-engineer/SKILL.md`, then carry out the task.
|
|
@@ -12,7 +12,8 @@ You are the {{PROJECT_NAME}} QA Architect. Read and follow the complete persona
|
|
|
12
12
|
|
|
13
13
|
## Scope & integrity (non-negotiable)
|
|
14
14
|
|
|
15
|
-
- Edit ONLY files your role owns (progression-test authoring). `opencode.json`, `.agents/memory/*`, `.agents/rules/*`, other personas' files, and unrelated code are READ-ONLY absent an explicit orchestrator grant.
|
|
15
|
+
- Edit ONLY files your role owns (progression-test authoring + `.agents/memory/qa-plan.md`). `opencode.json`, other `.agents/memory/*`, `.agents/rules/*`, other personas' files, and unrelated code are READ-ONLY absent an explicit orchestrator grant.
|
|
16
16
|
- You design and author tests only — route execution to `qa-runner`. Never fabricate dispatch-log or handoff entries that did not actually occur.
|
|
17
|
+
- Persist your plan's `## Coverage map` into `.agents/memory/qa-plan.md` — `npm run test:qa-plan` gates every changed code path against it. A change with code-path edits and no coverage map is not verifiable.
|
|
17
18
|
|
|
18
19
|
Return your final message in this exact order: **Result** (what shipped / decided) -> **Evidence** (files changed, commands run, observed output) -> **Deferred & risks** (follow-ups the orchestrator must handle). Keep it under ~15 lines.
|
|
@@ -3,6 +3,15 @@ description: Use when a change touches auth, secrets, input validation, external
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/deepseek-v4-pro
|
|
5
5
|
steps: 30
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} Security Engineer. Read and follow the complete persona instructions in `.agents/skills/security-engineer/SKILL.md`, then carry out the task.
|
|
@@ -3,6 +3,15 @@ description: Use when changing the Prisma schema, data model, multi-tenant bound
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/deepseek-v4-pro
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} System Architect. Read and follow the complete persona instructions in `.agents/skills/system-architect/SKILL.md`, then carry out the task.
|
|
@@ -3,6 +3,15 @@ description: Use when designing the visual system — design tokens, theming, an
|
|
|
3
3
|
mode: subagent
|
|
4
4
|
model: opencode/gpt-5.6-luna
|
|
5
5
|
steps: 40
|
|
6
|
+
permission:
|
|
7
|
+
edit:
|
|
8
|
+
"*": allow
|
|
9
|
+
"opencode.json": deny
|
|
10
|
+
".harness.json": deny
|
|
11
|
+
".opencode/**": deny
|
|
12
|
+
".agents/**": deny
|
|
13
|
+
bash:
|
|
14
|
+
"*": allow
|
|
6
15
|
---
|
|
7
16
|
|
|
8
17
|
You are the {{PROJECT_NAME}} UI Designer. Read and follow the complete persona instructions in `.agents/skills/ui-designer/SKILL.md`, then carry out the task.
|
package/templates/AGENTS.md
CHANGED
|
@@ -5,6 +5,7 @@ The always-loaded rules summary lives in [`.agents/rules/00-operating.md`](.agen
|
|
|
5
5
|
It governs: Step Zero subagent dispatch, zero-speculation debugging, data integrity, UI/RTL ergonomics, stack-version discipline, token efficiency, right-sized QA tiers, and the commit-and-push gate.
|
|
6
6
|
|
|
7
7
|
- **Dispatch personas** via the `task` tool (see `.agents/rules/00-operating.md` §Step Zero); consult the relevant `.agents/skills/<persona>/SKILL.md`.
|
|
8
|
+
- **Every change touching code paths gets a QA plan** (`.agents/memory/qa-plan.md`) from `qa-architect` — if it isn't dispatched, the main orchestrator plans in its place. `npm run test:qa-plan` fails on any changed code path with no covering assertion or waiver, and on any touched flow (`.agents/memory/flow-map.md`) whose sibling surfaces / declared variants the plan doesn't address.
|
|
8
9
|
- **At session start**, read `.agents/memory/locations.md` and `.agents/memory/model-routing.md`.
|
|
9
10
|
|
|
10
11
|
> Deployed and maintained by `@dombaras/agent-harness` (`npx @dombaras/agent-harness init` / `update`). Do not hand-edit harness-owned files — regenerate them and commit the deltas.
|
|
@@ -12,6 +12,8 @@
|
|
|
12
12
|
* 3. Every agent body carries the output-contract marker (Result / Evidence /
|
|
13
13
|
* Deferred & risks) so subagents return a consistent report shape.
|
|
14
14
|
* 4. `opencode.json` main `model` + `small_model` are also real provider IDs.
|
|
15
|
+
* 5. `.agents/memory/model-routing.md` does not name a concrete model that is
|
|
16
|
+
* no longer pinned by any agent or opencode.json (doc↔config drift).
|
|
15
17
|
*
|
|
16
18
|
* Allowlist refresh (only when the provider adds/removes models):
|
|
17
19
|
* npx opencode models > scripts/qa/models.allowlist.txt
|
|
@@ -62,7 +64,12 @@ function frontmatterModel(file) {
|
|
|
62
64
|
return modelLine.replace(/^model:\s*/i, "").trim();
|
|
63
65
|
}
|
|
64
66
|
|
|
67
|
+
function escapeRe(s) {
|
|
68
|
+
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
69
|
+
}
|
|
70
|
+
|
|
65
71
|
// 1. every agent file exists, pins a real model, references a real skill
|
|
72
|
+
const agentModels = new Set();
|
|
66
73
|
const agentNames = fs
|
|
67
74
|
.readdirSync(agentsDir)
|
|
68
75
|
.filter((f) => f.endsWith(".md"))
|
|
@@ -72,6 +79,7 @@ for (const f of agentNames) {
|
|
|
72
79
|
const name = f.replace(/\.md$/, "");
|
|
73
80
|
const file = path.join(agentsDir, f);
|
|
74
81
|
const model = frontmatterModel(file);
|
|
82
|
+
if (model) agentModels.add(model);
|
|
75
83
|
const body = fs.readFileSync(file, "utf8");
|
|
76
84
|
const skillMd = path.join(skillsDir, name, "SKILL.md");
|
|
77
85
|
if (!model) fail(`agent ${f}: missing frontmatter model: (all agents must pin a model)`);
|
|
@@ -95,16 +103,40 @@ if (skillNames.join("|") === [...agentNames].map((f) => f.replace(/\.md$/, "")).
|
|
|
95
103
|
pass(`persona parity: ${agentNames.length} agents == ${skillNames.length} skills`);
|
|
96
104
|
|
|
97
105
|
// 3. opencode.json main + small model are real provider IDs
|
|
106
|
+
const configModels = new Set();
|
|
98
107
|
if (fs.existsSync(opencodeJson)) {
|
|
99
108
|
const cfg = JSON.parse(fs.readFileSync(opencodeJson, "utf8"));
|
|
100
109
|
for (const key of ["model", "small_model"]) {
|
|
101
110
|
const v = cfg[key];
|
|
111
|
+
if (v) configModels.add(v);
|
|
102
112
|
if (!v) warn(`opencode.json: no \`${key}\` set`);
|
|
103
113
|
else if (allowlist.has(v)) pass(`opencode.json ${key} \`${v}\` is a real provider model`);
|
|
104
114
|
else fail(`opencode.json ${key} \`${v}\` NOT in provider allowlist`);
|
|
105
115
|
}
|
|
106
116
|
} else warn("opencode.json not found (skip)");
|
|
107
117
|
|
|
118
|
+
// 4. model-routing.md ↔ model: pin consistency (doc↔config drift)
|
|
119
|
+
const usedModels = new Set([...agentModels, ...configModels]);
|
|
120
|
+
const modelRoutingFile = path.join(root, ".agents", "memory", "model-routing.md");
|
|
121
|
+
if (fs.existsSync(modelRoutingFile)) {
|
|
122
|
+
const doc = fs.readFileSync(modelRoutingFile, "utf8");
|
|
123
|
+
const modelPattern = [...allowlist].map(escapeRe).join("|");
|
|
124
|
+
const mentioned = new Set();
|
|
125
|
+
const re = new RegExp(modelPattern, "g");
|
|
126
|
+
let m;
|
|
127
|
+
while ((m = re.exec(doc)) !== null) mentioned.add(m[0]);
|
|
128
|
+
if (mentioned.size === 0) {
|
|
129
|
+
pass("model-routing.md names no concrete model IDs (tier policy only) — nothing to cross-check");
|
|
130
|
+
} else {
|
|
131
|
+
for (const mid of mentioned) {
|
|
132
|
+
if (usedModels.has(mid)) pass(`model-routing.md \`${mid}\` matches an actual pin`);
|
|
133
|
+
else fail(`model-routing.md references \`${mid}\` but no agent/config pins it (doc↔config drift)`);
|
|
134
|
+
}
|
|
135
|
+
}
|
|
136
|
+
} else {
|
|
137
|
+
warn("model-routing.md not found (skip doc↔config drift check)");
|
|
138
|
+
}
|
|
139
|
+
|
|
108
140
|
console.log(
|
|
109
141
|
`\nRESULT: ${agentNames.length} agents checked \u2014 ` +
|
|
110
142
|
(failures.length ? `${failures.length} FAILURE(S)` : "0 failures") +
|
|
@@ -0,0 +1,321 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
"use strict";
|
|
3
|
+
/*
|
|
4
|
+
* test:qa-plan — diff-coverage gate (harness-shipped).
|
|
5
|
+
*
|
|
6
|
+
* The anti-verification-from-unrelated-suites gate. A tier label never proves
|
|
7
|
+
* coverage; this gate forces a change's QA plan to actually cover the code it
|
|
8
|
+
* modified — otherwise "verified" is meaningless.
|
|
9
|
+
*
|
|
10
|
+
* Two stages:
|
|
11
|
+
*
|
|
12
|
+
* 1. Coverage map — inspects the live git diff and requires every changed CODE
|
|
13
|
+
* path to appear in the QA plan's `## Coverage map` (persisted by
|
|
14
|
+
* qa-architect, or by the main orchestrator as the qa-architect fallback)
|
|
15
|
+
* BEFORE a change may be declared verified. Trivial diffs (copy/docs/not
|
|
16
|
+
* code) and waivered paths are exempt.
|
|
17
|
+
*
|
|
18
|
+
* 2. Flow closure — reads `.agents/memory/flow-map.md` (project-owned): a
|
|
19
|
+
* registry of user-facing flows, each listing the SURFACES that implement the
|
|
20
|
+
* same behavior (web / mobile / API — any codebase/layer) and its optional
|
|
21
|
+
* VARIANTS. Any flow whose surfaces intersect this diff must have every
|
|
22
|
+
* sibling surface and every declared variant addressed in the plan's
|
|
23
|
+
* `## Parallel-surface & variant audit` (covered `->` or `audited:` with a
|
|
24
|
+
* reason). This is the gate that makes "Fix One, Fix All" mechanical: a
|
|
25
|
+
* web-only diff can no longer silently leave the mobile twin on old behavior.
|
|
26
|
+
* If flow-map.md is absent or lists no flows, the stage warns but never fails
|
|
27
|
+
* (backward compatible — projects opt into flow closure by filling it in).
|
|
28
|
+
*
|
|
29
|
+
* node scripts/qa/check-qa-plan.js # diff vs HEAD
|
|
30
|
+
* node scripts/qa/check-qa-plan.js --base main # diff vs a ref
|
|
31
|
+
* node scripts/qa/check-qa-plan.js --strict # non-code changes too
|
|
32
|
+
*
|
|
33
|
+
* Exit 0 on: no code-path changes, full coverage, valid waivers, full flow closure.
|
|
34
|
+
* Exit 1 on: a changed code path with no covering assertion/waiver, or a flow
|
|
35
|
+
* sibling surface / declared variant the QA plan does not address.
|
|
36
|
+
*/
|
|
37
|
+
const fs = require("fs");
|
|
38
|
+
const path = require("path");
|
|
39
|
+
const { execFileSync } = require("child_process");
|
|
40
|
+
|
|
41
|
+
const root = path.resolve(__dirname, "..", "..");
|
|
42
|
+
|
|
43
|
+
const argv = process.argv.slice(2);
|
|
44
|
+
const base = (argv.find((a) => a.startsWith("--base=")) || "").split("=")[1] || "HEAD";
|
|
45
|
+
const strict = argv.includes("--strict");
|
|
46
|
+
|
|
47
|
+
// Paths that are tooling/harness, not application code under test. A change
|
|
48
|
+
// that only touches these needs no runtime QA plan (they are guarded by
|
|
49
|
+
// test:dispatch / test:governance themselves).
|
|
50
|
+
const TOOLING = [
|
|
51
|
+
"opencode.json",
|
|
52
|
+
".harness.json",
|
|
53
|
+
".opencode/",
|
|
54
|
+
".agents/",
|
|
55
|
+
"scripts/qa/",
|
|
56
|
+
".github/",
|
|
57
|
+
"package.json",
|
|
58
|
+
];
|
|
59
|
+
// Non-code files never need a runtime path (copy/docs/styling/data-flat).
|
|
60
|
+
const NON_CODE = /\.(md|mdx|png|jpg|jpeg|gif|svg|webp|ico|json|lock|txt|env|yml|yaml|toml|csv)$/i;
|
|
61
|
+
|
|
62
|
+
const CODE = /\.(ts|tsx|js|jsx|css|scss|sass|prisma|sql)$/i;
|
|
63
|
+
|
|
64
|
+
function norm(p) {
|
|
65
|
+
return p.replace(/\\/g, "/");
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
function gitChangedFiles() {
|
|
69
|
+
let files = [];
|
|
70
|
+
try {
|
|
71
|
+
const out = execFileSync("git", ["diff", "--name-only", base, "--"], {
|
|
72
|
+
cwd: root,
|
|
73
|
+
encoding: "utf8",
|
|
74
|
+
});
|
|
75
|
+
files = out.split(/\r?\n/).filter(Boolean);
|
|
76
|
+
} catch (e) {
|
|
77
|
+
// 128 = not a git repo / bad ref / no base history (e.g. empty repo).
|
|
78
|
+
if (e.status === 128) return null;
|
|
79
|
+
throw e;
|
|
80
|
+
}
|
|
81
|
+
// include untracked files so brand-new sources can't silently skip the gate
|
|
82
|
+
try {
|
|
83
|
+
const untracked = execFileSync(
|
|
84
|
+
"git",
|
|
85
|
+
["ls-files", "--others", "--exclude-standard", "--", root],
|
|
86
|
+
{ cwd: root, encoding: "utf8" }
|
|
87
|
+
)
|
|
88
|
+
.split(/\r?\n/)
|
|
89
|
+
.filter(Boolean);
|
|
90
|
+
files = [...new Set([...files, ...untracked])];
|
|
91
|
+
} catch (_) {
|
|
92
|
+
/* ls-files is best-effort */
|
|
93
|
+
}
|
|
94
|
+
return files.map(norm);
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
function isTooling(p) {
|
|
98
|
+
const n = norm(p);
|
|
99
|
+
return TOOLING.some((t) => n === t || n.startsWith(t));
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
/* Parse the QA plan file into { covered:Set, waivers:Set } of normalized paths.
|
|
103
|
+
* Coverage map format (one entry per changed path, under `## Coverage map`):
|
|
104
|
+
* - components/return/ReturnConfirmDrawer.tsx -> test:quick::progression (click-through)
|
|
105
|
+
* Waivers live anywhere and carry a reason on the same line:
|
|
106
|
+
* waived: components/nav.tsx reason: pure-copy change, no runtime path
|
|
107
|
+
*/
|
|
108
|
+
function parsePlan(file) {
|
|
109
|
+
const out = { covered: new Set(), waivers: new Set() };
|
|
110
|
+
if (!fs.existsSync(file)) return out;
|
|
111
|
+
const src = fs.readFileSync(file, "utf8");
|
|
112
|
+
const lines = src.split(/\r?\n/);
|
|
113
|
+
|
|
114
|
+
// Coverage map section only
|
|
115
|
+
let inMap = false;
|
|
116
|
+
for (const raw of lines) {
|
|
117
|
+
const line = raw.trim();
|
|
118
|
+
if (!line) continue;
|
|
119
|
+
if (/^#{1,6}\s*Coverage map\s*$/i.test(line)) {
|
|
120
|
+
inMap = true;
|
|
121
|
+
continue;
|
|
122
|
+
}
|
|
123
|
+
if (/^[#;/]/.test(line)) continue;
|
|
124
|
+
if (inMap && /^#{1,6}\s/.test(line)) inMap = false;
|
|
125
|
+
if (!inMap) continue;
|
|
126
|
+
|
|
127
|
+
const m = line.match(/^[-*]\s*["']?([^\s"'>]+)/);
|
|
128
|
+
if (m) out.covered.add(norm(m[1]));
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
// Waivers (anywhere) — path then reason
|
|
132
|
+
for (const raw of lines) {
|
|
133
|
+
const line = raw.trim();
|
|
134
|
+
if (!/\bwaived:\s*\S/i.test(line)) continue;
|
|
135
|
+
const p = (line.match(/\bwaived:\s*["']?([^\s"',]+)/i) || [])[1];
|
|
136
|
+
if (p && /\breason:\s*\S/i.test(line)) out.waivers.add(norm(p));
|
|
137
|
+
}
|
|
138
|
+
return out;
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
function isCovered(pathi, covered, waivers) {
|
|
142
|
+
if (covered.has(pathi) || waivers.has(pathi)) return true;
|
|
143
|
+
// directory-prefix coverage (a plan may cover a whole folder)
|
|
144
|
+
for (const c of covered) if (pathi.startsWith(c.replace(/\/?$/, "/"))) return true;
|
|
145
|
+
for (const w of waivers) if (pathi.startsWith(w.replace(/\/?$/, "/"))) return true;
|
|
146
|
+
return false;
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/* Parse `.agents/memory/flow-map.md` into [{ name, surfaces:[], variants:[] }].
|
|
150
|
+
* Format (one flow per `## flow:` header):
|
|
151
|
+
* ## flow: return
|
|
152
|
+
* behavior: "Mark as Returned" — optional condition payload (condition selector)
|
|
153
|
+
* surfaces: components/LendingActionDrawers.tsx, mobile/features/return/ReturnStatusSheet.tsx, app/api/transactions/[id]/return/route.ts
|
|
154
|
+
* variants: condition-selector-present, condition-selector-absent
|
|
155
|
+
*/
|
|
156
|
+
function parseFlowMap(file) {
|
|
157
|
+
const flows = [];
|
|
158
|
+
if (!fs.existsSync(file)) return flows;
|
|
159
|
+
let cur = null;
|
|
160
|
+
for (const raw of fs.readFileSync(file, "utf8").split(/\r?\n/)) {
|
|
161
|
+
const line = raw.trim();
|
|
162
|
+
if (!line) continue;
|
|
163
|
+
const h = line.match(/^#{1,6}\s*flow:\s*(.+)$/i);
|
|
164
|
+
if (h) {
|
|
165
|
+
cur = { name: h[1].trim(), surfaces: [], variants: [] };
|
|
166
|
+
flows.push(cur);
|
|
167
|
+
continue;
|
|
168
|
+
}
|
|
169
|
+
if (/^[#;]/.test(line)) continue;
|
|
170
|
+
if (!cur) continue;
|
|
171
|
+
const s = line.match(/^surfaces?:\s*(.+)$/i);
|
|
172
|
+
if (s) cur.surfaces = s[1].split(",").map((x) => norm(x.trim())).filter(Boolean);
|
|
173
|
+
const v = line.match(/^variants?:\s*(.+)$/i);
|
|
174
|
+
if (v) cur.variants = v[1].split(",").map((x) => x.trim()).filter(Boolean);
|
|
175
|
+
}
|
|
176
|
+
return flows.filter((f) => f.name && f.surfaces.length > 0);
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
/* Parse the plan's `## Parallel-surface & variant audit` entries.
|
|
180
|
+
* Entry forms:
|
|
181
|
+
* - <surface> -> <assertion>
|
|
182
|
+
* - <surface> audited: <reason>
|
|
183
|
+
* - flow: <name> variant: <label> -> <assertion>
|
|
184
|
+
* - flow: <name> variant: <label> audited: <reason>
|
|
185
|
+
*/
|
|
186
|
+
function parseAudit(file) {
|
|
187
|
+
const entries = [];
|
|
188
|
+
if (!fs.existsSync(file)) return entries;
|
|
189
|
+
let inAudit = false;
|
|
190
|
+
for (const raw of fs.readFileSync(file, "utf8").split(/\r?\n/)) {
|
|
191
|
+
const line = raw.trim();
|
|
192
|
+
if (!line) continue;
|
|
193
|
+
if (/^#{1,6}\s*Parallel[ -]surface.{0,40}(?:&\s*|and\s+).{0,40}audit\s*$/i.test(line)) {
|
|
194
|
+
inAudit = true;
|
|
195
|
+
continue;
|
|
196
|
+
}
|
|
197
|
+
if (/^[#;]/.test(line)) continue;
|
|
198
|
+
if (inAudit && /^#{1,6}\s/.test(line)) inAudit = false;
|
|
199
|
+
if (!inAudit) continue;
|
|
200
|
+
|
|
201
|
+
const vm = line.match(/^[-*]\s*flow:\s*["']?([^\s"']+)\s+variant:\s*["']?([^\s"']+)\s+(audited:.+|->\s*.+)$/i);
|
|
202
|
+
if (vm) {
|
|
203
|
+
entries.push({ flow: vm[1], variant: vm[2], surface: null, disposition: vm[3].trim() });
|
|
204
|
+
continue;
|
|
205
|
+
}
|
|
206
|
+
const pm = line.match(/^[-*]\s*["']?([^\s"'>]+)\s+(audited:.+|->\s*.+)$/);
|
|
207
|
+
if (pm) entries.push({ surface: norm(pm[1]), disposition: pm[2].trim() });
|
|
208
|
+
}
|
|
209
|
+
return entries;
|
|
210
|
+
}
|
|
211
|
+
|
|
212
|
+
/* True when path a and path b are the same file/dir or one nests under the other
|
|
213
|
+
* as a directory prefix — lenient matching for surfaces vs coverage/audit paths. */
|
|
214
|
+
function matches(a, b) {
|
|
215
|
+
const A = norm(a).replace(/\/$/, "");
|
|
216
|
+
const B = norm(b).replace(/\/$/, "");
|
|
217
|
+
return A === B || A.startsWith(B + "/") || B.startsWith(A + "/");
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
const failures = [];
|
|
221
|
+
const pass = (m) => console.log(" \u2713 " + m);
|
|
222
|
+
const fail = (m) => {
|
|
223
|
+
failures.push(m);
|
|
224
|
+
console.log(" \u2717 " + m);
|
|
225
|
+
};
|
|
226
|
+
|
|
227
|
+
console.log("diff-coverage gate \u2014 QA plan covers every changed code path");
|
|
228
|
+
|
|
229
|
+
const changed = gitChangedFiles();
|
|
230
|
+
if (changed == null) {
|
|
231
|
+
console.log(" \u26a0 not a git repo or no base history \u2014 coverage gate skipped");
|
|
232
|
+
console.log("\nRESULT: skipped (0 failures)");
|
|
233
|
+
process.exit(0);
|
|
234
|
+
}
|
|
235
|
+
if (changed.length === 0) {
|
|
236
|
+
console.log(" \u2713 no diff vs " + base + " \u2014 nothing to cover");
|
|
237
|
+
console.log("\nRESULT: green");
|
|
238
|
+
process.exit(0);
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
const plan = path.join(root, ".agents", "memory", "qa-plan.md");
|
|
242
|
+
const audit = parseAudit(plan);
|
|
243
|
+
const { covered, waivers } = parsePlan(plan);
|
|
244
|
+
|
|
245
|
+
console.log(` checked against: ${path.relative(root, plan)}`);
|
|
246
|
+
|
|
247
|
+
const changedCode = [];
|
|
248
|
+
for (const f of changed) {
|
|
249
|
+
if (isTooling(f)) {
|
|
250
|
+
pass(`${f} (harness/tooling \u2014 self-guarded)`);
|
|
251
|
+
continue;
|
|
252
|
+
}
|
|
253
|
+
if (NON_CODE.test(f) && !strict) {
|
|
254
|
+
pass(`${f} (non-code \u2014 no runtime path required)`);
|
|
255
|
+
continue;
|
|
256
|
+
}
|
|
257
|
+
if (!CODE.test(f)) {
|
|
258
|
+
pass(`${f} (not a code path)`);
|
|
259
|
+
continue;
|
|
260
|
+
}
|
|
261
|
+
changedCode.push(f);
|
|
262
|
+
if (isCovered(f, covered, waivers)) pass(`${f} covered by QA plan`);
|
|
263
|
+
else
|
|
264
|
+
fail(
|
|
265
|
+
`${f} NOT covered \u2014 add it to the \`## Coverage map\` in qa-plan.md (or a waivered entry with a reason)`
|
|
266
|
+
);
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
// ------------------------------------------------------------ flow closure
|
|
270
|
+
const flowMap = parseFlowMap(path.join(root, ".agents", "memory", "flow-map.md"));
|
|
271
|
+
const hitFlows = new Set();
|
|
272
|
+
for (const f of flowMap) {
|
|
273
|
+
if (changedCode.some((c) => f.surfaces.some((s) => matches(c, s)))) hitFlows.add(f);
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
if (!flowMap.length) {
|
|
277
|
+
console.log(
|
|
278
|
+
" \u26a0 flow-map.md has no flows \u2014 sibling/variant closure NOT enforced (add flows to secure \"Fix One, Fix All\")"
|
|
279
|
+
);
|
|
280
|
+
} else if (hitFlows.size) {
|
|
281
|
+
console.log("flow-closure gate \u2014 every sibling surface & variant of a touched flow is addressed");
|
|
282
|
+
for (const f of flowMap) {
|
|
283
|
+
if (!hitFlows.has(f)) {
|
|
284
|
+
pass(`flow "${f.name}" not touched \u2014 no sibling requirement`);
|
|
285
|
+
continue;
|
|
286
|
+
}
|
|
287
|
+
// Sibling surfaces: every declared surface must be addressed by the diff
|
|
288
|
+
// (as a changed path) or by a named audit entry, or audited with a reason.
|
|
289
|
+
for (const s of f.surfaces) {
|
|
290
|
+
if (changedCode.some((c) => matches(c, s))) {
|
|
291
|
+
continue; // the changed path itself is required in the Coverage map above
|
|
292
|
+
}
|
|
293
|
+
const addressed = audit.some((e) => e.surface && matches(e.surface, s));
|
|
294
|
+
if (addressed) pass(`flow "${f.name}": sibling \`${s}\` addressed in audit`);
|
|
295
|
+
else
|
|
296
|
+
fail(
|
|
297
|
+
`flow "${f.name}": sibling surface \`${s}\` not addressed \u2014 add it to \`## Parallel-surface & variant audit\` as \`-> <assertion>\` or \`audited: <reason>\``
|
|
298
|
+
);
|
|
299
|
+
}
|
|
300
|
+
// Declared variants: each must be named with a disposition for this flow.
|
|
301
|
+
for (const v of f.variants) {
|
|
302
|
+
const addressed = audit.some((e) => e.flow && e.variant && e.flow === f.name && e.variant === v);
|
|
303
|
+
if (addressed) pass(`flow "${f.name}": variant \`${v}\` addressed`);
|
|
304
|
+
else
|
|
305
|
+
fail(
|
|
306
|
+
`flow "${f.name}": declared variant \`${v}\` not addressed \u2014 add \`flow: ${f.name} variant: ${v} -> <assertion>\` or \`... audited: <reason>\` to \`## Parallel-surface & variant audit\``
|
|
307
|
+
);
|
|
308
|
+
}
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
|
|
312
|
+
if (!changedCode.length) {
|
|
313
|
+
console.log(" \u2713 no application code paths changed");
|
|
314
|
+
}
|
|
315
|
+
console.log(
|
|
316
|
+
"\nRESULT: " +
|
|
317
|
+
(failures.length
|
|
318
|
+
? `${failures.length} FAILURE(S) \u2014 every changed code path needs a covering assertion or a waivered reason; every touched flow's sibling surfaces & variants must be addressed in the audit`
|
|
319
|
+
: "green \u2014 all changed code paths have a covering QA plan and all touched flows are closed")
|
|
320
|
+
);
|
|
321
|
+
process.exit(failures.length ? 1 : 0);
|
|
@@ -22,6 +22,7 @@ const strict = process.argv.includes("--strict");
|
|
|
22
22
|
const EXPECTED = {
|
|
23
23
|
"test:dispatch": "harness — persona model-pin preflight (shipped)",
|
|
24
24
|
"test:governance": "harness — session-governance gate (shipped)",
|
|
25
|
+
"test:qa-plan": "harness — diff-coverage + flow-closure gate (changed code paths covered; touched flows' sibling surfaces/variants audited)",
|
|
25
26
|
"test:quick": "Tier 1 — types + translations + mobile smoke",
|
|
26
27
|
"test:routes": "Tier 2 — route/page render",
|
|
27
28
|
"test:api": "Tier 3 — API/ORM edge cases",
|