@activate-core/test-forge 1.2.2 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +67 -28
- package/dist/cli/index.js +98 -34
- package/dist/core/docs/cache.js +2 -1
- package/dist/core/docs/discover.js +51 -4
- package/dist/core/docs/doc-guidance.js +61 -23
- package/dist/core/docs/excerpts.js +245 -0
- package/dist/core/docs/grounding.js +51 -24
- package/dist/core/docs/reference-text.js +78 -0
- package/dist/core/docs/reference.js +68 -5
- package/dist/core/docs/review.js +81 -0
- package/dist/core/docs/source-of-truth.js +1 -1
- package/dist/core/docs/tickets.js +56 -0
- package/dist/core/gates/coverage.js +12 -19
- package/dist/core/gates/validate.js +277 -4
- package/dist/core/generate/ai-generator.js +243 -96
- package/dist/core/generate/ai-session.js +8 -0
- package/dist/core/generate/ai-usage.js +150 -0
- package/dist/core/generate/bind.js +91 -0
- package/dist/core/generate/defect-review.js +213 -20
- package/dist/core/generate/doc-diff.js +139 -0
- package/dist/core/generate/harness.js +52 -32
- package/dist/core/generate/journey-memory.js +325 -0
- package/dist/core/generate/plan-cache.js +114 -7
- package/dist/core/generate/required-data.js +75 -0
- package/dist/core/generate/review-journey.js +104 -0
- package/dist/core/generate/run-generate.js +193 -80
- package/dist/core/generate/seed-generator.js +2 -2
- package/dist/core/generate/suite-planner.js +138 -25
- package/dist/core/generate/test-blocks.js +84 -0
- package/dist/core/generate/update-journey.js +115 -0
- package/dist/core/generate/verify.js +9 -4
- package/dist/core/paths.js +90 -13
- package/dist/core/project/config.js +25 -2
- package/dist/core/project/env-diff.js +122 -0
- package/dist/core/project/env-template.js +3 -2
- package/dist/core/project/init.js +2 -2
- package/dist/core/project/map.js +2 -2
- package/dist/core/run/ci.js +94 -0
- package/dist/core/run/clean.js +111 -0
- package/dist/core/run/readiness.js +119 -0
- package/dist/core/run/report.js +2 -0
- package/dist/core/run/run-tests.js +23 -1
- package/package.json +1 -1
- package/reporters/_shared.cjs +24 -3
- package/reporters/jest-reporter.cjs +3 -1
- package/reporters/playwright-reporter.cjs +3 -1
- package/reporters/run-data.cjs +66 -0
- package/reporters/ui-helpers.cjs +133 -0
- package/dist/core/generate/verify-journey.js +0 -174
package/README.md
CHANGED
|
@@ -94,7 +94,7 @@ never touched.
|
|
|
94
94
|
| Command | What it does |
|
|
95
95
|
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
96
96
|
| `testforge setup` | One-time setup: runs init, map, and discover in order, with progress. |
|
|
97
|
-
| `testforge generate` |
|
|
97
|
+
| `testforge generate` | Writes the tests for each journey doc. A changed doc updates only the tests of the rules that changed. Writes only: it runs nothing. |
|
|
98
98
|
| `testforge update` | Refreshes the map, docs and tests when the **source code** or the **reference** has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |
|
|
99
99
|
|
|
100
100
|
### Advanced (run a single step)
|
|
@@ -105,7 +105,9 @@ never touched.
|
|
|
105
105
|
| `testforge map` | Rebuild the project map (`testforge.md`) only. |
|
|
106
106
|
| `testforge discover` | Redraft the per-journey docs in `docs/` only. |
|
|
107
107
|
| `testforge env` | Rebuild `.env.testing.example` from the docs only. |
|
|
108
|
-
| `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). |
|
|
108
|
+
| `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). Files whose feature is not in the code yet are held back as pending (`--all` runs them). |
|
|
109
|
+
| `testforge bind` | After a feature ships: which test files the code is ready for, which wait for the feature, which use a name the code spells differently. `--accept "doc=code"` takes the code's wording. No AI. |
|
|
110
|
+
| `testforge review` | Ask an AI reviewer whether the last failures are defects, test mistakes or the environment. On demand only. |
|
|
109
111
|
| `testforge report` | Publish results to the dashboard — `--results` after a run, `--inventory` to register every generated check. |
|
|
110
112
|
|
|
111
113
|
### Flags worth knowing
|
|
@@ -115,6 +117,9 @@ never touched.
|
|
|
115
117
|
| `--journey <name>` | discover, generate, run | Work on one journey instead of all of them. |
|
|
116
118
|
| `--area <name>` | discover, generate, run | Work on one area (api, web, ops…). |
|
|
117
119
|
| `--force` | generate | Regenerate even when the doc has not changed. |
|
|
120
|
+
| `--plan-only` | generate | Plan and check each journey (`testforge/memory/<area>/<id>/plan-check.md`), write no tests. |
|
|
121
|
+
| `--refresh` | discover | Fetch the reference again instead of the copy kept from the last run. |
|
|
122
|
+
| `--all` | run | Also run files whose feature is not in the code yet. |
|
|
118
123
|
| `--refresh-docs` | update | Re-draft the docs even when nothing detectably changed — see below. |
|
|
119
124
|
| `--reference <file\|url>` | setup | Base the docs on a spec instead of the code map. |
|
|
120
125
|
| `--source map` | setup | Base the docs on `testforge.md`, no prompt. |
|
|
@@ -198,7 +203,7 @@ _expected_ behavior — a spec or requirements doc in the repo, a PDF, a
|
|
|
198
203
|
website, or a Notion page. testforge then:
|
|
199
204
|
|
|
200
205
|
1. reads the reference and compares it against `testforge.md`,
|
|
201
|
-
2. writes a full comparison to **`testforge
|
|
206
|
+
2. writes a full comparison to **`testforge/reference/diff.md`** and prints a short
|
|
202
207
|
summary of the differences,
|
|
203
208
|
3. asks you to confirm, and on confirmation **drafts the docs from the
|
|
204
209
|
reference** — expected behavior comes from the reference, while routes,
|
|
@@ -207,9 +212,9 @@ website, or a Notion page. testforge then:
|
|
|
207
212
|
|
|
208
213
|
The reference becomes the source of truth — expected behavior from a human
|
|
209
214
|
or client, not just what the code happens to do. The choice is remembered
|
|
210
|
-
in `
|
|
215
|
+
in `testforge/source.json`, so a later `testforge update` refreshes the
|
|
211
216
|
docs from the same source without asking again. Your review of
|
|
212
|
-
`testforge
|
|
217
|
+
`testforge/reference/diff.md` stays as an audit of what differed and why.
|
|
213
218
|
|
|
214
219
|
**Non-interactive use (CI, scripts):**
|
|
215
220
|
|
|
@@ -285,6 +290,22 @@ the project map list the env var names each journey needs.
|
|
|
285
290
|
|
|
286
291
|
The test runner produces its own pass/fail report.
|
|
287
292
|
|
|
293
|
+
### In CI (GitHub Actions)
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
npx testforge ci --github # run only when started by hand
|
|
297
|
+
npx testforge ci --github --schedule weekdays # also Mon–Fri 06:00 UTC (daily, weekly or a cron line too)
|
|
298
|
+
npx testforge ci --github --env-name staging --force
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The workflow runs `testforge run` when started by hand (and on the schedule, if given), against a
|
|
302
|
+
deployed environment, and publishes each run to the dashboard. It reads its
|
|
303
|
+
values from repository secrets with the same names as your env file
|
|
304
|
+
(TestForge reads only the names, never the values), plus `TESTFORGE_KEY`,
|
|
305
|
+
and the dashboard address from the repository variable
|
|
306
|
+
`TESTFORGE_DASHBOARD_URL`. Point the base URL secrets at the deployed app.
|
|
307
|
+
Test results are kept as a workflow artifact for 14 days.
|
|
308
|
+
|
|
288
309
|
## Configuration
|
|
289
310
|
|
|
290
311
|
`testforge setup` writes `testforge.config.json` for you. You can edit it:
|
|
@@ -341,29 +362,47 @@ The test runner produces its own pass/fail report.
|
|
|
341
362
|
reported as a pending decision, never as a defect. Keeps the reference
|
|
342
363
|
you share with a client clean while the tests still know what is undecided.
|
|
343
364
|
|
|
344
|
-
## How `generate`
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
2. **
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
365
|
+
## How `generate` keeps the AI cost down
|
|
366
|
+
|
|
367
|
+
The journey doc is read once and kept as the journey's **memory**
|
|
368
|
+
(`testforge/memory/<area>/<id>/`): its rules with their source, level, tests and
|
|
369
|
+
history, the plan, and the code facts. No later step sends the whole doc again.
|
|
370
|
+
|
|
371
|
+
1. **Rules.** The doc is split into rules (R1, R2 …) without AI. Rule ids stay
|
|
372
|
+
stable when the doc changes. Every rule records its source, the files that
|
|
373
|
+
test it, why it was skipped, and its earlier wordings.
|
|
374
|
+
2. **Plan, without AI.** Each numbered section of the doc becomes one test file
|
|
375
|
+
(`"planner": "ai"` asks the AI instead). Rules another journey already tests
|
|
376
|
+
are not planned again. The plan check (`plan-check.md`) lists every rule that
|
|
377
|
+
gets tests, every skip with its reason, and any **must** rule left out.
|
|
378
|
+
3. **Code facts, once.** The source the doc points at (the files it cites, the
|
|
379
|
+
components its controls are named after, the pages and hooks that reach
|
|
380
|
+
them) is collected without AI and summarised in one call.
|
|
381
|
+
4. **Writing.** Each call writes a few files (`filesPerCall`, default 3) and
|
|
382
|
+
gets only their rules, their controls' facts and the testkit's exports; the
|
|
383
|
+
shared rules for writing a test are a cached system prompt. Tests find every
|
|
384
|
+
element with `kit.find(page, kind, name)`, check outcomes rather than markup,
|
|
385
|
+
and stub a state they cannot create.
|
|
386
|
+
5. **Checks before anything runs.** A file is refused, and written once more
|
|
387
|
+
with the reason, if it sleeps for a fixed time, skips because an element is
|
|
388
|
+
missing, has no test or no assertion, spends more than `maxTestsPerRule`
|
|
389
|
+
tests on one rule, or names a label found neither in the code nor the doc.
|
|
390
|
+
6. **A changed doc.** Only what changed is touched: a renamed label is replaced
|
|
391
|
+
in its tests, a removed rule's tests are deleted (both without AI); a
|
|
392
|
+
reworded or new rule gets its tests written by one small call.
|
|
393
|
+
|
|
394
|
+
`generate` runs nothing. `testforge run` runs the suite; `testforge review`
|
|
395
|
+
judges failures when you ask for it.
|
|
396
|
+
|
|
397
|
+
Every Claude call reports its real token use and cost; `TESTFORGE_TRACE=1`
|
|
398
|
+
also records each call's prompt and events under `testforge/runs/trace/`.
|
|
399
|
+
|
|
400
|
+
## Project state
|
|
401
|
+
|
|
402
|
+
Everything TestForge keeps lives in `testforge/`. Commit it, except `runs/`
|
|
403
|
+
(run output and AI logs): a teammate who pulls it starts from your memory, plans
|
|
404
|
+
and code facts instead of paying for them again. A project set up by an older
|
|
405
|
+
version is moved from `test-artifacts/` on the first command.
|
|
367
406
|
|
|
368
407
|
## Notes
|
|
369
408
|
|
package/dist/cli/index.js
CHANGED
|
@@ -20,6 +20,14 @@
|
|
|
20
20
|
*/
|
|
21
21
|
import { VERSION } from '../core/index.js';
|
|
22
22
|
import { runGenerate } from '../core/generate/run-generate.js';
|
|
23
|
+
import { runReview } from '../core/generate/review-journey.js';
|
|
24
|
+
import { migrateState } from '../core/paths.js';
|
|
25
|
+
import { acceptNames, bindJourneys } from '../core/generate/bind.js';
|
|
26
|
+
import { runClean } from '../core/run/clean.js';
|
|
27
|
+
import { aiUsage, describeUsage } from '../core/generate/ai-usage.js';
|
|
28
|
+
import { envDiff, describeEnvDiff } from '../core/project/env-diff.js';
|
|
29
|
+
import { bundleTickets } from '../core/docs/tickets.js';
|
|
30
|
+
import { writeGithubWorkflow } from '../core/run/ci.js';
|
|
23
31
|
import { runTests } from '../core/run/run-tests.js';
|
|
24
32
|
import { runDiscover } from '../core/docs/discover.js';
|
|
25
33
|
import { runMap } from '../core/project/map.js';
|
|
@@ -45,10 +53,18 @@ function flagValue(name) {
|
|
|
45
53
|
}
|
|
46
54
|
/** Source-of-truth options shared by setup and discover. */
|
|
47
55
|
function sourceOptions() {
|
|
56
|
+
const tickets = flagValue('--tickets');
|
|
57
|
+
let reference = flagValue('--reference');
|
|
58
|
+
if (!reference && tickets) {
|
|
59
|
+
const bundle = bundleTickets(tickets);
|
|
60
|
+
console.log(`Using ${bundle.ids.length} exported ticket(s) from ${tickets} as the reference: ${bundle.ids.slice(0, 8).join(', ')}${bundle.ids.length > 8 ? ', …' : ''}`);
|
|
61
|
+
reference = bundle.path;
|
|
62
|
+
}
|
|
48
63
|
return {
|
|
49
|
-
reference
|
|
64
|
+
reference,
|
|
50
65
|
source: flagValue('--source'),
|
|
51
66
|
replace: process.argv.includes('--replace'),
|
|
67
|
+
refresh: process.argv.includes('--refresh'),
|
|
52
68
|
};
|
|
53
69
|
}
|
|
54
70
|
/**
|
|
@@ -78,12 +94,13 @@ function printUsage() {
|
|
|
78
94
|
console.log(' testforge generate --basic Use the basic generator only');
|
|
79
95
|
console.log(' testforge generate --force Regenerate everything');
|
|
80
96
|
console.log(' testforge generate --no-wait Stop when the AI hits its usage limit (default: wait for the reset and resume)');
|
|
81
|
-
console.log(' testforge generate --
|
|
97
|
+
console.log(' testforge generate --plan-only Plan and check each journey (testforge/memory/<area>/<id>/plan-check.md), write no tests');
|
|
82
98
|
console.log(' testforge generate --journey <name> Generate just ONE journey');
|
|
83
99
|
console.log(' testforge generate --area <area> Generate a whole area (e.g. api)');
|
|
84
100
|
console.log(' testforge discover Draft journey docs (asks: map or reference)');
|
|
85
|
-
console.log(' testforge discover --journey <name> Draft just ONE journey doc');
|
|
101
|
+
console.log(' testforge discover --journey <name> Draft just ONE journey doc', ' testforge discover --journey <name> --reference <x> [--refresh] One journey from a reference: the reference is read once and kept (--refresh reads it again)');
|
|
86
102
|
console.log(' testforge discover --reference <file|url> Base docs on a reference');
|
|
103
|
+
console.log(' testforge discover --tickets <folder> Base docs on exported tickets (one file per ticket); each source names its ticket');
|
|
87
104
|
console.log(' (a reference only writes the journeys it');
|
|
88
105
|
console.log(' describes — add --replace to let it');
|
|
89
106
|
console.log(' redraft the whole project)');
|
|
@@ -91,12 +108,21 @@ function printUsage() {
|
|
|
91
108
|
console.log(' testforge init Inspect project and write config');
|
|
92
109
|
console.log(' testforge update Re-check and update only what changed');
|
|
93
110
|
console.log(' testforge env Write .env.testing.example from docs');
|
|
111
|
+
console.log(' testforge env --diff Only the variables the suite reads that the env file does not set');
|
|
112
|
+
console.log(' testforge ci --github Write a GitHub Actions workflow that runs the suite against a deployed environment');
|
|
113
|
+
console.log(' testforge bind [--journey j] [--accept "doc=code"] Which tests the code is ready for, which wait for the feature, which use another name; --accept takes the code\'s wording (no AI)');
|
|
114
|
+
console.log(' testforge review [--journey j] [--rerun] Judge the last failures again (defect / test fault / environment …), no repair');
|
|
115
|
+
console.log(' testforge clean [--run <id> | --older-than <days>] [--yes] Remove what earlier runs created (dry run without --yes)');
|
|
94
116
|
console.log(' testforge run [--area a] [--journey j] Run the generated tests (Jest/Playwright)');
|
|
95
117
|
console.log(' testforge run --env-file <file> Run against another server (e.g. .env.dev / .env.staging)');
|
|
96
118
|
console.log(' testforge run --headed | --ui | --debug Watch the browser / open the Playwright UI');
|
|
97
119
|
console.log(' testforge run -- <flags> Pass anything else straight to the runner');
|
|
98
120
|
}
|
|
99
121
|
async function main() {
|
|
122
|
+
// A project set up by an older version keeps its state under test-artifacts/: move it once.
|
|
123
|
+
const moved = command && command !== 'help' && command !== '--help' ? migrateState() : [];
|
|
124
|
+
if (moved.length)
|
|
125
|
+
console.log(`Moved TestForge's state into testforge/ (${moved.length} item(s)); nothing was rebuilt.\n`);
|
|
100
126
|
if (command === 'update') {
|
|
101
127
|
await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
|
|
102
128
|
}
|
|
@@ -116,6 +142,59 @@ async function main() {
|
|
|
116
142
|
console.log(output);
|
|
117
143
|
console.log('\nProject map written to testforge.md');
|
|
118
144
|
}
|
|
145
|
+
else if (command === 'ci') {
|
|
146
|
+
if (!process.argv.includes('--github')) {
|
|
147
|
+
console.log('Usage: testforge ci --github [--schedule none|daily|weekdays|weekly|"<cron>"] [--env-name <label>] [--force]');
|
|
148
|
+
}
|
|
149
|
+
else {
|
|
150
|
+
console.log(writeGithubWorkflow({ envName: flagValue('--env-name'), force: process.argv.includes('--force'), schedule: flagValue('--schedule') }));
|
|
151
|
+
}
|
|
152
|
+
}
|
|
153
|
+
else if (command === 'clean') {
|
|
154
|
+
const days = flagValue('--older-than');
|
|
155
|
+
console.log(await runClean({ runId: flagValue('--run'), olderThanDays: days != null ? Number(days) : undefined, yes: process.argv.includes('--yes') }));
|
|
156
|
+
}
|
|
157
|
+
else if (command === 'bind') {
|
|
158
|
+
// After a feature ships: which tests can run, which wait for the code, which use a different name.
|
|
159
|
+
const accept = process.argv.flatMap((a, k) => (process.argv[k - 1] === '--accept' ? [a] : []));
|
|
160
|
+
if (accept.length) {
|
|
161
|
+
const pairs = accept.map((a) => a.split('=')).filter((p) => p.length === 2 && p[0] && p[1]);
|
|
162
|
+
const notes = acceptNames(pairs, { journey: flagValue('--journey') });
|
|
163
|
+
for (const n of notes)
|
|
164
|
+
console.log(` ${n}`);
|
|
165
|
+
console.log(notes.length ? '\nRun `testforge generate` to record the rename in the journey\'s memory (no AI).' : 'Nothing used those names.');
|
|
166
|
+
}
|
|
167
|
+
else {
|
|
168
|
+
for (const r of bindJourneys({ journey: flagValue('--journey') })) {
|
|
169
|
+
const by = (s) => r.files.filter((f) => f.state === s);
|
|
170
|
+
console.log(` ${r.journey}: ${by('built').length} ready, ${by('partial').length} partly built, ${by('not-built').length} not built yet, ${by('renamed').length} with a different name`);
|
|
171
|
+
for (const f of r.files) {
|
|
172
|
+
if (f.state === 'built')
|
|
173
|
+
continue;
|
|
174
|
+
const what = f.state === 'renamed'
|
|
175
|
+
? f.renamed.map((x) => `"${x.name}" → code has "${x.nearest}"`).join('; ')
|
|
176
|
+
: `not in the code: ${f.missing.slice(0, 4).map((m) => `"${m}"`).join(', ')}${f.missing.length > 4 ? ' …' : ''}`;
|
|
177
|
+
console.log(` ${f.state.padEnd(9)} ${f.file}: ${what}`);
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
console.log('\nA different name is not a defect: change the code to the doc, or accept the code\'s wording with --accept "doc wording=code wording".');
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
else if (command === 'review') {
|
|
184
|
+
console.log('Reviewing the last failures of each journey (no repair)...\n');
|
|
185
|
+
const summaries = await runReview({ journey: flagValue('--journey'), rerun: process.argv.includes('--rerun') });
|
|
186
|
+
if (!summaries.length)
|
|
187
|
+
console.log('No generated journey matched.');
|
|
188
|
+
for (const r of summaries) {
|
|
189
|
+
if (r.skipped)
|
|
190
|
+
console.log(` ${r.journey}: not reviewed — ${r.skipped}`);
|
|
191
|
+
else if (!r.failures)
|
|
192
|
+
console.log(` ${r.journey}: no failures to review`);
|
|
193
|
+
else
|
|
194
|
+
console.log(` ${r.journey}: ${r.failures} failure(s) → ` + Object.entries(r.verdicts).map(([k, n]) => `${n} ${k}`).join(', '));
|
|
195
|
+
}
|
|
196
|
+
console.log('\nVerdicts written to testforge/runs/defect-review/. The next run publishes them with each failed test.');
|
|
197
|
+
}
|
|
119
198
|
else if (command === 'discover') {
|
|
120
199
|
console.log('Drafting journey files with AI...');
|
|
121
200
|
console.log('This writes drafts into docs/ from your chosen source.\n');
|
|
@@ -125,7 +204,10 @@ async function main() {
|
|
|
125
204
|
console.log('then run `testforge generate`.');
|
|
126
205
|
}
|
|
127
206
|
else if (command === 'env') {
|
|
128
|
-
|
|
207
|
+
if (process.argv.includes('--diff'))
|
|
208
|
+
console.log(describeEnvDiff(envDiff('.')));
|
|
209
|
+
else
|
|
210
|
+
runEnvTemplate('docs', '.env.testing.example', '.', generatedScanDirs());
|
|
129
211
|
}
|
|
130
212
|
else if (command === 'run') {
|
|
131
213
|
// Anything after `--`, plus the runner flags people reach for most, goes
|
|
@@ -141,6 +223,7 @@ async function main() {
|
|
|
141
223
|
journey: flagValue('--journey'),
|
|
142
224
|
envFile: flagValue('--env-file'),
|
|
143
225
|
passThrough,
|
|
226
|
+
all: process.argv.includes('--all'),
|
|
144
227
|
});
|
|
145
228
|
process.exitCode = code;
|
|
146
229
|
}
|
|
@@ -155,11 +238,7 @@ async function main() {
|
|
|
155
238
|
: process.argv.includes('--wait-for-limit')
|
|
156
239
|
? true
|
|
157
240
|
: undefined, // undefined -> config.waitForLimit (default on)
|
|
158
|
-
|
|
159
|
-
? false
|
|
160
|
-
: process.argv.includes('--verify')
|
|
161
|
-
? true
|
|
162
|
-
: undefined, // undefined -> config.verify (default on)
|
|
241
|
+
planOnly: process.argv.includes('--plan-only'),
|
|
163
242
|
});
|
|
164
243
|
if (results.length === 0) {
|
|
165
244
|
console.log('No journey files found in docs/. Add a .md file and try again.');
|
|
@@ -190,9 +269,6 @@ async function main() {
|
|
|
190
269
|
for (const w of r.warnings ?? []) {
|
|
191
270
|
console.log(` ! ${w}`);
|
|
192
271
|
}
|
|
193
|
-
if (r.verified === true)
|
|
194
|
-
console.log(' verified: every file ran; every failure left was reviewed and is listed below');
|
|
195
|
-
printReviewed(r.reviewed ?? []);
|
|
196
272
|
}
|
|
197
273
|
const generated = results.filter((r) => r.status !== 'skipped').length;
|
|
198
274
|
// Rows with no test dir are notices (a non-journey file, the AI going
|
|
@@ -202,6 +278,16 @@ async function main() {
|
|
|
202
278
|
const seedFailures = results.filter((r) => r.seedStatus === 'failed').length;
|
|
203
279
|
console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged)` +
|
|
204
280
|
(notices ? `, ${notices} notice(s) above.` : '.'));
|
|
281
|
+
const spent = aiUsage();
|
|
282
|
+
if (spent.calls)
|
|
283
|
+
console.log(`AI this run: ${describeUsage(spent)}.`);
|
|
284
|
+
// What the journeys just written still need from the env file.
|
|
285
|
+
const touched = results.filter((r) => r.testDir && r.status !== 'skipped').map((r) => basename(r.testDir));
|
|
286
|
+
if (touched.length) {
|
|
287
|
+
const d = envDiff('.', touched);
|
|
288
|
+
if (d.journeys.some((j) => j.missing.some((v) => !v.hasDefault)))
|
|
289
|
+
console.log('\n' + describeEnvDiff({ ...d, journeys: d.journeys.map((j) => ({ ...j, missing: j.missing.filter((v) => !v.hasDefault) })) }));
|
|
290
|
+
}
|
|
205
291
|
if (seedFailures > 0) {
|
|
206
292
|
console.log(`Warning: ${seedFailures} seed file(s) could not be generated — see notes above.`);
|
|
207
293
|
}
|
|
@@ -260,28 +346,6 @@ async function main() {
|
|
|
260
346
|
printUsage();
|
|
261
347
|
}
|
|
262
348
|
}
|
|
263
|
-
/** Reviewed failures, one line per root cause, strongest first. */
|
|
264
|
-
function printReviewed(reviewed) {
|
|
265
|
-
const sections = [
|
|
266
|
-
['defect', 'DEFECTS — a [spec] requirement the app breaks'],
|
|
267
|
-
['finding', 'FINDINGS — real app behaviour against a [baseline]/[observed] criterion (lower priority)'],
|
|
268
|
-
['pending-decision', 'PENDING DECISIONS — not requirements yet; take them to product, do not report as defects'],
|
|
269
|
-
];
|
|
270
|
-
for (const [verdict, heading] of sections) {
|
|
271
|
-
const rows = reviewed.filter((r) => r.verdict === verdict);
|
|
272
|
-
if (!rows.length)
|
|
273
|
-
continue;
|
|
274
|
-
console.log(` ${heading}:`);
|
|
275
|
-
for (const r of rows) {
|
|
276
|
-
const where = r.tests.slice(0, 3).map((t) => `${basename(t.file)}:${t.line}`).join(', ');
|
|
277
|
-
const more = r.tests.length > 3 ? ` +${r.tests.length - 3} more` : '';
|
|
278
|
-
console.log(` - ${r.claim}`);
|
|
279
|
-
console.log(` ${r.tests.length} test(s): ${where}${more}${r.evidence ? ` · app: ${r.evidence}` : ''}`);
|
|
280
|
-
if (r.reason)
|
|
281
|
-
console.log(` ${r.reason}`);
|
|
282
|
-
}
|
|
283
|
-
}
|
|
284
|
-
}
|
|
285
349
|
main().catch((err) => {
|
|
286
350
|
console.error(err instanceof Error ? err.message : String(err));
|
|
287
351
|
process.exit(1);
|
package/dist/core/docs/cache.js
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
import { createHash } from 'node:crypto';
|
|
13
13
|
import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, statSync, } from 'node:fs';
|
|
14
14
|
import { dirname, join, extname } from 'node:path';
|
|
15
|
-
import { STATE_DIR, LEGACY_STATE_DIR } from '../paths.js';
|
|
15
|
+
import { STATE_DIR, LEGACY_STATE_DIR, OLD_STATE_DIR } from '../paths.js';
|
|
16
16
|
/** Make a short fingerprint of a file's content. */
|
|
17
17
|
export function hashContent(content) {
|
|
18
18
|
return createHash('sha256').update(content).digest('hex');
|
|
@@ -66,6 +66,7 @@ const SKIP_DIRS = new Set([
|
|
|
66
66
|
'docs',
|
|
67
67
|
STATE_DIR,
|
|
68
68
|
LEGACY_STATE_DIR,
|
|
69
|
+
OLD_STATE_DIR,
|
|
69
70
|
'generated-tests',
|
|
70
71
|
]);
|
|
71
72
|
export function hashProjectSource(rootDir) {
|
|
@@ -14,7 +14,6 @@
|
|
|
14
14
|
* tools) rather than stuffing all source code into the prompt, which
|
|
15
15
|
* keeps this simple and works for projects of any size.
|
|
16
16
|
*/
|
|
17
|
-
import { execFileSync } from 'node:child_process';
|
|
18
17
|
import { existsSync, readdirSync, readFileSync, writeFileSync, mkdirSync, renameSync, } from 'node:fs';
|
|
19
18
|
import { join, relative, sep, basename } from 'node:path';
|
|
20
19
|
import { notAJourneyDoc } from './parser.js';
|
|
@@ -22,8 +21,11 @@ import { loadConfig, splitCommand, withEditTools } from '../project/config.js';
|
|
|
22
21
|
import { DEEP_DOC_GUIDANCE } from './doc-guidance.js';
|
|
23
22
|
import { promptConfirm } from '../prompt.js';
|
|
24
23
|
import { loadSourceChoice, resolveDocsSource, saveJourneyReferences, saveSourceChoice, } from './source-of-truth.js';
|
|
25
|
-
import { analyzeReference, draftDocsFromReference, DIFF_PATH, } from './reference.js';
|
|
26
|
-
import { reviewDocs } from './review.js';
|
|
24
|
+
import { analyzeReference, buildJourneyDraftPrompt, draftDocsFromReference, draftJourneyDoc, DIFF_PATH, } from './reference.js';
|
|
25
|
+
import { buildLeanReviewPrompt, enforceSpecSources, reviewDocs } from './review.js';
|
|
26
|
+
import { readReference, sectionFor } from './reference-text.js';
|
|
27
|
+
import { sourceExcerpts } from './excerpts.js';
|
|
28
|
+
import { runAiSync } from '../generate/ai-usage.js';
|
|
27
29
|
/** Scope line: one journey, or all of them. */
|
|
28
30
|
function scopeLines(journey) {
|
|
29
31
|
return journey
|
|
@@ -125,7 +127,7 @@ function runCodeDiscover(srcDir, docsDir, journey) {
|
|
|
125
127
|
// are tool-specific; for Claude we add acceptEdits. Other tools may
|
|
126
128
|
// need different setup (see README).
|
|
127
129
|
const { program, args } = withEditTools(splitCommand(loadConfig().aiCommand));
|
|
128
|
-
const output =
|
|
130
|
+
const output = runAiSync(program, args, {
|
|
129
131
|
input: prompt,
|
|
130
132
|
encoding: 'utf-8',
|
|
131
133
|
maxBuffer: 10 * 1024 * 1024,
|
|
@@ -168,6 +170,12 @@ function snapshotDocs(docsDir) {
|
|
|
168
170
|
*/
|
|
169
171
|
export async function runDiscover(srcDir, docsDir, opts = {}, journey) {
|
|
170
172
|
const source = await resolveDocsSource(opts);
|
|
173
|
+
// One journey from a reference: no comparison of the whole product, the
|
|
174
|
+
// reference read once and cut to this journey, drafted and reviewed in one
|
|
175
|
+
// call each with no tools.
|
|
176
|
+
if (source.mode === 'reference' && source.reference && journey) {
|
|
177
|
+
return draftOneJourney(srcDir, docsDir, source.reference, journey, opts.refresh);
|
|
178
|
+
}
|
|
171
179
|
if (source.mode === 'reference' && source.reference) {
|
|
172
180
|
console.log(`\nComparing the reference against testforge.md: ${source.reference}`);
|
|
173
181
|
console.log('(reading the reference and the map — this may take a moment)\n');
|
|
@@ -287,3 +295,42 @@ async function discoverFromCode(srcDir, docsDir, journey) {
|
|
|
287
295
|
await reviewDocs(docsDir, changed, srcDir);
|
|
288
296
|
return out;
|
|
289
297
|
}
|
|
298
|
+
const docTitle = (text) => /^# Journey:\s*(.+)$/m.exec(text)?.[1]?.trim();
|
|
299
|
+
const kebab = (t) => t.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
|
|
300
|
+
async function draftOneJourney(srcDir, docsDir, reference, journey, refresh = false) {
|
|
301
|
+
const config = loadConfig();
|
|
302
|
+
const text = readReference(reference, refresh);
|
|
303
|
+
const section = sectionFor(text, journey);
|
|
304
|
+
const excerpts = sourceExcerpts(section, [srcDir, ...(config.relatedDirs ?? [])]);
|
|
305
|
+
const docs = [...snapshotDocs(docsDir)].filter(([p, t]) => !notAJourneyDoc(p, t));
|
|
306
|
+
const existing = docs.find(([, t]) => kebab(docTitle(t) ?? '') === kebab(journey))?.[0];
|
|
307
|
+
const others = docs.filter(([p]) => p !== existing).map(([p, t]) => `${docTitle(t) ?? p} (${p})`);
|
|
308
|
+
const areas = Object.keys(config.testOutput ?? {}).length ? Object.keys(config.testOutput ?? {}) : ['web', 'api'];
|
|
309
|
+
const owner = loadSourceChoice()?.journeys?.[existing ?? ''];
|
|
310
|
+
if (existing && owner && owner !== reference) {
|
|
311
|
+
throw new Error(`${existing} comes from another reference (${owner}); not redrafting it from ${reference}.`);
|
|
312
|
+
}
|
|
313
|
+
console.log(`\nDrafting "${journey}" from the reference (${section.length.toLocaleString('en-GB')} characters of it, one call, no tools)...`);
|
|
314
|
+
const draft = draftJourneyDoc(buildJourneyDraftPrompt(journey, section, excerpts, areas, others));
|
|
315
|
+
if (!draft.markdown)
|
|
316
|
+
throw new Error('The AI answered without a journey doc.');
|
|
317
|
+
console.log('Reviewing it for missing scenarios (one call, no tools)...');
|
|
318
|
+
let doc = draft.markdown;
|
|
319
|
+
try {
|
|
320
|
+
const reviewed = draftJourneyDoc(buildLeanReviewPrompt(doc, section, excerpts, others));
|
|
321
|
+
if (reviewed.markdown)
|
|
322
|
+
doc = reviewed.markdown;
|
|
323
|
+
}
|
|
324
|
+
catch (e) {
|
|
325
|
+
console.log(` review failed, kept as drafted (${e.message.split('\n')[0]})`);
|
|
326
|
+
}
|
|
327
|
+
const { markdown, retagged } = enforceSpecSources(doc);
|
|
328
|
+
if (retagged)
|
|
329
|
+
console.log(` ${retagged} [spec] line(s) without a quoted source re-tagged [observed]`);
|
|
330
|
+
const rel = existing ?? join(draft.area && areas.includes(draft.area) ? draft.area : areas[0], `${kebab(docTitle(markdown) ?? journey)}.md`);
|
|
331
|
+
mkdirSync(join(docsDir, rel, '..'), { recursive: true });
|
|
332
|
+
writeFileSync(join(docsDir, rel), markdown, 'utf-8');
|
|
333
|
+
if (!existing)
|
|
334
|
+
saveJourneyReferences([rel], reference);
|
|
335
|
+
return `${existing ? 'Rewrote' : 'Wrote'} ${join(docsDir, rel)} from the reference. Please review it, then run \`testforge generate --journey ${kebab(journey)}\`.`;
|
|
336
|
+
}
|
|
@@ -45,30 +45,68 @@ export const DEEP_DOC_GUIDANCE = [
|
|
|
45
45
|
`- "[spec]" — the source states this behaviour. A failure is a defect`,
|
|
46
46
|
` against the spec.`,
|
|
47
47
|
`- "[baseline]" — universal robustness that holds for ANY well-built app.`,
|
|
48
|
-
`
|
|
49
|
-
`
|
|
48
|
+
` The client describes what to build; it rarely says how it must NOT`,
|
|
49
|
+
` break. That is your job: think like a tester who wants to break it and`,
|
|
50
|
+
` like a user who is technical enough to find its loopholes. Add these`,
|
|
51
|
+
` YOURSELF for every screen, form, list, action and endpoint the journey`,
|
|
52
|
+
` has, even when the source says nothing about them. Write each one's`,
|
|
53
|
+
` LEVEL and CATEGORY right after the tag, e.g. "[baseline] (must,`,
|
|
54
|
+
` security)". Every "must" that fits the journey has to be there.`,
|
|
55
|
+
` MUST — security:`,
|
|
56
|
+
` * One user never reaches another user's data or actions: changing an id`,
|
|
57
|
+
` in a URL, a request or a hidden field is refused (not "not found by`,
|
|
58
|
+
` accident" — refused). A page or endpoint that needs a login is refused`,
|
|
59
|
+
` without one; a role cannot use another role's actions.`,
|
|
60
|
+
` * Whatever the client sends is not trusted: a price, an amount, a`,
|
|
61
|
+
` discount, a role, an owner id or a status changed in the request is`,
|
|
62
|
+
` refused or recomputed by the server.`,
|
|
63
|
+
` * An HTML, script or injection payload in any input is stored and shown`,
|
|
64
|
+
` as text, never executed, never a server error (5xx).`,
|
|
65
|
+
` * Passwords are masked; passwords, tokens and codes never appear in a`,
|
|
66
|
+
` URL, a query string or a log the user can see.`,
|
|
67
|
+
` * Repeated wrong attempts (password, code, coupon, PIN) are limited and`,
|
|
68
|
+
` the limit is told to the user.`,
|
|
69
|
+
` MUST — money and data:`,
|
|
70
|
+
` * Double submission: an action with a side effect (pay, book, order,`,
|
|
71
|
+
` send, create) done`,
|
|
72
|
+
` twice — a double click, a repeated Enter, two tabs, a retry after a`,
|
|
73
|
+
` timeout — leaves exactly one record, one charge, one message.`,
|
|
74
|
+
` * A flow that fails half-way leaves no half-made record and no money`,
|
|
75
|
+
` taken without the thing bought; the user sees what happened.`,
|
|
76
|
+
` * Amounts, totals, discounts and limits are the same on every screen and`,
|
|
77
|
+
` in the server's answer.`,
|
|
78
|
+
` MUST — concurrency:`,
|
|
79
|
+
` * Two users or two tabs acting on the same thing at once (the same slot,`,
|
|
80
|
+
` the last item, a code with limited uses, the same record edited) —`,
|
|
81
|
+
` exactly one wins and the other gets a clear reason.`,
|
|
82
|
+
` * Nothing hangs: every action ends within its timeout with a result or a`,
|
|
83
|
+
` visible error; a request that never answers never leaves a spinner`,
|
|
84
|
+
` forever or a control disabled for good.`,
|
|
85
|
+
` MUST — error boundaries:`,
|
|
86
|
+
` * Every load and every action, when the server answers 500, the network`,
|
|
87
|
+
` drops, it times out, or the answer is empty or malformed: a visible`,
|
|
88
|
+
` message, the page stays usable, no blank page, no crash, the user's`,
|
|
89
|
+
` input is kept.`,
|
|
90
|
+
` * One part of a page failing never takes the whole page down.`,
|
|
91
|
+
` SHOULD — validation:`,
|
|
50
92
|
` * Inputs: empty and whitespace-only values; leading/trailing spaces;`,
|
|
51
|
-
` a very long value (5,000 characters); unicode and emoji
|
|
52
|
-
`
|
|
53
|
-
`
|
|
54
|
-
`
|
|
55
|
-
` *
|
|
56
|
-
`
|
|
57
|
-
`
|
|
58
|
-
`
|
|
59
|
-
`
|
|
60
|
-
`
|
|
61
|
-
`
|
|
62
|
-
`
|
|
63
|
-
`
|
|
64
|
-
`
|
|
65
|
-
`
|
|
66
|
-
` *
|
|
67
|
-
`
|
|
68
|
-
` * Access: a page or action that needs a login is not usable logged`,
|
|
69
|
-
` out; one user never sees another user's data.`,
|
|
70
|
-
` * Choice controls (dropdown, radio, checkbox, list): every option`,
|
|
71
|
-
` shown can be selected and submitted.`,
|
|
93
|
+
` a very long value (5,000 characters); unicode and emoji. Each is`,
|
|
94
|
+
` accepted or rejected with a visible reason, never a crash.`,
|
|
95
|
+
` * No silent failure: every rejected submit says why, next to the field`,
|
|
96
|
+
` or the form (a wrong password with no message is a defect).`,
|
|
97
|
+
` * The screen and the server accept exactly the same values.`,
|
|
98
|
+
` SHOULD — state and navigation:`,
|
|
99
|
+
` * Refresh, Back, Forward and opening a mid-flow URL directly never crash`,
|
|
100
|
+
` or leave a blank page; an expired session sends the user to log in and`,
|
|
101
|
+
` back; after logging out nothing of the account is shown.`,
|
|
102
|
+
` * Choice controls (dropdown, radio, checkbox, list): every option shown`,
|
|
103
|
+
` can be selected and submitted; long lists page or scroll.`,
|
|
104
|
+
` SHOULD — regression:`,
|
|
105
|
+
` * Every defect the source or the doc already names keeps a test, so it`,
|
|
106
|
+
` cannot quietly come back.`,
|
|
107
|
+
` COULD — accessibility:`,
|
|
108
|
+
` * The form can be filled and submitted with the keyboard alone; Enter`,
|
|
109
|
+
` submits; pop-ups close with Escape; controls have readable names.`,
|
|
72
110
|
` [baseline] is ROBUSTNESS only. Never present product POLICY you`,
|
|
73
111
|
` invented (a maximum length, a password rule, an error wording, a`,
|
|
74
112
|
` limit) as [baseline].`,
|