@activate-core/test-forge 1.3.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +67 -28
- package/dist/cli/index.js +39 -33
- package/dist/core/docs/cache.js +2 -1
- package/dist/core/docs/discover.js +49 -2
- package/dist/core/docs/doc-guidance.js +70 -29
- package/dist/core/docs/excerpts.js +294 -0
- package/dist/core/docs/grounding.js +55 -22
- package/dist/core/docs/reference-text.js +78 -0
- package/dist/core/docs/reference.js +60 -3
- package/dist/core/docs/review.js +38 -0
- package/dist/core/docs/source-of-truth.js +1 -1
- package/dist/core/docs/tickets.js +3 -3
- package/dist/core/gates/coverage.js +1 -16
- package/dist/core/gates/test-quality.js +295 -0
- package/dist/core/gates/validate.js +171 -7
- package/dist/core/generate/ai-generator.js +267 -117
- package/dist/core/generate/ai-usage.js +116 -5
- package/dist/core/generate/bind.js +91 -0
- package/dist/core/generate/complete-file.js +77 -0
- package/dist/core/generate/defect-review.js +59 -9
- package/dist/core/generate/doc-diff.js +140 -0
- package/dist/core/generate/harness.js +18 -34
- package/dist/core/generate/journey-memory.js +394 -0
- package/dist/core/generate/plan-cache.js +114 -7
- package/dist/core/generate/review-journey.js +3 -3
- package/dist/core/generate/run-generate.js +240 -82
- package/dist/core/generate/suite-planner.js +128 -23
- package/dist/core/generate/test-blocks.js +84 -0
- package/dist/core/generate/update-journey.js +118 -0
- package/dist/core/paths.js +90 -13
- package/dist/core/project/config.js +25 -2
- package/dist/core/project/env-template.js +2 -1
- package/dist/core/run/ci.js +1 -1
- package/dist/core/run/clean.js +1 -1
- package/dist/core/run/readiness.js +146 -0
- package/dist/core/run/run-tests.js +26 -2
- package/package.json +1 -1
- package/reporters/_shared.cjs +2 -2
- package/reporters/run-data.cjs +1 -1
- package/reporters/ui-helpers.cjs +78 -1
- package/dist/core/generate/verify-journey.js +0 -215
package/README.md
CHANGED
|
@@ -94,7 +94,7 @@ never touched.
|
|
|
94
94
|
| Command | What it does |
|
|
95
95
|
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
96
96
|
| `testforge setup` | One-time setup: runs init, map, and discover in order, with progress. |
|
|
97
|
-
| `testforge generate` |
|
|
97
|
+
| `testforge generate` | Writes the tests for each journey doc. A changed doc updates only the tests of the rules that changed. Writes only: it runs nothing. |
|
|
98
98
|
| `testforge update` | Refreshes the map, docs and tests when the **source code** or the **reference** has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |
|
|
99
99
|
|
|
100
100
|
### Advanced (run a single step)
|
|
@@ -105,7 +105,9 @@ never touched.
|
|
|
105
105
|
| `testforge map` | Rebuild the project map (`testforge.md`) only. |
|
|
106
106
|
| `testforge discover` | Redraft the per-journey docs in `docs/` only. |
|
|
107
107
|
| `testforge env` | Rebuild `.env.testing.example` from the docs only. |
|
|
108
|
-
| `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). |
|
|
108
|
+
| `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). Files whose feature is not in the code yet are held back as pending (`--all` runs them). |
|
|
109
|
+
| `testforge bind` | After a feature ships: which test files the code is ready for, which wait for the feature, which use a name the code spells differently. `--accept "doc=code"` takes the code's wording. No AI. |
|
|
110
|
+
| `testforge review` | Ask an AI reviewer whether the last failures are defects, test mistakes or the environment. On demand only. |
|
|
109
111
|
| `testforge report` | Publish results to the dashboard — `--results` after a run, `--inventory` to register every generated check. |
|
|
110
112
|
|
|
111
113
|
### Flags worth knowing
|
|
@@ -115,6 +117,9 @@ never touched.
|
|
|
115
117
|
| `--journey <name>` | discover, generate, run | Work on one journey instead of all of them. |
|
|
116
118
|
| `--area <name>` | discover, generate, run | Work on one area (api, web, ops…). |
|
|
117
119
|
| `--force` | generate | Regenerate even when the doc has not changed. |
|
|
120
|
+
| `--plan-only` | generate | Plan and check each journey (`testforge/memory/<area>/<id>/plan-check.md`), write no tests. |
|
|
121
|
+
| `--refresh` | discover | Fetch the reference again instead of the copy kept from the last run. |
|
|
122
|
+
| `--all` | run | Also run files whose feature is not in the code yet. |
|
|
118
123
|
| `--refresh-docs` | update | Re-draft the docs even when nothing detectably changed — see below. |
|
|
119
124
|
| `--reference <file\|url>` | setup | Base the docs on a spec instead of the code map. |
|
|
120
125
|
| `--source map` | setup | Base the docs on `testforge.md`, no prompt. |
|
|
@@ -198,7 +203,7 @@ _expected_ behavior — a spec or requirements doc in the repo, a PDF, a
|
|
|
198
203
|
website, or a Notion page. testforge then:
|
|
199
204
|
|
|
200
205
|
1. reads the reference and compares it against `testforge.md`,
|
|
201
|
-
2. writes a full comparison to **`testforge
|
|
206
|
+
2. writes a full comparison to **`testforge/reference/diff.md`** and prints a short
|
|
202
207
|
summary of the differences,
|
|
203
208
|
3. asks you to confirm, and on confirmation **drafts the docs from the
|
|
204
209
|
reference** — expected behavior comes from the reference, while routes,
|
|
@@ -207,9 +212,9 @@ website, or a Notion page. testforge then:
|
|
|
207
212
|
|
|
208
213
|
The reference becomes the source of truth — expected behavior from a human
|
|
209
214
|
or client, not just what the code happens to do. The choice is remembered
|
|
210
|
-
in `
|
|
215
|
+
in `testforge/source.json`, so a later `testforge update` refreshes the
|
|
211
216
|
docs from the same source without asking again. Your review of
|
|
212
|
-
`testforge
|
|
217
|
+
`testforge/reference/diff.md` stays as an audit of what differed and why.
|
|
213
218
|
|
|
214
219
|
**Non-interactive use (CI, scripts):**
|
|
215
220
|
|
|
@@ -357,29 +362,63 @@ Test results are kept as a workflow artifact for 14 days.
|
|
|
357
362
|
reported as a pending decision, never as a defect. Keeps the reference
|
|
358
363
|
you share with a client clean while the tests still know what is undecided.
|
|
359
364
|
|
|
360
|
-
## How `generate`
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
2. **
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
365
|
+
## How `generate` keeps the AI cost down
|
|
366
|
+
|
|
367
|
+
The journey doc is read once and kept as the journey's **memory**
|
|
368
|
+
(`testforge/memory/<area>/<id>/`): its rules with their source, level, tests and
|
|
369
|
+
history, the plan, and the code facts. No later step sends the whole doc again.
|
|
370
|
+
|
|
371
|
+
1. **Rules.** The doc is split into rules (R1, R2 …) without AI. Rule ids stay
|
|
372
|
+
stable when the doc changes. Every rule records its source, the files that
|
|
373
|
+
test it, why it was skipped, and its earlier wordings.
|
|
374
|
+
2. **Plan, without AI.** Each numbered section of the doc becomes one test file
|
|
375
|
+
(`"planner": "ai"` asks the AI instead). Rules another journey already tests
|
|
376
|
+
are not planned again. The plan check (`plan-check.md`) lists every rule that
|
|
377
|
+
gets tests, every skip with its reason, and any **must** rule left out.
|
|
378
|
+
3. **Code facts, once.** The source the doc points at (the files it cites, the
|
|
379
|
+
components its controls are named after, the pages and hooks that reach
|
|
380
|
+
them) is collected without AI and summarised in one call.
|
|
381
|
+
4. **Writing.** Each call writes a few files (`filesPerCall`, default 3) and
|
|
382
|
+
gets only their rules, their controls' facts and the testkit's exports; the
|
|
383
|
+
shared rules for writing a test are a cached system prompt. Tests find every
|
|
384
|
+
element with `kit.find(page, kind, name)`, check outcomes rather than markup,
|
|
385
|
+
and stub a state they cannot create.
|
|
386
|
+
5. **Checks before anything runs.** A file is refused, and written once more
|
|
387
|
+
with the reason, if it sleeps for a fixed time, skips because an element is
|
|
388
|
+
missing, has no test or no assertion, spends more than `maxTestsPerRule`
|
|
389
|
+
tests on one rule, or names a label found neither in the code nor the doc.
|
|
390
|
+
6. **Every rule, a test that can fail.** Each test is checked on its own:
|
|
391
|
+
every planned rule has a test, or a `// cannot-test: R7 — <the state this
|
|
392
|
+
area cannot set up>` line ("the locator is unknown" is not such a state);
|
|
393
|
+
no assertion accepts the good and the bad outcome alike, compares the page's
|
|
394
|
+
whole text, sits only inside an `if`, or repeats what its own stub returned;
|
|
395
|
+
`kit.find` is awaited and finds elements by the names the screen shows;
|
|
396
|
+
`kit.openStep` names a step the testkit has. The tests that fail a check are
|
|
397
|
+
taken out and their rules written again in ONE small call; what is still
|
|
398
|
+
wrong is removed and reported. The summary line says how many planned rules
|
|
399
|
+
have a test, which cannot be tested here and why, and which have none, and
|
|
400
|
+
`rules.json` / `plan-check.md` record the same.
|
|
401
|
+
7. **A changed doc.** Only what changed is touched: a renamed label is replaced
|
|
402
|
+
in its tests, a removed rule's tests are deleted (both without AI); a
|
|
403
|
+
reworded or new rule gets its tests written by one small call.
|
|
404
|
+
|
|
405
|
+
**A feature that is not live yet.** Put `Status: not built` or
|
|
406
|
+
`Status: behind flag <name>` in the journey doc's header. Every file of the
|
|
407
|
+
journey is stamped `// testforge-pending: …`, and `testforge run` (and `bind`)
|
|
408
|
+
hold it back until the line is removed; `--all` runs it anyway.
|
|
409
|
+
|
|
410
|
+
`generate` runs nothing. `testforge run` runs the suite; `testforge review`
|
|
411
|
+
judges failures when you ask for it.
|
|
412
|
+
|
|
413
|
+
Every Claude call reports its real token use and cost; `TESTFORGE_TRACE=1`
|
|
414
|
+
also records each call's prompt and events under `testforge/runs/trace/`.
|
|
415
|
+
|
|
416
|
+
## Project state
|
|
417
|
+
|
|
418
|
+
Everything TestForge keeps lives in `testforge/`. Commit it, except `runs/`
|
|
419
|
+
(run output and AI logs): a teammate who pulls it starts from your memory, plans
|
|
420
|
+
and code facts instead of paying for them again. A project set up by an older
|
|
421
|
+
version is moved from `test-artifacts/` on the first command.
|
|
383
422
|
|
|
384
423
|
## Notes
|
|
385
424
|
|
package/dist/cli/index.js
CHANGED
|
@@ -21,6 +21,8 @@
|
|
|
21
21
|
import { VERSION } from '../core/index.js';
|
|
22
22
|
import { runGenerate } from '../core/generate/run-generate.js';
|
|
23
23
|
import { runReview } from '../core/generate/review-journey.js';
|
|
24
|
+
import { migrateState } from '../core/paths.js';
|
|
25
|
+
import { acceptNames, bindJourneys } from '../core/generate/bind.js';
|
|
24
26
|
import { runClean } from '../core/run/clean.js';
|
|
25
27
|
import { aiUsage, describeUsage } from '../core/generate/ai-usage.js';
|
|
26
28
|
import { envDiff, describeEnvDiff } from '../core/project/env-diff.js';
|
|
@@ -62,6 +64,7 @@ function sourceOptions() {
|
|
|
62
64
|
reference,
|
|
63
65
|
source: flagValue('--source'),
|
|
64
66
|
replace: process.argv.includes('--replace'),
|
|
67
|
+
refresh: process.argv.includes('--refresh'),
|
|
65
68
|
};
|
|
66
69
|
}
|
|
67
70
|
/**
|
|
@@ -91,11 +94,11 @@ function printUsage() {
|
|
|
91
94
|
console.log(' testforge generate --basic Use the basic generator only');
|
|
92
95
|
console.log(' testforge generate --force Regenerate everything');
|
|
93
96
|
console.log(' testforge generate --no-wait Stop when the AI hits its usage limit (default: wait for the reset and resume)');
|
|
94
|
-
console.log(' testforge generate --
|
|
97
|
+
console.log(' testforge generate --plan-only Plan and check each journey (testforge/memory/<area>/<id>/plan-check.md), write no tests');
|
|
95
98
|
console.log(' testforge generate --journey <name> Generate just ONE journey');
|
|
96
99
|
console.log(' testforge generate --area <area> Generate a whole area (e.g. api)');
|
|
97
100
|
console.log(' testforge discover Draft journey docs (asks: map or reference)');
|
|
98
|
-
console.log(' testforge discover --journey <name> Draft just ONE journey doc');
|
|
101
|
+
console.log(' testforge discover --journey <name> Draft just ONE journey doc', ' testforge discover --journey <name> --reference <x> [--refresh] One journey from a reference: the reference is read once and kept (--refresh reads it again)');
|
|
99
102
|
console.log(' testforge discover --reference <file|url> Base docs on a reference');
|
|
100
103
|
console.log(' testforge discover --tickets <folder> Base docs on exported tickets (one file per ticket); each source names its ticket');
|
|
101
104
|
console.log(' (a reference only writes the journeys it');
|
|
@@ -107,6 +110,7 @@ function printUsage() {
|
|
|
107
110
|
console.log(' testforge env Write .env.testing.example from docs');
|
|
108
111
|
console.log(' testforge env --diff Only the variables the suite reads that the env file does not set');
|
|
109
112
|
console.log(' testforge ci --github Write a GitHub Actions workflow that runs the suite against a deployed environment');
|
|
113
|
+
console.log(' testforge bind [--journey j] [--accept "doc=code"] Which tests the code is ready for, which wait for the feature, which use another name; --accept takes the code\'s wording (no AI)');
|
|
110
114
|
console.log(' testforge review [--journey j] [--rerun] Judge the last failures again (defect / test fault / environment …), no repair');
|
|
111
115
|
console.log(' testforge clean [--run <id> | --older-than <days>] [--yes] Remove what earlier runs created (dry run without --yes)');
|
|
112
116
|
console.log(' testforge run [--area a] [--journey j] Run the generated tests (Jest/Playwright)');
|
|
@@ -115,6 +119,10 @@ function printUsage() {
|
|
|
115
119
|
console.log(' testforge run -- <flags> Pass anything else straight to the runner');
|
|
116
120
|
}
|
|
117
121
|
async function main() {
|
|
122
|
+
// A project set up by an older version keeps its state under test-artifacts/: move it once.
|
|
123
|
+
const moved = command && command !== 'help' && command !== '--help' ? migrateState() : [];
|
|
124
|
+
if (moved.length)
|
|
125
|
+
console.log(`Moved TestForge's state into testforge/ (${moved.length} item(s)); nothing was rebuilt.\n`);
|
|
118
126
|
if (command === 'update') {
|
|
119
127
|
await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
|
|
120
128
|
}
|
|
@@ -146,6 +154,32 @@ async function main() {
|
|
|
146
154
|
const days = flagValue('--older-than');
|
|
147
155
|
console.log(await runClean({ runId: flagValue('--run'), olderThanDays: days != null ? Number(days) : undefined, yes: process.argv.includes('--yes') }));
|
|
148
156
|
}
|
|
157
|
+
else if (command === 'bind') {
|
|
158
|
+
// After a feature ships: which tests can run, which wait for the code, which use a different name.
|
|
159
|
+
const accept = process.argv.flatMap((a, k) => (process.argv[k - 1] === '--accept' ? [a] : []));
|
|
160
|
+
if (accept.length) {
|
|
161
|
+
const pairs = accept.map((a) => a.split('=')).filter((p) => p.length === 2 && p[0] && p[1]);
|
|
162
|
+
const notes = acceptNames(pairs, { journey: flagValue('--journey') });
|
|
163
|
+
for (const n of notes)
|
|
164
|
+
console.log(` ${n}`);
|
|
165
|
+
console.log(notes.length ? '\nRun `testforge generate` to record the rename in the journey\'s memory (no AI).' : 'Nothing used those names.');
|
|
166
|
+
}
|
|
167
|
+
else {
|
|
168
|
+
for (const r of bindJourneys({ journey: flagValue('--journey') })) {
|
|
169
|
+
const by = (s) => r.files.filter((f) => f.state === s);
|
|
170
|
+
console.log(` ${r.journey}: ${by('built').length} ready, ${by('partial').length} partly built, ${by('not-built').length} not built yet, ${by('renamed').length} with a different name`);
|
|
171
|
+
for (const f of r.files) {
|
|
172
|
+
if (f.state === 'built')
|
|
173
|
+
continue;
|
|
174
|
+
const what = f.state === 'renamed'
|
|
175
|
+
? f.renamed.map((x) => `"${x.name}" → code has "${x.nearest}"`).join('; ')
|
|
176
|
+
: `not in the code: ${f.missing.slice(0, 4).map((m) => `"${m}"`).join(', ')}${f.missing.length > 4 ? ' …' : ''}`;
|
|
177
|
+
console.log(` ${f.state.padEnd(9)} ${f.file}: ${what}`);
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
console.log('\nA different name is not a defect: change the code to the doc, or accept the code\'s wording with --accept "doc wording=code wording".');
|
|
181
|
+
}
|
|
182
|
+
}
|
|
149
183
|
else if (command === 'review') {
|
|
150
184
|
console.log('Reviewing the last failures of each journey (no repair)...\n');
|
|
151
185
|
const summaries = await runReview({ journey: flagValue('--journey'), rerun: process.argv.includes('--rerun') });
|
|
@@ -159,7 +193,7 @@ async function main() {
|
|
|
159
193
|
else
|
|
160
194
|
console.log(` ${r.journey}: ${r.failures} failure(s) → ` + Object.entries(r.verdicts).map(([k, n]) => `${n} ${k}`).join(', '));
|
|
161
195
|
}
|
|
162
|
-
console.log('\nVerdicts written to
|
|
196
|
+
console.log('\nVerdicts written to testforge/runs/defect-review/. The next run publishes them with each failed test.');
|
|
163
197
|
}
|
|
164
198
|
else if (command === 'discover') {
|
|
165
199
|
console.log('Drafting journey files with AI...');
|
|
@@ -189,6 +223,7 @@ async function main() {
|
|
|
189
223
|
journey: flagValue('--journey'),
|
|
190
224
|
envFile: flagValue('--env-file'),
|
|
191
225
|
passThrough,
|
|
226
|
+
all: process.argv.includes('--all'),
|
|
192
227
|
});
|
|
193
228
|
process.exitCode = code;
|
|
194
229
|
}
|
|
@@ -203,11 +238,7 @@ async function main() {
|
|
|
203
238
|
: process.argv.includes('--wait-for-limit')
|
|
204
239
|
? true
|
|
205
240
|
: undefined, // undefined -> config.waitForLimit (default on)
|
|
206
|
-
|
|
207
|
-
? false
|
|
208
|
-
: process.argv.includes('--verify')
|
|
209
|
-
? true
|
|
210
|
-
: undefined, // undefined -> config.verify (default on)
|
|
241
|
+
planOnly: process.argv.includes('--plan-only'),
|
|
211
242
|
});
|
|
212
243
|
if (results.length === 0) {
|
|
213
244
|
console.log('No journey files found in docs/. Add a .md file and try again.');
|
|
@@ -238,9 +269,6 @@ async function main() {
|
|
|
238
269
|
for (const w of r.warnings ?? []) {
|
|
239
270
|
console.log(` ! ${w}`);
|
|
240
271
|
}
|
|
241
|
-
if (r.verified === true)
|
|
242
|
-
console.log(' verified: every file ran; every failure left was reviewed and is listed below');
|
|
243
|
-
printReviewed(r.reviewed ?? []);
|
|
244
272
|
}
|
|
245
273
|
const generated = results.filter((r) => r.status !== 'skipped').length;
|
|
246
274
|
// Rows with no test dir are notices (a non-journey file, the AI going
|
|
@@ -318,28 +346,6 @@ async function main() {
|
|
|
318
346
|
printUsage();
|
|
319
347
|
}
|
|
320
348
|
}
|
|
321
|
-
/** Reviewed failures, one line per root cause, strongest first. */
|
|
322
|
-
function printReviewed(reviewed) {
|
|
323
|
-
const sections = [
|
|
324
|
-
['defect', 'DEFECTS — a [spec] requirement the app breaks'],
|
|
325
|
-
['finding', 'FINDINGS — real app behaviour against a [baseline]/[observed] criterion (lower priority)'],
|
|
326
|
-
['pending-decision', 'PENDING DECISIONS — not requirements yet; take them to product, do not report as defects'],
|
|
327
|
-
];
|
|
328
|
-
for (const [verdict, heading] of sections) {
|
|
329
|
-
const rows = reviewed.filter((r) => r.verdict === verdict);
|
|
330
|
-
if (!rows.length)
|
|
331
|
-
continue;
|
|
332
|
-
console.log(` ${heading}:`);
|
|
333
|
-
for (const r of rows) {
|
|
334
|
-
const where = r.tests.slice(0, 3).map((t) => `${basename(t.file)}:${t.line}`).join(', ');
|
|
335
|
-
const more = r.tests.length > 3 ? ` +${r.tests.length - 3} more` : '';
|
|
336
|
-
console.log(` - ${r.claim}`);
|
|
337
|
-
console.log(` ${r.tests.length} test(s): ${where}${more}${r.evidence ? ` · app: ${r.evidence}` : ''}`);
|
|
338
|
-
if (r.reason)
|
|
339
|
-
console.log(` ${r.reason}`);
|
|
340
|
-
}
|
|
341
|
-
}
|
|
342
|
-
}
|
|
343
349
|
main().catch((err) => {
|
|
344
350
|
console.error(err instanceof Error ? err.message : String(err));
|
|
345
351
|
process.exit(1);
|
package/dist/core/docs/cache.js
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
import { createHash } from 'node:crypto';
|
|
13
13
|
import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, statSync, } from 'node:fs';
|
|
14
14
|
import { dirname, join, extname } from 'node:path';
|
|
15
|
-
import { STATE_DIR, LEGACY_STATE_DIR } from '../paths.js';
|
|
15
|
+
import { STATE_DIR, LEGACY_STATE_DIR, OLD_STATE_DIR } from '../paths.js';
|
|
16
16
|
/** Make a short fingerprint of a file's content. */
|
|
17
17
|
export function hashContent(content) {
|
|
18
18
|
return createHash('sha256').update(content).digest('hex');
|
|
@@ -66,6 +66,7 @@ const SKIP_DIRS = new Set([
|
|
|
66
66
|
'docs',
|
|
67
67
|
STATE_DIR,
|
|
68
68
|
LEGACY_STATE_DIR,
|
|
69
|
+
OLD_STATE_DIR,
|
|
69
70
|
'generated-tests',
|
|
70
71
|
]);
|
|
71
72
|
export function hashProjectSource(rootDir) {
|
|
@@ -21,8 +21,10 @@ import { loadConfig, splitCommand, withEditTools } from '../project/config.js';
|
|
|
21
21
|
import { DEEP_DOC_GUIDANCE } from './doc-guidance.js';
|
|
22
22
|
import { promptConfirm } from '../prompt.js';
|
|
23
23
|
import { loadSourceChoice, resolveDocsSource, saveJourneyReferences, saveSourceChoice, } from './source-of-truth.js';
|
|
24
|
-
import { analyzeReference, draftDocsFromReference, DIFF_PATH, } from './reference.js';
|
|
25
|
-
import { reviewDocs } from './review.js';
|
|
24
|
+
import { analyzeReference, buildJourneyDraftPrompt, draftDocsFromReference, draftJourneyDoc, DIFF_PATH, } from './reference.js';
|
|
25
|
+
import { buildLeanReviewPrompt, enforceSpecSources, reviewDocs } from './review.js';
|
|
26
|
+
import { readReference, sectionFor } from './reference-text.js';
|
|
27
|
+
import { sourceExcerpts } from './excerpts.js';
|
|
26
28
|
import { runAiSync } from '../generate/ai-usage.js';
|
|
27
29
|
/** Scope line: one journey, or all of them. */
|
|
28
30
|
function scopeLines(journey) {
|
|
@@ -168,6 +170,12 @@ function snapshotDocs(docsDir) {
|
|
|
168
170
|
*/
|
|
169
171
|
export async function runDiscover(srcDir, docsDir, opts = {}, journey) {
|
|
170
172
|
const source = await resolveDocsSource(opts);
|
|
173
|
+
// One journey from a reference: no comparison of the whole product, the
|
|
174
|
+
// reference read once and cut to this journey, drafted and reviewed in one
|
|
175
|
+
// call each with no tools.
|
|
176
|
+
if (source.mode === 'reference' && source.reference && journey) {
|
|
177
|
+
return draftOneJourney(srcDir, docsDir, source.reference, journey, opts.refresh);
|
|
178
|
+
}
|
|
171
179
|
if (source.mode === 'reference' && source.reference) {
|
|
172
180
|
console.log(`\nComparing the reference against testforge.md: ${source.reference}`);
|
|
173
181
|
console.log('(reading the reference and the map — this may take a moment)\n');
|
|
@@ -287,3 +295,42 @@ async function discoverFromCode(srcDir, docsDir, journey) {
|
|
|
287
295
|
await reviewDocs(docsDir, changed, srcDir);
|
|
288
296
|
return out;
|
|
289
297
|
}
|
|
298
|
+
const docTitle = (text) => /^# Journey:\s*(.+)$/m.exec(text)?.[1]?.trim();
|
|
299
|
+
const kebab = (t) => t.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
|
|
300
|
+
async function draftOneJourney(srcDir, docsDir, reference, journey, refresh = false) {
|
|
301
|
+
const config = loadConfig();
|
|
302
|
+
const text = readReference(reference, refresh);
|
|
303
|
+
const section = sectionFor(text, journey);
|
|
304
|
+
const excerpts = sourceExcerpts(section, [srcDir, ...(config.relatedDirs ?? [])]);
|
|
305
|
+
const docs = [...snapshotDocs(docsDir)].filter(([p, t]) => !notAJourneyDoc(p, t));
|
|
306
|
+
const existing = docs.find(([, t]) => kebab(docTitle(t) ?? '') === kebab(journey))?.[0];
|
|
307
|
+
const others = docs.filter(([p]) => p !== existing).map(([p, t]) => `${docTitle(t) ?? p} (${p})`);
|
|
308
|
+
const areas = Object.keys(config.testOutput ?? {}).length ? Object.keys(config.testOutput ?? {}) : ['web', 'api'];
|
|
309
|
+
const owner = loadSourceChoice()?.journeys?.[existing ?? ''];
|
|
310
|
+
if (existing && owner && owner !== reference) {
|
|
311
|
+
throw new Error(`${existing} comes from another reference (${owner}); not redrafting it from ${reference}.`);
|
|
312
|
+
}
|
|
313
|
+
console.log(`\nDrafting "${journey}" from the reference (${section.length.toLocaleString('en-GB')} characters of it, one call, no tools)...`);
|
|
314
|
+
const draft = draftJourneyDoc(buildJourneyDraftPrompt(journey, section, excerpts, areas, others));
|
|
315
|
+
if (!draft.markdown)
|
|
316
|
+
throw new Error('The AI answered without a journey doc.');
|
|
317
|
+
console.log('Reviewing it for missing scenarios (one call, no tools)...');
|
|
318
|
+
let doc = draft.markdown;
|
|
319
|
+
try {
|
|
320
|
+
const reviewed = draftJourneyDoc(buildLeanReviewPrompt(doc, section, excerpts, others));
|
|
321
|
+
if (reviewed.markdown)
|
|
322
|
+
doc = reviewed.markdown;
|
|
323
|
+
}
|
|
324
|
+
catch (e) {
|
|
325
|
+
console.log(` review failed, kept as drafted (${e.message.split('\n')[0]})`);
|
|
326
|
+
}
|
|
327
|
+
const { markdown, retagged } = enforceSpecSources(doc);
|
|
328
|
+
if (retagged)
|
|
329
|
+
console.log(` ${retagged} [spec] line(s) without a quoted source re-tagged [observed]`);
|
|
330
|
+
const rel = existing ?? join(draft.area && areas.includes(draft.area) ? draft.area : areas[0], `${kebab(docTitle(markdown) ?? journey)}.md`);
|
|
331
|
+
mkdirSync(join(docsDir, rel, '..'), { recursive: true });
|
|
332
|
+
writeFileSync(join(docsDir, rel), markdown, 'utf-8');
|
|
333
|
+
if (!existing)
|
|
334
|
+
saveJourneyReferences([rel], reference);
|
|
335
|
+
return `${existing ? 'Rewrote' : 'Wrote'} ${join(docsDir, rel)} from the reference. Please review it, then run \`testforge generate --journey ${kebab(journey)}\`.`;
|
|
336
|
+
}
|
|
@@ -45,30 +45,68 @@ export const DEEP_DOC_GUIDANCE = [
|
|
|
45
45
|
`- "[spec]" — the source states this behaviour. A failure is a defect`,
|
|
46
46
|
` against the spec.`,
|
|
47
47
|
`- "[baseline]" — universal robustness that holds for ANY well-built app.`,
|
|
48
|
-
`
|
|
49
|
-
`
|
|
48
|
+
` The client describes what to build; it rarely says how it must NOT`,
|
|
49
|
+
` break. That is your job: think like a tester who wants to break it and`,
|
|
50
|
+
` like a user who is technical enough to find its loopholes. Add these`,
|
|
51
|
+
` YOURSELF for every screen, form, list, action and endpoint the journey`,
|
|
52
|
+
` has, even when the source says nothing about them. Write each one's`,
|
|
53
|
+
` LEVEL and CATEGORY right after the tag, e.g. "[baseline] (must,`,
|
|
54
|
+
` security)". Every "must" that fits the journey has to be there.`,
|
|
55
|
+
` MUST — security:`,
|
|
56
|
+
` * One user never reaches another user's data or actions: changing an id`,
|
|
57
|
+
` in a URL, a request or a hidden field is refused (not "not found by`,
|
|
58
|
+
` accident" — refused). A page or endpoint that needs a login is refused`,
|
|
59
|
+
` without one; a role cannot use another role's actions.`,
|
|
60
|
+
` * Whatever the client sends is not trusted: a price, an amount, a`,
|
|
61
|
+
` discount, a role, an owner id or a status changed in the request is`,
|
|
62
|
+
` refused or recomputed by the server.`,
|
|
63
|
+
` * An HTML, script or injection payload in any input is stored and shown`,
|
|
64
|
+
` as text, never executed, never a server error (5xx).`,
|
|
65
|
+
` * Passwords are masked; passwords, tokens and codes never appear in a`,
|
|
66
|
+
` URL, a query string or a log the user can see.`,
|
|
67
|
+
` * Repeated wrong attempts (password, code, coupon, PIN) are limited and`,
|
|
68
|
+
` the limit is told to the user.`,
|
|
69
|
+
` MUST — money and data:`,
|
|
70
|
+
` * Double submission: an action with a side effect (pay, book, order,`,
|
|
71
|
+
` send, create) done`,
|
|
72
|
+
` twice — a double click, a repeated Enter, two tabs, a retry after a`,
|
|
73
|
+
` timeout — leaves exactly one record, one charge, one message.`,
|
|
74
|
+
` * A flow that fails half-way leaves no half-made record and no money`,
|
|
75
|
+
` taken without the thing bought; the user sees what happened.`,
|
|
76
|
+
` * Amounts, totals, discounts and limits are the same on every screen and`,
|
|
77
|
+
` in the server's answer.`,
|
|
78
|
+
` MUST — concurrency:`,
|
|
79
|
+
` * Two users or two tabs acting on the same thing at once (the same slot,`,
|
|
80
|
+
` the last item, a code with limited uses, the same record edited) —`,
|
|
81
|
+
` exactly one wins and the other gets a clear reason.`,
|
|
82
|
+
` * Nothing hangs: every action ends within its timeout with a result or a`,
|
|
83
|
+
` visible error; a request that never answers never leaves a spinner`,
|
|
84
|
+
` forever or a control disabled for good.`,
|
|
85
|
+
` MUST — error boundaries:`,
|
|
86
|
+
` * Every load and every action, when the server answers 500, the network`,
|
|
87
|
+
` drops, it times out, or the answer is empty or malformed: a visible`,
|
|
88
|
+
` message, the page stays usable, no blank page, no crash, the user's`,
|
|
89
|
+
` input is kept.`,
|
|
90
|
+
` * One part of a page failing never takes the whole page down.`,
|
|
91
|
+
` SHOULD — validation:`,
|
|
50
92
|
` * Inputs: empty and whitespace-only values; leading/trailing spaces;`,
|
|
51
|
-
` a very long value (5,000 characters); unicode and emoji
|
|
52
|
-
`
|
|
53
|
-
`
|
|
54
|
-
`
|
|
55
|
-
` *
|
|
56
|
-
`
|
|
57
|
-
`
|
|
58
|
-
`
|
|
59
|
-
`
|
|
60
|
-
`
|
|
61
|
-
`
|
|
62
|
-
`
|
|
63
|
-
`
|
|
64
|
-
`
|
|
65
|
-
`
|
|
66
|
-
` *
|
|
67
|
-
`
|
|
68
|
-
` * Access: a page or action that needs a login is not usable logged`,
|
|
69
|
-
` out; one user never sees another user's data.`,
|
|
70
|
-
` * Choice controls (dropdown, radio, checkbox, list): every option`,
|
|
71
|
-
` shown can be selected and submitted.`,
|
|
93
|
+
` a very long value (5,000 characters); unicode and emoji. Each is`,
|
|
94
|
+
` accepted or rejected with a visible reason, never a crash.`,
|
|
95
|
+
` * No silent failure: every rejected submit says why, next to the field`,
|
|
96
|
+
` or the form (a wrong password with no message is a defect).`,
|
|
97
|
+
` * The screen and the server accept exactly the same values.`,
|
|
98
|
+
` SHOULD — state and navigation:`,
|
|
99
|
+
` * Refresh, Back, Forward and opening a mid-flow URL directly never crash`,
|
|
100
|
+
` or leave a blank page; an expired session sends the user to log in and`,
|
|
101
|
+
` back; after logging out nothing of the account is shown.`,
|
|
102
|
+
` * Choice controls (dropdown, radio, checkbox, list): every option shown`,
|
|
103
|
+
` can be selected and submitted; long lists page or scroll.`,
|
|
104
|
+
` SHOULD — regression:`,
|
|
105
|
+
` * Every defect the source or the doc already names keeps a test, so it`,
|
|
106
|
+
` cannot quietly come back.`,
|
|
107
|
+
` COULD — accessibility:`,
|
|
108
|
+
` * The form can be filled and submitted with the keyboard alone; Enter`,
|
|
109
|
+
` submits; pop-ups close with Escape; controls have readable names.`,
|
|
72
110
|
` [baseline] is ROBUSTNESS only. Never present product POLICY you`,
|
|
73
111
|
` invented (a maximum length, a password rule, an error wording, a`,
|
|
74
112
|
` limit) as [baseline].`,
|
|
@@ -89,15 +127,18 @@ export const DEEP_DOC_GUIDANCE = [
|
|
|
89
127
|
` becomes a test that fails on a correct product.`,
|
|
90
128
|
` An [observed] criterion is about what the customer sees or what the`,
|
|
91
129
|
` server accepts or refuses — never an implementation detail (a storage`,
|
|
92
|
-
` key name, a CSS class, an internal flag, a request field name
|
|
93
|
-
` source itself is about it.`,
|
|
94
|
-
` Only what a test can drive is a criterion. Behaviour no
|
|
95
|
-
` endpoint can reach (
|
|
96
|
-
`
|
|
130
|
+
` key name, a CSS class, an internal flag, a request field name, a log`,
|
|
131
|
+
` line, a lock, a cache) unless the source itself is about it.`,
|
|
132
|
+
` Only what a test in THIS area can drive is a criterion. Behaviour no`,
|
|
133
|
+
` screen or endpoint of this area can reach (a backend job, another app,`,
|
|
134
|
+
` an operator's action) is not tagged — list it under "## Not requirements"`,
|
|
135
|
+
` with where it is checked, or mention it in Test setup if the generator`,
|
|
136
|
+
` needs to know.`,
|
|
97
137
|
``,
|
|
98
138
|
`METHOD. Work through these steps for every journey, in order:`,
|
|
99
139
|
`1. CONTROLS. Start the doc with a "## Controls" table listing EVERY`,
|
|
100
|
-
` interactive element and where it appears
|
|
140
|
+
` interactive element and where it appears, each row's first cell its id`,
|
|
141
|
+
` C1, C2, C3 …: each input, button, link,`,
|
|
101
142
|
` checkbox, code box, dropdown, pop-up, menu item, and the page itself.`,
|
|
102
143
|
` Secondary controls count as much as the main form: Resend, Back,`,
|
|
103
144
|
` show/hide, close, apply, sign-out, provider buttons. Apply every`,
|