@activate-core/test-forge 1.2.2 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +67 -28
  2. package/dist/cli/index.js +98 -34
  3. package/dist/core/docs/cache.js +2 -1
  4. package/dist/core/docs/discover.js +51 -4
  5. package/dist/core/docs/doc-guidance.js +61 -23
  6. package/dist/core/docs/excerpts.js +245 -0
  7. package/dist/core/docs/grounding.js +51 -24
  8. package/dist/core/docs/reference-text.js +78 -0
  9. package/dist/core/docs/reference.js +68 -5
  10. package/dist/core/docs/review.js +81 -0
  11. package/dist/core/docs/source-of-truth.js +1 -1
  12. package/dist/core/docs/tickets.js +56 -0
  13. package/dist/core/gates/coverage.js +12 -19
  14. package/dist/core/gates/validate.js +277 -4
  15. package/dist/core/generate/ai-generator.js +243 -96
  16. package/dist/core/generate/ai-session.js +8 -0
  17. package/dist/core/generate/ai-usage.js +150 -0
  18. package/dist/core/generate/bind.js +91 -0
  19. package/dist/core/generate/defect-review.js +213 -20
  20. package/dist/core/generate/doc-diff.js +139 -0
  21. package/dist/core/generate/harness.js +52 -32
  22. package/dist/core/generate/journey-memory.js +325 -0
  23. package/dist/core/generate/plan-cache.js +114 -7
  24. package/dist/core/generate/required-data.js +75 -0
  25. package/dist/core/generate/review-journey.js +104 -0
  26. package/dist/core/generate/run-generate.js +193 -80
  27. package/dist/core/generate/seed-generator.js +2 -2
  28. package/dist/core/generate/suite-planner.js +138 -25
  29. package/dist/core/generate/test-blocks.js +84 -0
  30. package/dist/core/generate/update-journey.js +115 -0
  31. package/dist/core/generate/verify.js +9 -4
  32. package/dist/core/paths.js +90 -13
  33. package/dist/core/project/config.js +25 -2
  34. package/dist/core/project/env-diff.js +122 -0
  35. package/dist/core/project/env-template.js +3 -2
  36. package/dist/core/project/init.js +2 -2
  37. package/dist/core/project/map.js +2 -2
  38. package/dist/core/run/ci.js +94 -0
  39. package/dist/core/run/clean.js +111 -0
  40. package/dist/core/run/readiness.js +119 -0
  41. package/dist/core/run/report.js +2 -0
  42. package/dist/core/run/run-tests.js +23 -1
  43. package/package.json +1 -1
  44. package/reporters/_shared.cjs +24 -3
  45. package/reporters/jest-reporter.cjs +3 -1
  46. package/reporters/playwright-reporter.cjs +3 -1
  47. package/reporters/run-data.cjs +66 -0
  48. package/reporters/ui-helpers.cjs +133 -0
  49. package/dist/core/generate/verify-journey.js +0 -174
package/README.md CHANGED
@@ -94,7 +94,7 @@ never touched.
94
94
  | Command | What it does |
95
95
  | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
96
96
  | `testforge setup` | One-time setup: runs init, map, and discover in order, with progress. |
97
- | `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. |
97
+ | `testforge generate` | Writes the tests for each journey doc. A changed doc updates only the tests of the rules that changed. Writes only: it runs nothing. |
98
98
  | `testforge update` | Refreshes the map, docs and tests when the **source code** or the **reference** has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |
99
99
 
100
100
  ### Advanced (run a single step)
@@ -105,7 +105,9 @@ never touched.
105
105
  | `testforge map` | Rebuild the project map (`testforge.md`) only. |
106
106
  | `testforge discover` | Redraft the per-journey docs in `docs/` only. |
107
107
  | `testforge env` | Rebuild `.env.testing.example` from the docs only. |
108
- | `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). |
108
+ | `testforge run` | Run the generated tests (`--journey` / `--area` / `--env-file`). Files whose feature is not in the code yet are held back as pending (`--all` runs them). |
109
+ | `testforge bind` | After a feature ships: which test files the code is ready for, which wait for the feature, which use a name the code spells differently. `--accept "doc=code"` takes the code's wording. No AI. |
110
+ | `testforge review` | Ask an AI reviewer whether the last failures are defects, test mistakes or the environment. On demand only. |
109
111
  | `testforge report` | Publish results to the dashboard — `--results` after a run, `--inventory` to register every generated check. |
110
112
 
111
113
  ### Flags worth knowing
@@ -115,6 +117,9 @@ never touched.
115
117
  | `--journey <name>` | discover, generate, run | Work on one journey instead of all of them. |
116
118
  | `--area <name>` | discover, generate, run | Work on one area (api, web, ops…). |
117
119
  | `--force` | generate | Regenerate even when the doc has not changed. |
120
+ | `--plan-only` | generate | Plan and check each journey (`testforge/memory/<area>/<id>/plan-check.md`), write no tests. |
121
+ | `--refresh` | discover | Fetch the reference again instead of the copy kept from the last run. |
122
+ | `--all` | run | Also run files whose feature is not in the code yet. |
118
123
  | `--refresh-docs` | update | Re-draft the docs even when nothing detectably changed — see below. |
119
124
  | `--reference <file\|url>` | setup | Base the docs on a spec instead of the code map. |
120
125
  | `--source map` | setup | Base the docs on `testforge.md`, no prompt. |
@@ -198,7 +203,7 @@ _expected_ behavior — a spec or requirements doc in the repo, a PDF, a
198
203
  website, or a Notion page. testforge then:
199
204
 
200
205
  1. reads the reference and compares it against `testforge.md`,
201
- 2. writes a full comparison to **`testforge.diff.md`** and prints a short
206
+ 2. writes a full comparison to **`testforge/reference/diff.md`** and prints a short
202
207
  summary of the differences,
203
208
  3. asks you to confirm, and on confirmation **drafts the docs from the
204
209
  reference** — expected behavior comes from the reference, while routes,
@@ -207,9 +212,9 @@ website, or a Notion page. testforge then:
207
212
 
208
213
  The reference becomes the source of truth — expected behavior from a human
209
214
  or client, not just what the code happens to do. The choice is remembered
210
- in `test-artifacts/source.json`, so a later `testforge update` refreshes the
215
+ in `testforge/source.json`, so a later `testforge update` refreshes the
211
216
  docs from the same source without asking again. Your review of
212
- `testforge.diff.md` stays as an audit of what differed and why.
217
+ `testforge/reference/diff.md` stays as an audit of what differed and why.
213
218
 
214
219
  **Non-interactive use (CI, scripts):**
215
220
 
@@ -285,6 +290,22 @@ the project map list the env var names each journey needs.
285
290
 
286
291
  The test runner produces its own pass/fail report.
287
292
 
293
+ ### In CI (GitHub Actions)
294
+
295
+ ```bash
296
+ npx testforge ci --github # run only when started by hand
297
+ npx testforge ci --github --schedule weekdays # also Mon–Fri 06:00 UTC (daily, weekly or a cron line too)
298
+ npx testforge ci --github --env-name staging --force
299
+ ```
300
+
301
+ The workflow runs `testforge run` when started by hand (and on the schedule, if given), against a
302
+ deployed environment, and publishes each run to the dashboard. It reads its
303
+ values from repository secrets with the same names as your env file
304
+ (TestForge reads only the names, never the values), plus `TESTFORGE_KEY`,
305
+ and the dashboard address from the repository variable
306
+ `TESTFORGE_DASHBOARD_URL`. Point the base URL secrets at the deployed app.
307
+ Test results are kept as a workflow artifact for 14 days.
308
+
288
309
  ## Configuration
289
310
 
290
311
  `testforge setup` writes `testforge.config.json` for you. You can edit it:
@@ -341,29 +362,47 @@ The test runner produces its own pass/fail report.
341
362
  reported as a pending decision, never as a defect. Keeps the reference
342
363
  you share with a client clean while the tests still know what is undecided.
343
364
 
344
- ## How `generate` checks its own work
345
-
346
- Every test is run against the real app before it is handed over, so a
347
- mistake in a generated test does not reach your team as a product defect:
348
-
349
- 1. **Each file on its own.** A test that fails because of its own mistake
350
- (a wrong locator, a field the server never sends, a bad precondition) is
351
- repaired from the runner's output. A guard refuses any repair that
352
- deletes a test, adds a skip or removes an assertion.
353
- 2. **The whole journey together**, the way CI runs it. Files that pass alone
354
- but break each other (one signs in again as a shared account and ends
355
- another's session) are repaired with that context.
356
- 3. **An independent review of every claimed defect.** A separate reviewer
357
- starts from "not proven" and must find the application code at fault.
358
- `generate` then lists, one line per root cause:
359
- - **defects**: a [spec] requirement the app breaks;
360
- - **findings**: real behaviour against a [baseline]/[observed] criterion;
361
- - **pending decisions**: covered by `decisions`.
362
- A claim it rejects is repaired once more. Anything still unexplained is
363
- reported as unverified, with the advice not to report it.
364
-
365
- Each run is time-limited (`verifyTimeoutMs`, per file). On expiry the
366
- runner stops with everything it started.
365
+ ## How `generate` keeps the AI cost down
366
+
367
+ The journey doc is read once and kept as the journey's **memory**
368
+ (`testforge/memory/<area>/<id>/`): its rules with their source, level, tests and
369
+ history, the plan, and the code facts. No later step sends the whole doc again.
370
+
371
+ 1. **Rules.** The doc is split into rules (R1, R2 …) without AI. Rule ids stay
372
+ stable when the doc changes. Every rule records its source, the files that
373
+ test it, why it was skipped, and its earlier wordings.
374
+ 2. **Plan, without AI.** Each numbered section of the doc becomes one test file
375
+ (`"planner": "ai"` asks the AI instead). Rules another journey already tests
376
+ are not planned again. The plan check (`plan-check.md`) lists every rule that
377
+ gets tests, every skip with its reason, and any **must** rule left out.
378
+ 3. **Code facts, once.** The source the doc points at (the files it cites, the
379
+ components its controls are named after, the pages and hooks that reach
380
+ them) is collected without AI and summarised in one call.
381
+ 4. **Writing.** Each call writes a few files (`filesPerCall`, default 3) and
382
+ gets only their rules, their controls' facts and the testkit's exports; the
383
+ shared rules for writing a test are a cached system prompt. Tests find every
384
+ element with `kit.find(page, kind, name)`, check outcomes rather than markup,
385
+ and stub a state they cannot create.
386
+ 5. **Checks before anything runs.** A file is refused, and written once more
387
+ with the reason, if it sleeps for a fixed time, skips because an element is
388
+ missing, has no test or no assertion, spends more than `maxTestsPerRule`
389
+ tests on one rule, or names a label found neither in the code nor the doc.
390
+ 6. **A changed doc.** Only what changed is touched: a renamed label is replaced
391
+ in its tests, a removed rule's tests are deleted (both without AI); a
392
+ reworded or new rule gets its tests written by one small call.
393
+
394
+ `generate` runs nothing. `testforge run` runs the suite; `testforge review`
395
+ judges failures when you ask for it.
396
+
397
+ Every Claude call reports its real token use and cost; `TESTFORGE_TRACE=1`
398
+ also records each call's prompt and events under `testforge/runs/trace/`.
399
+
400
+ ## Project state
401
+
402
+ Everything TestForge keeps lives in `testforge/`. Commit it, except `runs/`
403
+ (run output and AI logs): a teammate who pulls it starts from your memory, plans
404
+ and code facts instead of paying for them again. A project set up by an older
405
+ version is moved from `test-artifacts/` on the first command.
367
406
 
368
407
  ## Notes
369
408
 
package/dist/cli/index.js CHANGED
@@ -20,6 +20,14 @@
20
20
  */
21
21
  import { VERSION } from '../core/index.js';
22
22
  import { runGenerate } from '../core/generate/run-generate.js';
23
+ import { runReview } from '../core/generate/review-journey.js';
24
+ import { migrateState } from '../core/paths.js';
25
+ import { acceptNames, bindJourneys } from '../core/generate/bind.js';
26
+ import { runClean } from '../core/run/clean.js';
27
+ import { aiUsage, describeUsage } from '../core/generate/ai-usage.js';
28
+ import { envDiff, describeEnvDiff } from '../core/project/env-diff.js';
29
+ import { bundleTickets } from '../core/docs/tickets.js';
30
+ import { writeGithubWorkflow } from '../core/run/ci.js';
23
31
  import { runTests } from '../core/run/run-tests.js';
24
32
  import { runDiscover } from '../core/docs/discover.js';
25
33
  import { runMap } from '../core/project/map.js';
@@ -45,10 +53,18 @@ function flagValue(name) {
45
53
  }
46
54
  /** Source-of-truth options shared by setup and discover. */
47
55
  function sourceOptions() {
56
+ const tickets = flagValue('--tickets');
57
+ let reference = flagValue('--reference');
58
+ if (!reference && tickets) {
59
+ const bundle = bundleTickets(tickets);
60
+ console.log(`Using ${bundle.ids.length} exported ticket(s) from ${tickets} as the reference: ${bundle.ids.slice(0, 8).join(', ')}${bundle.ids.length > 8 ? ', …' : ''}`);
61
+ reference = bundle.path;
62
+ }
48
63
  return {
49
- reference: flagValue('--reference'),
64
+ reference,
50
65
  source: flagValue('--source'),
51
66
  replace: process.argv.includes('--replace'),
67
+ refresh: process.argv.includes('--refresh'),
52
68
  };
53
69
  }
54
70
  /**
@@ -78,12 +94,13 @@ function printUsage() {
78
94
  console.log(' testforge generate --basic Use the basic generator only');
79
95
  console.log(' testforge generate --force Regenerate everything');
80
96
  console.log(' testforge generate --no-wait Stop when the AI hits its usage limit (default: wait for the reset and resume)');
81
- console.log(' testforge generate --no-verify Skip running each journey once after generating (default: run and repair)');
97
+ console.log(' testforge generate --plan-only Plan and check each journey (testforge/memory/<area>/<id>/plan-check.md), write no tests');
82
98
  console.log(' testforge generate --journey <name> Generate just ONE journey');
83
99
  console.log(' testforge generate --area <area> Generate a whole area (e.g. api)');
84
100
  console.log(' testforge discover Draft journey docs (asks: map or reference)');
85
- console.log(' testforge discover --journey <name> Draft just ONE journey doc');
101
+ console.log(' testforge discover --journey <name> Draft just ONE journey doc', ' testforge discover --journey <name> --reference <x> [--refresh] One journey from a reference: the reference is read once and kept (--refresh reads it again)');
86
102
  console.log(' testforge discover --reference <file|url> Base docs on a reference');
103
+ console.log(' testforge discover --tickets <folder> Base docs on exported tickets (one file per ticket); each source names its ticket');
87
104
  console.log(' (a reference only writes the journeys it');
88
105
  console.log(' describes — add --replace to let it');
89
106
  console.log(' redraft the whole project)');
@@ -91,12 +108,21 @@ function printUsage() {
91
108
  console.log(' testforge init Inspect project and write config');
92
109
  console.log(' testforge update Re-check and update only what changed');
93
110
  console.log(' testforge env Write .env.testing.example from docs');
111
+ console.log(' testforge env --diff Only the variables the suite reads that the env file does not set');
112
+ console.log(' testforge ci --github Write a GitHub Actions workflow that runs the suite against a deployed environment');
113
+ console.log(' testforge bind [--journey j] [--accept "doc=code"] Which tests the code is ready for, which wait for the feature, which use another name; --accept takes the code\'s wording (no AI)');
114
+ console.log(' testforge review [--journey j] [--rerun] Judge the last failures again (defect / test fault / environment …), no repair');
115
+ console.log(' testforge clean [--run <id> | --older-than <days>] [--yes] Remove what earlier runs created (dry run without --yes)');
94
116
  console.log(' testforge run [--area a] [--journey j] Run the generated tests (Jest/Playwright)');
95
117
  console.log(' testforge run --env-file <file> Run against another server (e.g. .env.dev / .env.staging)');
96
118
  console.log(' testforge run --headed | --ui | --debug Watch the browser / open the Playwright UI');
97
119
  console.log(' testforge run -- <flags> Pass anything else straight to the runner');
98
120
  }
99
121
  async function main() {
122
+ // A project set up by an older version keeps its state under test-artifacts/: move it once.
123
+ const moved = command && command !== 'help' && command !== '--help' ? migrateState() : [];
124
+ if (moved.length)
125
+ console.log(`Moved TestForge's state into testforge/ (${moved.length} item(s)); nothing was rebuilt.\n`);
100
126
  if (command === 'update') {
101
127
  await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
102
128
  }
@@ -116,6 +142,59 @@ async function main() {
116
142
  console.log(output);
117
143
  console.log('\nProject map written to testforge.md');
118
144
  }
145
+ else if (command === 'ci') {
146
+ if (!process.argv.includes('--github')) {
147
+ console.log('Usage: testforge ci --github [--schedule none|daily|weekdays|weekly|"<cron>"] [--env-name <label>] [--force]');
148
+ }
149
+ else {
150
+ console.log(writeGithubWorkflow({ envName: flagValue('--env-name'), force: process.argv.includes('--force'), schedule: flagValue('--schedule') }));
151
+ }
152
+ }
153
+ else if (command === 'clean') {
154
+ const days = flagValue('--older-than');
155
+ console.log(await runClean({ runId: flagValue('--run'), olderThanDays: days != null ? Number(days) : undefined, yes: process.argv.includes('--yes') }));
156
+ }
157
+ else if (command === 'bind') {
158
+ // After a feature ships: which tests can run, which wait for the code, which use a different name.
159
+ const accept = process.argv.flatMap((a, k) => (process.argv[k - 1] === '--accept' ? [a] : []));
160
+ if (accept.length) {
161
+ const pairs = accept.map((a) => a.split('=')).filter((p) => p.length === 2 && p[0] && p[1]);
162
+ const notes = acceptNames(pairs, { journey: flagValue('--journey') });
163
+ for (const n of notes)
164
+ console.log(` ${n}`);
165
+ console.log(notes.length ? '\nRun `testforge generate` to record the rename in the journey\'s memory (no AI).' : 'Nothing used those names.');
166
+ }
167
+ else {
168
+ for (const r of bindJourneys({ journey: flagValue('--journey') })) {
169
+ const by = (s) => r.files.filter((f) => f.state === s);
170
+ console.log(` ${r.journey}: ${by('built').length} ready, ${by('partial').length} partly built, ${by('not-built').length} not built yet, ${by('renamed').length} with a different name`);
171
+ for (const f of r.files) {
172
+ if (f.state === 'built')
173
+ continue;
174
+ const what = f.state === 'renamed'
175
+ ? f.renamed.map((x) => `"${x.name}" → code has "${x.nearest}"`).join('; ')
176
+ : `not in the code: ${f.missing.slice(0, 4).map((m) => `"${m}"`).join(', ')}${f.missing.length > 4 ? ' …' : ''}`;
177
+ console.log(` ${f.state.padEnd(9)} ${f.file}: ${what}`);
178
+ }
179
+ }
180
+ console.log('\nA different name is not a defect: change the code to the doc, or accept the code\'s wording with --accept "doc wording=code wording".');
181
+ }
182
+ }
183
+ else if (command === 'review') {
184
+ console.log('Reviewing the last failures of each journey (no repair)...\n');
185
+ const summaries = await runReview({ journey: flagValue('--journey'), rerun: process.argv.includes('--rerun') });
186
+ if (!summaries.length)
187
+ console.log('No generated journey matched.');
188
+ for (const r of summaries) {
189
+ if (r.skipped)
190
+ console.log(` ${r.journey}: not reviewed — ${r.skipped}`);
191
+ else if (!r.failures)
192
+ console.log(` ${r.journey}: no failures to review`);
193
+ else
194
+ console.log(` ${r.journey}: ${r.failures} failure(s) → ` + Object.entries(r.verdicts).map(([k, n]) => `${n} ${k}`).join(', '));
195
+ }
196
+ console.log('\nVerdicts written to testforge/runs/defect-review/. The next run publishes them with each failed test.');
197
+ }
119
198
  else if (command === 'discover') {
120
199
  console.log('Drafting journey files with AI...');
121
200
  console.log('This writes drafts into docs/ from your chosen source.\n');
@@ -125,7 +204,10 @@ async function main() {
125
204
  console.log('then run `testforge generate`.');
126
205
  }
127
206
  else if (command === 'env') {
128
- runEnvTemplate('docs', '.env.testing.example', '.', generatedScanDirs());
207
+ if (process.argv.includes('--diff'))
208
+ console.log(describeEnvDiff(envDiff('.')));
209
+ else
210
+ runEnvTemplate('docs', '.env.testing.example', '.', generatedScanDirs());
129
211
  }
130
212
  else if (command === 'run') {
131
213
  // Anything after `--`, plus the runner flags people reach for most, goes
@@ -141,6 +223,7 @@ async function main() {
141
223
  journey: flagValue('--journey'),
142
224
  envFile: flagValue('--env-file'),
143
225
  passThrough,
226
+ all: process.argv.includes('--all'),
144
227
  });
145
228
  process.exitCode = code;
146
229
  }
@@ -155,11 +238,7 @@ async function main() {
155
238
  : process.argv.includes('--wait-for-limit')
156
239
  ? true
157
240
  : undefined, // undefined -> config.waitForLimit (default on)
158
- verify: process.argv.includes('--no-verify')
159
- ? false
160
- : process.argv.includes('--verify')
161
- ? true
162
- : undefined, // undefined -> config.verify (default on)
241
+ planOnly: process.argv.includes('--plan-only'),
163
242
  });
164
243
  if (results.length === 0) {
165
244
  console.log('No journey files found in docs/. Add a .md file and try again.');
@@ -190,9 +269,6 @@ async function main() {
190
269
  for (const w of r.warnings ?? []) {
191
270
  console.log(` ! ${w}`);
192
271
  }
193
- if (r.verified === true)
194
- console.log(' verified: every file ran; every failure left was reviewed and is listed below');
195
- printReviewed(r.reviewed ?? []);
196
272
  }
197
273
  const generated = results.filter((r) => r.status !== 'skipped').length;
198
274
  // Rows with no test dir are notices (a non-journey file, the AI going
@@ -202,6 +278,16 @@ async function main() {
202
278
  const seedFailures = results.filter((r) => r.seedStatus === 'failed').length;
203
279
  console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged)` +
204
280
  (notices ? `, ${notices} notice(s) above.` : '.'));
281
+ const spent = aiUsage();
282
+ if (spent.calls)
283
+ console.log(`AI this run: ${describeUsage(spent)}.`);
284
+ // What the journeys just written still need from the env file.
285
+ const touched = results.filter((r) => r.testDir && r.status !== 'skipped').map((r) => basename(r.testDir));
286
+ if (touched.length) {
287
+ const d = envDiff('.', touched);
288
+ if (d.journeys.some((j) => j.missing.some((v) => !v.hasDefault)))
289
+ console.log('\n' + describeEnvDiff({ ...d, journeys: d.journeys.map((j) => ({ ...j, missing: j.missing.filter((v) => !v.hasDefault) })) }));
290
+ }
205
291
  if (seedFailures > 0) {
206
292
  console.log(`Warning: ${seedFailures} seed file(s) could not be generated — see notes above.`);
207
293
  }
@@ -260,28 +346,6 @@ async function main() {
260
346
  printUsage();
261
347
  }
262
348
  }
263
- /** Reviewed failures, one line per root cause, strongest first. */
264
- function printReviewed(reviewed) {
265
- const sections = [
266
- ['defect', 'DEFECTS — a [spec] requirement the app breaks'],
267
- ['finding', 'FINDINGS — real app behaviour against a [baseline]/[observed] criterion (lower priority)'],
268
- ['pending-decision', 'PENDING DECISIONS — not requirements yet; take them to product, do not report as defects'],
269
- ];
270
- for (const [verdict, heading] of sections) {
271
- const rows = reviewed.filter((r) => r.verdict === verdict);
272
- if (!rows.length)
273
- continue;
274
- console.log(` ${heading}:`);
275
- for (const r of rows) {
276
- const where = r.tests.slice(0, 3).map((t) => `${basename(t.file)}:${t.line}`).join(', ');
277
- const more = r.tests.length > 3 ? ` +${r.tests.length - 3} more` : '';
278
- console.log(` - ${r.claim}`);
279
- console.log(` ${r.tests.length} test(s): ${where}${more}${r.evidence ? ` · app: ${r.evidence}` : ''}`);
280
- if (r.reason)
281
- console.log(` ${r.reason}`);
282
- }
283
- }
284
- }
285
349
  main().catch((err) => {
286
350
  console.error(err instanceof Error ? err.message : String(err));
287
351
  process.exit(1);
@@ -12,7 +12,7 @@
12
12
  import { createHash } from 'node:crypto';
13
13
  import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, statSync, } from 'node:fs';
14
14
  import { dirname, join, extname } from 'node:path';
15
- import { STATE_DIR, LEGACY_STATE_DIR } from '../paths.js';
15
+ import { STATE_DIR, LEGACY_STATE_DIR, OLD_STATE_DIR } from '../paths.js';
16
16
  /** Make a short fingerprint of a file's content. */
17
17
  export function hashContent(content) {
18
18
  return createHash('sha256').update(content).digest('hex');
@@ -66,6 +66,7 @@ const SKIP_DIRS = new Set([
66
66
  'docs',
67
67
  STATE_DIR,
68
68
  LEGACY_STATE_DIR,
69
+ OLD_STATE_DIR,
69
70
  'generated-tests',
70
71
  ]);
71
72
  export function hashProjectSource(rootDir) {
@@ -14,7 +14,6 @@
14
14
  * tools) rather than stuffing all source code into the prompt, which
15
15
  * keeps this simple and works for projects of any size.
16
16
  */
17
- import { execFileSync } from 'node:child_process';
18
17
  import { existsSync, readdirSync, readFileSync, writeFileSync, mkdirSync, renameSync, } from 'node:fs';
19
18
  import { join, relative, sep, basename } from 'node:path';
20
19
  import { notAJourneyDoc } from './parser.js';
@@ -22,8 +21,11 @@ import { loadConfig, splitCommand, withEditTools } from '../project/config.js';
22
21
  import { DEEP_DOC_GUIDANCE } from './doc-guidance.js';
23
22
  import { promptConfirm } from '../prompt.js';
24
23
  import { loadSourceChoice, resolveDocsSource, saveJourneyReferences, saveSourceChoice, } from './source-of-truth.js';
25
- import { analyzeReference, draftDocsFromReference, DIFF_PATH, } from './reference.js';
26
- import { reviewDocs } from './review.js';
24
+ import { analyzeReference, buildJourneyDraftPrompt, draftDocsFromReference, draftJourneyDoc, DIFF_PATH, } from './reference.js';
25
+ import { buildLeanReviewPrompt, enforceSpecSources, reviewDocs } from './review.js';
26
+ import { readReference, sectionFor } from './reference-text.js';
27
+ import { sourceExcerpts } from './excerpts.js';
28
+ import { runAiSync } from '../generate/ai-usage.js';
27
29
  /** Scope line: one journey, or all of them. */
28
30
  function scopeLines(journey) {
29
31
  return journey
@@ -125,7 +127,7 @@ function runCodeDiscover(srcDir, docsDir, journey) {
125
127
  // are tool-specific; for Claude we add acceptEdits. Other tools may
126
128
  // need different setup (see README).
127
129
  const { program, args } = withEditTools(splitCommand(loadConfig().aiCommand));
128
- const output = execFileSync(program, args, {
130
+ const output = runAiSync(program, args, {
129
131
  input: prompt,
130
132
  encoding: 'utf-8',
131
133
  maxBuffer: 10 * 1024 * 1024,
@@ -168,6 +170,12 @@ function snapshotDocs(docsDir) {
168
170
  */
169
171
  export async function runDiscover(srcDir, docsDir, opts = {}, journey) {
170
172
  const source = await resolveDocsSource(opts);
173
+ // One journey from a reference: no comparison of the whole product, the
174
+ // reference read once and cut to this journey, drafted and reviewed in one
175
+ // call each with no tools.
176
+ if (source.mode === 'reference' && source.reference && journey) {
177
+ return draftOneJourney(srcDir, docsDir, source.reference, journey, opts.refresh);
178
+ }
171
179
  if (source.mode === 'reference' && source.reference) {
172
180
  console.log(`\nComparing the reference against testforge.md: ${source.reference}`);
173
181
  console.log('(reading the reference and the map — this may take a moment)\n');
@@ -287,3 +295,42 @@ async function discoverFromCode(srcDir, docsDir, journey) {
287
295
  await reviewDocs(docsDir, changed, srcDir);
288
296
  return out;
289
297
  }
298
+ const docTitle = (text) => /^# Journey:\s*(.+)$/m.exec(text)?.[1]?.trim();
299
+ const kebab = (t) => t.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
300
+ async function draftOneJourney(srcDir, docsDir, reference, journey, refresh = false) {
301
+ const config = loadConfig();
302
+ const text = readReference(reference, refresh);
303
+ const section = sectionFor(text, journey);
304
+ const excerpts = sourceExcerpts(section, [srcDir, ...(config.relatedDirs ?? [])]);
305
+ const docs = [...snapshotDocs(docsDir)].filter(([p, t]) => !notAJourneyDoc(p, t));
306
+ const existing = docs.find(([, t]) => kebab(docTitle(t) ?? '') === kebab(journey))?.[0];
307
+ const others = docs.filter(([p]) => p !== existing).map(([p, t]) => `${docTitle(t) ?? p} (${p})`);
308
+ const areas = Object.keys(config.testOutput ?? {}).length ? Object.keys(config.testOutput ?? {}) : ['web', 'api'];
309
+ const owner = loadSourceChoice()?.journeys?.[existing ?? ''];
310
+ if (existing && owner && owner !== reference) {
311
+ throw new Error(`${existing} comes from another reference (${owner}); not redrafting it from ${reference}.`);
312
+ }
313
+ console.log(`\nDrafting "${journey}" from the reference (${section.length.toLocaleString('en-GB')} characters of it, one call, no tools)...`);
314
+ const draft = draftJourneyDoc(buildJourneyDraftPrompt(journey, section, excerpts, areas, others));
315
+ if (!draft.markdown)
316
+ throw new Error('The AI answered without a journey doc.');
317
+ console.log('Reviewing it for missing scenarios (one call, no tools)...');
318
+ let doc = draft.markdown;
319
+ try {
320
+ const reviewed = draftJourneyDoc(buildLeanReviewPrompt(doc, section, excerpts, others));
321
+ if (reviewed.markdown)
322
+ doc = reviewed.markdown;
323
+ }
324
+ catch (e) {
325
+ console.log(` review failed, kept as drafted (${e.message.split('\n')[0]})`);
326
+ }
327
+ const { markdown, retagged } = enforceSpecSources(doc);
328
+ if (retagged)
329
+ console.log(` ${retagged} [spec] line(s) without a quoted source re-tagged [observed]`);
330
+ const rel = existing ?? join(draft.area && areas.includes(draft.area) ? draft.area : areas[0], `${kebab(docTitle(markdown) ?? journey)}.md`);
331
+ mkdirSync(join(docsDir, rel, '..'), { recursive: true });
332
+ writeFileSync(join(docsDir, rel), markdown, 'utf-8');
333
+ if (!existing)
334
+ saveJourneyReferences([rel], reference);
335
+ return `${existing ? 'Rewrote' : 'Wrote'} ${join(docsDir, rel)} from the reference. Please review it, then run \`testforge generate --journey ${kebab(journey)}\`.`;
336
+ }
@@ -45,30 +45,68 @@ export const DEEP_DOC_GUIDANCE = [
45
45
  `- "[spec]" — the source states this behaviour. A failure is a defect`,
46
46
  ` against the spec.`,
47
47
  `- "[baseline]" — universal robustness that holds for ANY well-built app.`,
48
- ` Add these YOURSELF for every field, control and action the journey`,
49
- ` has, even when the source says nothing about them:`,
48
+ ` The client describes what to build; it rarely says how it must NOT`,
49
+ ` break. That is your job: think like a tester who wants to break it and`,
50
+ ` like a user who is technical enough to find its loopholes. Add these`,
51
+ ` YOURSELF for every screen, form, list, action and endpoint the journey`,
52
+ ` has, even when the source says nothing about them. Write each one's`,
53
+ ` LEVEL and CATEGORY right after the tag, e.g. "[baseline] (must,`,
54
+ ` security)". Every "must" that fits the journey has to be there.`,
55
+ ` MUST — security:`,
56
+ ` * One user never reaches another user's data or actions: changing an id`,
57
+ ` in a URL, a request or a hidden field is refused (not "not found by`,
58
+ ` accident" — refused). A page or endpoint that needs a login is refused`,
59
+ ` without one; a role cannot use another role's actions.`,
60
+ ` * Whatever the client sends is not trusted: a price, an amount, a`,
61
+ ` discount, a role, an owner id or a status changed in the request is`,
62
+ ` refused or recomputed by the server.`,
63
+ ` * An HTML, script or injection payload in any input is stored and shown`,
64
+ ` as text, never executed, never a server error (5xx).`,
65
+ ` * Passwords are masked; passwords, tokens and codes never appear in a`,
66
+ ` URL, a query string or a log the user can see.`,
67
+ ` * Repeated wrong attempts (password, code, coupon, PIN) are limited and`,
68
+ ` the limit is told to the user.`,
69
+ ` MUST — money and data:`,
70
+ ` * Double submission: an action with a side effect (pay, book, order,`,
71
+ ` send, create) done`,
72
+ ` twice — a double click, a repeated Enter, two tabs, a retry after a`,
73
+ ` timeout — leaves exactly one record, one charge, one message.`,
74
+ ` * A flow that fails half-way leaves no half-made record and no money`,
75
+ ` taken without the thing bought; the user sees what happened.`,
76
+ ` * Amounts, totals, discounts and limits are the same on every screen and`,
77
+ ` in the server's answer.`,
78
+ ` MUST — concurrency:`,
79
+ ` * Two users or two tabs acting on the same thing at once (the same slot,`,
80
+ ` the last item, a code with limited uses, the same record edited) —`,
81
+ ` exactly one wins and the other gets a clear reason.`,
82
+ ` * Nothing hangs: every action ends within its timeout with a result or a`,
83
+ ` visible error; a request that never answers never leaves a spinner`,
84
+ ` forever or a control disabled for good.`,
85
+ ` MUST — error boundaries:`,
86
+ ` * Every load and every action, when the server answers 500, the network`,
87
+ ` drops, it times out, or the answer is empty or malformed: a visible`,
88
+ ` message, the page stays usable, no blank page, no crash, the user's`,
89
+ ` input is kept.`,
90
+ ` * One part of a page failing never takes the whole page down.`,
91
+ ` SHOULD — validation:`,
50
92
  ` * Inputs: empty and whitespace-only values; leading/trailing spaces;`,
51
- ` a very long value (5,000 characters); unicode and emoji; an HTML or`,
52
- ` script payload. The app never crashes, never answers with a server`,
53
- ` error (5xx), never executes the payload (it is shown as text), and`,
54
- ` either accepts the value or rejects it with a visible reason.`,
55
- ` * No silent failure: every rejected submit shows a visible reason; a`,
56
- ` server error, network failure or timeout shows a visible message`,
57
- ` and leaves the form usable.`,
58
- ` * Double submission: a double click or a repeated Enter sends at most`,
59
- ` one request with a side effect (one record, one charge, one`,
60
- ` message); the control is disabled or ignored while a request is`,
61
- ` pending.`,
62
- ` * Keyboard: the form can be filled and submitted with the keyboard`,
63
- ` alone; Enter submits.`,
64
- ` * Navigation: refresh, Back, and opening a mid-flow URL directly never`,
65
- ` crash or leave a blank, broken page.`,
66
- ` * Sensitive data: passwords are masked; passwords and tokens never`,
67
- ` appear in a URL or query string.`,
68
- ` * Access: a page or action that needs a login is not usable logged`,
69
- ` out; one user never sees another user's data.`,
70
- ` * Choice controls (dropdown, radio, checkbox, list): every option`,
71
- ` shown can be selected and submitted.`,
93
+ ` a very long value (5,000 characters); unicode and emoji. Each is`,
94
+ ` accepted or rejected with a visible reason, never a crash.`,
95
+ ` * No silent failure: every rejected submit says why, next to the field`,
96
+ ` or the form (a wrong password with no message is a defect).`,
97
+ ` * The screen and the server accept exactly the same values.`,
98
+ ` SHOULD — state and navigation:`,
99
+ ` * Refresh, Back, Forward and opening a mid-flow URL directly never crash`,
100
+ ` or leave a blank page; an expired session sends the user to log in and`,
101
+ ` back; after logging out nothing of the account is shown.`,
102
+ ` * Choice controls (dropdown, radio, checkbox, list): every option shown`,
103
+ ` can be selected and submitted; long lists page or scroll.`,
104
+ ` SHOULD — regression:`,
105
+ ` * Every defect the source or the doc already names keeps a test, so it`,
106
+ ` cannot quietly come back.`,
107
+ ` COULD — accessibility:`,
108
+ ` * The form can be filled and submitted with the keyboard alone; Enter`,
109
+ ` submits; pop-ups close with Escape; controls have readable names.`,
72
110
  ` [baseline] is ROBUSTNESS only. Never present product POLICY you`,
73
111
  ` invented (a maximum length, a password rule, an error wording, a`,
74
112
  ` limit) as [baseline].`,