@activate-core/test-forge 2.0.0 → 2.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -127,9 +127,9 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
127
127
  ` how to establish preconditions — follow it.`,
128
128
  ` - Assert what the line states, at the loosest form that still proves it.`,
129
129
  ` Add no condition it does not state, and no exact copy, ordering, id,`,
130
- ` timestamp or count it does not fix. A [baseline] robustness line ("opens`,
131
- ` without a crash") is proved by the page loading with no error and no`,
132
- ` 5xx, not by a URL, heading or element it never names. "The customer is`,
130
+ ` timestamp or count it does not fix. Only a line that says exactly "opens`,
131
+ ` without a crash" is proved by the page loading with no error and no 5xx;`,
132
+ ` every other line is proved by its own outcome, never by "no 5xx". "The customer is`,
133
133
  ` told why" is proved by a non-empty message VISIBLE where the app shows`,
134
134
  ` it, unless the line fixes the words. Check where the page actually`,
135
135
  ` renders a value before asserting it there.`,
@@ -252,10 +252,25 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
252
252
  ``,
253
253
  `A LATER STEP OF A FLOW IS OPENED THROUGH THE TESTKIT. Never page.goto() a`,
254
254
  `later step of a multi-step flow by its URL alone: call`,
255
- `kit.openStep(page, step, input) and skip with its reason when it does not`,
256
- `resolve true. Only a test that checks the guard itself (what a visitor who`,
255
+ `kit.openStep(page, step, input) with a step name the TESTKIT lists for it`,
256
+ `(no other name exists), and skip with its reason when it does not resolve`,
257
+ `true. A screen after the last listed step is reached from that step through`,
258
+ `the UI. Only a test that checks the guard itself (what a visitor who`,
257
259
  `skipped the earlier steps sees) opens the bare URL.`,
258
260
  ``,
261
+ `kit.find RETURNS A PROMISE. Always \`const pay = await kit.find(page, 'button',`,
262
+ `'Pay')\` first, then use pay. Never chain on kit.find(...) (.first(), .click()),`,
263
+ `never keep it in a variable without await, and never pass it to expect()`,
264
+ `without await.`,
265
+ ``,
266
+ `kit.login PREPARES ONE PAGE. It sets the session for the page it is given and`,
267
+ `restores it when that page loads again. So:`,
268
+ ` - log out through the app's own Log Out control, never by clearing storage or`,
269
+ ` cookies (the restored session comes straight back);`,
270
+ ` - a second tab or page is not logged in by it: call kit.login on that page`,
271
+ ` too. "Logged in in another tab" — one tab's login reaching another — is not`,
272
+ ` something the testkit can set up: such a rule gets a cannot-test line.`,
273
+ ``,
259
274
  `DATA A TEST CREATES. Other runs use the same accounts, so what they left`,
260
275
  `behind is already there when this test starts.`,
261
276
  ` - A test that creates a record (a booking, an order, a vehicle, an`,
@@ -268,6 +283,10 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
268
283
  ` or a list being empty. "Exactly one booking for this request" is checked`,
269
284
  ` among the records that carry this test's unique value.`,
270
285
  ` - A test that only reads data uses the fixture role and the seeded ids.`,
286
+ ` - Two DIFFERENT people at once (two customers paying for one slot): log one`,
287
+ ` in as the fixture role and one as "run", and skip at the top unless`,
288
+ ` kit.distinctRunAccount() is true (the environment has no second account).`,
289
+ ` A TESTKIT without distinctRunAccount gives that rule a cannot-test line.`,
271
290
  ``,
272
291
  `AT MOST ${maxPerRule()} TESTS FOR ONE CRITERION. Two tests whose oracle quotes the same`,
273
292
  `criterion must reach different branches; more inputs for that criterion go`,
@@ -294,11 +313,36 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
294
313
  ` declarations. A precondition the environment lacks is a run-time skip`,
295
314
  ` inside a real test, stating what was observed.`,
296
315
  ``,
316
+ `EVERY RULE GETS A TEST THAT CAN FAIL. Each rule id this file is given gets`,
317
+ `at least one test, and that test fails when the product breaks the rule.`,
318
+ ` - A rule whose STATE this area cannot set up — data that lives in another`,
319
+ ` system, a backend-only effect the screen never shows, an external service`,
320
+ ` it cannot drive — gets ONE line instead of a test:`,
321
+ ` // cannot-test: R7 — <the state this area cannot set up>`,
322
+ ` An element, locator or route you were not shown is NEVER that reason:`,
323
+ ` find the element with kit.find(page, kind, <the name the doc gives it>),`,
324
+ ` which needs nothing from the CODE FACTS. Two different people at once`,
325
+ ` is not that reason either: see kit.distinctRunAccount() below.`,
326
+ ` Never leave a rule out silently, and never write a test that passes`,
327
+ ` whatever the product does in its place.`,
328
+ ` - Assert the ONE outcome the rule expects. Never a regex or a condition`,
329
+ ` that accepts the good and the bad outcome alike (/confirm|not available/),`,
330
+ ` never the page's whole text (body or main innerText) compared with`,
331
+ ` itself, and never an expect() only inside an if, else or catch.`,
332
+ ` - A rule about a specific thing (the operator, the total, the time, the`,
333
+ ` link's address) reads THAT element and compares its value: "the same`,
334
+ ` operator" means the operator's name read before and after.`,
335
+ ` - "No 5xx", "the page loads", "body is not empty", "a £ sign is shown",`,
336
+ ` "the count of a text that never appears is 0" prove only a rule that says`,
337
+ ` exactly that.`,
338
+ ` - A stub's own words are never the assertion: assert what the app does`,
339
+ ` with the stubbed state.`,
340
+ ``,
297
341
  `SELF-CHECK BEFORE YOU OUTPUT. Go through every test you wrote and name, for`,
298
- `each line, its evidence under E1–E5, and the distinct outcome each test`,
299
- `adds. A line with no evidence is fixed, or its test is removed; a test that`,
300
- `adds no new outcome is removed. When in doubt, report LESS: a missing test`,
301
- `costs one line of coverage; a false failure costs the team a release.`,
342
+ `each line, its evidence under E1–E5, and the outcome that makes it fail. A`,
343
+ `line with no evidence is fixed; a test that cannot fail is rewritten, or`,
344
+ `its rule gets a cannot-test line. Then check every rule id you were given`,
345
+ `has a test or a cannot-test line.`,
302
346
  ``,
303
347
  ...(memory
304
348
  ? [
@@ -307,9 +351,9 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
307
351
  `once for the whole journey, the TESTKIT list is the harness's real`,
308
352
  `exports, and the JOURNEY block holds exactly the rules this file tests`,
309
353
  `with their context. Those are the MECHANICS: HOW to reach and drive`,
310
- `the behaviour. Never invent a selector, route, field or helper that`,
311
- `appears nowhere below; a case you cannot reach with what is here is`,
312
- `left out. But the EXPECTED OUTCOME you`,
354
+ `the behaviour. Never invent a test id, route, request field or helper`,
355
+ `that appears nowhere below. An element is found by the name the doc`,
356
+ `gives it, with kit.find. But the EXPECTED OUTCOME you`,
313
357
  ]
314
358
  : grounding
315
359
  ? [
@@ -411,7 +455,11 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
411
455
  `FIND EVERY ELEMENT WITH kit.find(page, kind, name): kind is what the doc`,
412
456
  `says it is (button, link, input, dropdown, checkbox, radio, toggle, tab,`,
413
457
  `menuitem, dialog, heading, alert, toast, image, list, table …), name is`,
414
- `the text the doc gives it. It tries the proper role first and falls back`,
458
+ `the text the SCREEN shows for it — read it from the CODE FACTS (the`,
459
+ `component's text, label or aria-label); the doc's word only when the facts`,
460
+ `do not show that element ("help link" in the doc may read "FAQs" on the`,
461
+ `screen). Written out, never a value the test made up (kit.unique, a stub's`,
462
+ `id). It tries the proper role first and falls back`,
415
463
  `to the label, placeholder, aria-label, alt text and visible text, adding`,
416
464
  `a finding when the element is not built as what it is — so a test written`,
417
465
  `before the feature still finds it once it ships. Use page.getBy* only for`,
@@ -430,8 +478,8 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
430
478
  `ground on role/label/text/name instead of inventing a test-id.`,
431
479
  `NEVER invent a data-testid/data-cy: an unverified test-id is`,
432
480
  `REJECTED by the selector-grounding gate and wastes the whole file.`,
433
- `Only when an element cannot be identified by ANY real handle, leave`,
434
- `that single case out and name the control in a comment — never guess.`,
481
+ `An element the doc names is found with kit.find(page, kind, <that name>)`,
482
+ `even when the source shows no handle for it; never guess a test id.`,
435
483
  `NAVIGATION: page.goto(...) takes ONLY a concrete URL or path —`,
436
484
  `never a description/comment, and never a route-file placeholder`,
437
485
  `like "/s/[shortCode]" or "/user/:id". Fill the path with a real`,
@@ -546,12 +594,12 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
546
594
  ` ${focus.title}`,
547
595
  ` ${focus.covers}`,
548
596
  `Do NOT test the journey's other concerns here — separate files`,
549
- `cover those. Go DEEP on this one concern (happy path + its own`,
550
- `edge/negative/security cases) rather than broad-but-shallow.`,
597
+ `cover those. ${memory?.rules.length ? 'Test each rule below; add no case the rules do not state.' : 'Go DEEP on this one concern (happy path + its own edge/negative/security cases) rather than broad-but-shallow.'}`,
551
598
  `Use this exact describe() title: "${focus.title}".`,
552
599
  ...(memory?.rules.length
553
600
  ? [
554
- `Test exactly these rules: ${memory.rules.join(', ')}. Name the rule id AND its`,
601
+ `Test exactly these rules: ${memory.rules.join(', ')}. EVERY one gets a test or a`,
602
+ `"// cannot-test: <id> — <why>" line. Name the rule id AND its`,
555
603
  `tag in each test's oracle comment, e.g. "// oracle: ${memory.rules[0]} [spec] …":`,
556
604
  `the tag is how a failure is later graded as a defect or a finding.`,
557
605
  ]
@@ -575,11 +623,15 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
575
623
  `- calls page.waitForTimeout() or sleeps for a fixed time;`,
576
624
  `- skips a test because an element is missing (test.skip(true, …), or a`,
577
625
  ` condition that counts or looks for an element) — that is a failure to assert;`,
578
- `- has a test with no expect(), or more than ${maxPerRule()} tests for one rule;`,
579
- `- has no test at all: every rule gets one; stub a state you cannot create;`,
626
+ `- has a test with no expect(), an expect() only inside an if/else/catch, or more`,
627
+ ` than ${maxPerRule()} tests for one rule;`,
628
+ `- leaves a rule with neither a test nor a "// cannot-test: <id> — <why>" line;`,
629
+ `- asserts a regex that accepts the good and the bad outcome, compares the page's`,
630
+ ` whole text, or asserts the words its own stub returns;`,
631
+ `- uses kit.find(...) without await, or a kit.openStep step the TESTKIT does not list;`,
580
632
  `- names a button, link, field or label that is in neither the CODE FACTS nor the rules;`,
581
633
  `- defines its own helper functions (use the TESTKIT) or comments anything but`,
582
- ` the one "// oracle: <rule id> [tag] …" line above each test.`,
634
+ ` the one "// oracle: <rule id> [tag] …" line above each test and cannot-test lines.`,
583
635
  `Keep each test short: the steps, then the expect()s. No logging.`,
584
636
  ``,
585
637
  ]
@@ -14,7 +14,7 @@ import { findMarkdownFiles } from '../find-files.js';
14
14
  import { notAJourneyDoc, parseJourney } from '../docs/parser.js';
15
15
  import { loadConfig, testDirForType } from '../project/config.js';
16
16
  import { matchesJourney } from './plan-cache.js';
17
- import { sourceText } from '../docs/excerpts.js';
17
+ import { sourceRoot, sourceText } from '../docs/excerpts.js';
18
18
  import { checkReadiness, shownTexts, testFilesIn } from '../run/readiness.js';
19
19
  function journeys(docsDir, only) {
20
20
  const config = loadConfig();
@@ -52,7 +52,7 @@ function rawSource(roots) {
52
52
  return parts.join('\n');
53
53
  }
54
54
  export function sourceRoots() {
55
- return ['src', ...(loadConfig().relatedDirs ?? [])];
55
+ return [sourceRoot(), ...(loadConfig().relatedDirs ?? [])];
56
56
  }
57
57
  export function bindJourneys(opts = {}) {
58
58
  const roots = sourceRoots();
@@ -0,0 +1,77 @@
1
+ /**
2
+ * testforge — finish a written file: every rule tested, every test able to fail.
3
+ *
4
+ * A file can pass every file-level gate and still leave rules out, or hold a
5
+ * test that passes whatever the app does. Those tests are taken out, and the
6
+ * rules left without a test are written again in ONE small call that sees only
7
+ * those rules, the file's top and why the earlier tests were refused. What
8
+ * that call still gets wrong is removed and reported, never sent again: a
9
+ * missing test is honest, a test that cannot fail is not.
10
+ */
11
+ import { factsSlice, journeySlice } from './journey-memory.js';
12
+ import { appendTests, findTestBlocks, spliceBlocks } from './test-blocks.js';
13
+ import { acceptedReason, describeProblems, ruleCoverage, testProblems } from '../gates/test-quality.js';
14
+ const review = (content, rules, steps, knownTexts) => ({
15
+ cov: ruleCoverage(content, rules),
16
+ problems: testProblems(content, { steps, knownTexts }),
17
+ });
18
+ /** The file without the tests that have a problem. */
19
+ function withoutBlocks(content, problems) {
20
+ const blocks = [...new Map(problems.map((p) => [p.block.start, p.block])).values()];
21
+ return blocks.length ? spliceBlocks(content, blocks.map((block) => ({ block, text: '' }))) : content;
22
+ }
23
+ /** Is the file complete as it is? */
24
+ export function fileIsComplete(content, rules, steps, knownTexts) {
25
+ const { cov, problems } = review(content, rules, steps, knownTexts);
26
+ return !cov.missing.length && !problems.length;
27
+ }
28
+ export async function completeFile(content, ctx) {
29
+ const notes = [];
30
+ const first = review(content, ctx.rules, ctx.steps, ctx.knownTexts);
31
+ let calls = 0;
32
+ let out = content;
33
+ const dropped = new Set();
34
+ if (first.cov.missing.length || first.problems.length) {
35
+ out = withoutBlocks(content, first.problems);
36
+ for (const p of first.problems)
37
+ dropped.add(p.title);
38
+ const left = ruleCoverage(out, ctx.rules).missing;
39
+ if (left.length) {
40
+ const blocks = findTestBlocks(out);
41
+ const head = out.slice(0, blocks[0] ? blocks[0].start : Math.min(out.length, 4000)).slice(0, 4000);
42
+ const title = /test\.describe(?:\.\w+)?\(\s*(['"`])((?:(?!\1).)*)\1/.exec(out)?.[2] ?? ctx.journey.name;
43
+ const slice = journeySlice(ctx.memory, left);
44
+ calls = 1;
45
+ try {
46
+ const tests = await ctx.write(ctx.journey, ctx.path, head, title, '', ctx.harnessPath, ctx.transport, factsSlice(ctx.facts, slice.controls), { journey: slice.text, helpers: ctx.helpers, rules: left }, describeProblems(first.problems, first.cov));
47
+ const candidate = appendTests(out, tests);
48
+ const refused = ctx.check(candidate);
49
+ if (refused)
50
+ notes.push(`tests written again for ${left.join(', ')} were refused (${refused.split('\n')[0]})`);
51
+ else
52
+ out = candidate;
53
+ }
54
+ catch (e) {
55
+ notes.push(`the call to write ${left.join(', ')} again failed — ${e.message.split('\n')[0].slice(0, 120)}`);
56
+ }
57
+ // What the second answer still gets wrong is removed, not sent again.
58
+ const second = testProblems(out, { steps: ctx.steps, knownTexts: ctx.knownTexts });
59
+ if (second.length) {
60
+ out = withoutBlocks(out, second);
61
+ for (const p of second)
62
+ dropped.add(p.title);
63
+ }
64
+ }
65
+ }
66
+ // A cannot-test line that is no longer true (its rule has a test now) or never
67
+ // was a reason ("the locator is unknown") does not stay in the file.
68
+ const has = new Set(findTestBlocks(out).flatMap((b) => b.rules));
69
+ out = out.replace(/^[ \t]*\/\/\s*cannot-test:\s*(R\d+(?:\s*,\s*R\d+)*)\s*[—–-]+\s*(.+)\n/gm, (line, ids, why) => ids.split(/\s*,\s*/).every((id) => has.has(id)) || !acceptedReason(why) ? '' : line);
70
+ const cov = ruleCoverage(out, ctx.rules);
71
+ const tested = [...has].filter((id) => ctx.rules.includes(id));
72
+ if (dropped.size)
73
+ notes.push(`took out ${dropped.size} test(s) that could not check their rule`);
74
+ if (cov.missing.length)
75
+ notes.push(`no test for ${cov.missing.join(', ')}`);
76
+ return { content: out, tested, cannotTest: cov.cannotTest, missing: cov.missing, dropped: [...dropped], calls, notes };
77
+ }
@@ -20,7 +20,7 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
20
20
  import { dirname, relative } from 'node:path';
21
21
  import { loadConfig, splitCommand, withNoTools, withReferenceTools } from '../project/config.js';
22
22
  import { readRules, ruleIndex } from './journey-memory.js';
23
- import { sourceExcerpts } from '../docs/excerpts.js';
23
+ import { sourceExcerpts, sourceRoot } from '../docs/excerpts.js';
24
24
  import { runWithStdin, DEFECT_MARK, FINDING_MARK } from './ai-generator.js';
25
25
  import { aiUnavailableReason, noteAiFailure, noteAiSuccess, waitForAi } from './ai-session.js';
26
26
  const IMPACTS = ['blocker', 'major', 'minor'];
@@ -349,7 +349,7 @@ checkpoint) {
349
349
  const noTools = !decisions.link;
350
350
  const tools = decisions.link ? withReferenceTools : withNoTools;
351
351
  const { program, args } = tools(splitCommand(config.aiCommand));
352
- const roots = ['src', ...(config.relatedDirs ?? [])];
352
+ const roots = [sourceRoot(), ...(config.relatedDirs ?? [])];
353
353
  const ask = io?.ask ?? ((prompt) => runWithStdin(program, args, prompt, config.aiTimeoutMs));
354
354
  const wait = io?.wait ?? (() => ((config.waitForLimit ?? true) ? waitForAi(config) : Promise.resolve(false)));
355
355
  const out = [];
@@ -49,7 +49,7 @@ function similarity(a, b) {
49
49
  * strings is a rename; a close match is a change; the rest is added or
50
50
  * removed. Pure: no AI, no I/O.
51
51
  */
52
- export function alignRules(prev, next) {
52
+ export function alignRules(prev, next, retired = []) {
53
53
  const diff = { same: [], changed: [], renamed: [], added: [], removed: [] };
54
54
  const free = new Set(prev.map((r) => r.id));
55
55
  const byText = new Map();
@@ -117,7 +117,8 @@ export function alignRules(prev, next) {
117
117
  diff.changed.push(best.id);
118
118
  }
119
119
  // 4. Everything else is new, numbered after the highest id ever used.
120
- let top = Math.max(0, ...prev.map((r) => Number(r.id.slice(1)) || 0));
120
+ // A removed rule's id is never given to a new one: old results still name it.
121
+ let top = Math.max(0, ...prev.map((r) => Number(r.id.slice(1)) || 0), ...retired.map((id) => Number(id.slice(1)) || 0));
121
122
  for (const r of next.rules) {
122
123
  if (id.has(r))
123
124
  continue;
@@ -43,9 +43,13 @@ export const HARNESS_CONTRACT = [
43
43
  ` - created(kind, id, cleanup?, role?): void record what the run made (H8).`,
44
44
  ` - the "run" role: token('run'), ensureToken('run'), login(page, 'run'),`,
45
45
  ` apiContext(request, 'run') — the account tests that CREATE data use (H8).`,
46
+ ` - distinctRunAccount(): boolean true only when the "run" role is a`,
47
+ ` different account from the fixture role (H8), so a test that needs two`,
48
+ ` different people at once can tell whether it has them.`,
46
49
  `Plus, for Playwright suites:`,
47
50
  ` - openStep(page, step, input?): Promise<true | string> open a later step`,
48
51
  ` of a multi-step flow with what the earlier steps would have left (H9).`,
52
+ ` - STEPS: object the step names openStep accepts (H9).`,
49
53
  ` - settledClick, fillFrameInputs, watchDialogs: re-exported unchanged from`,
50
54
  ` require('@activate-core/test-forge/reporters/ui-helpers.cjs') — never`,
51
55
  ` rewritten here.`,
@@ -170,7 +174,9 @@ export const HARNESS_CONTRACT = [
170
174
  ` signup — create a fresh account through the product's own sign-up, only`,
171
175
  ` when this environment can complete it (no code it cannot read and`,
172
176
  ` cannot bypass under H7); otherwise fall back to fixture and say so once.`,
173
- ` Mint and cache its token like any role.`,
177
+ ` Mint and cache its token like any role. distinctRunAccount() returns true`,
178
+ ` only in pool or signup mode, when the run account really differs from`,
179
+ ` the fixture one.`,
174
180
  `H9. STEPS OF A FLOW. A later step of a multi-step flow (onboarding, a`,
175
181
  ` booking or checkout wizard, a sign-up in stages) depends on what the`,
176
182
  ` earlier steps stored — a choice, an address, a cart, a plan — in URL`,
@@ -185,8 +191,14 @@ export const HARNESS_CONTRACT = [
185
191
  ` - It waits until the step is really shown and resolves true; when the`,
186
192
  ` product still sends the visitor elsewhere it resolves a string naming`,
187
193
  ` the step it landed on, so the test can skip with that reason.`,
188
- ` - Offer only the steps the flow has; when the project has no such flow,`,
189
- ` export openStep as a function resolving 'this product has no multi-step flow'.`,
194
+ ` - Offer EVERY step the flow has, up to and including its last page before`,
195
+ ` the result — the summary, review or checkout page where the visitor`,
196
+ ` confirms and pays — reaching a page that follows a choice (an operator,`,
197
+ ` a plan, a slot) by making that choice through the UI, as a visitor would.`,
198
+ ` Keep the names in one exported object STEPS (name → what identifies the`,
199
+ ` page), and look the step up in it: an unknown name resolves`,
200
+ ` 'unknown step "<name>" (steps: …)'. When the project has no such flow,`,
201
+ ` export STEPS = {} and openStep resolving 'this product has no multi-step flow'.`,
190
202
  ].join('\n');
191
203
  /**
192
204
  * Stamp written into every generated testkit: the contract it was built
@@ -9,6 +9,7 @@
9
9
  * repair was most of 9.6M tokens; the code below needs no AI at all.
10
10
  */
11
11
  import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
12
+ import { harnessSteps } from '../gates/test-quality.js';
12
13
  import { dirname, join } from 'node:path';
13
14
  import { journeyDir } from '../paths.js';
14
15
  /** Sections that are indexes or open questions, never something to test. */
@@ -73,13 +74,28 @@ export function readRules(markdown) {
73
74
  continue;
74
75
  }
75
76
  if (mode === 'controls') {
76
- const row = /^\|\s*(C\d+[a-z]?)\s*\|(.*)\|\s*$/.exec(line);
77
+ // "| C3 |" or a plain "| 3 |": both are control C3.
78
+ const row = /^\|\s*C?(\d+[a-z]?)\s*\|(.*)\|\s*$/.exec(line);
77
79
  if (row)
78
- controls[row[1]] = row[2].split('|').map((c) => c.trim()).filter(Boolean).join(' · ');
80
+ controls[`C${row[1]}`] = row[2].split('|').map((c) => c.trim()).filter(Boolean).join(' · ');
79
81
  continue;
80
82
  }
81
83
  if (mode !== 'rules')
82
84
  continue;
85
+ // A numbered rule ("1.4 [spec] …") owns everything below it — its steps,
86
+ // its notes, its "(source: …)" line — up to the next rule or heading.
87
+ const numbered = /^\s*\d+(?:\.\d+)*\.?\s+(\[(?:spec|baseline|observed)\][\s\S]*)$/i.exec(line);
88
+ if (numbered) {
89
+ close();
90
+ const tag = (TAG.exec(numbered[1])?.[1] ?? '');
91
+ current = { rule: { id: `R${rules.length + 1}`, section, group, tag, text: numbered[1], controls: [] }, indent: Number.MAX_SAFE_INTEGER, owns: true };
92
+ rules.push(current.rule);
93
+ continue;
94
+ }
95
+ if (current?.owns && !TAG.test(line)) {
96
+ current.rule.text += `\n${line}`;
97
+ continue;
98
+ }
83
99
  const bullet = /^(\s*)[-*]\s+(.*)$/.exec(line);
84
100
  if (bullet) {
85
101
  const indent = bullet[1].length;
@@ -147,6 +163,13 @@ function sliceText(memory, own, context, controlIds) {
147
163
  * the file, not asked of the AI, so every writer gets the same true list
148
164
  * without opening the testkit itself.
149
165
  */
166
+ /** TestForge's own runtime helpers, as a testkit re-exports them (find: uiHelpers.find). */
167
+ const RUNTIME_HELPERS = {
168
+ find: "async find(page, kind, name, { exact, timeout, within }) → Promise<Locator> — ALWAYS await it before use: `const b = await kit.find(page, 'button', 'Pay')`; never kit.find(...).first() and never expect(kit.find(...)). kind: button, link, input, dropdown, checkbox, radio, toggle, tab, menuitem, dialog, heading, alert, toast, image …; name: the text the doc gives it (or a RegExp).",
169
+ settledClick: 'async settledClick(locator, { settleMs, tries }) — clicks once the element has stopped moving (pop-ups, growing forms).',
170
+ fillFrameInputs: 'async fillFrameInputs(page, framePrefix, { name: value }) — fills inputs inside an embedded iframe (card fields).',
171
+ watchDialogs: 'watchDialogs(page) → string[] — accepts browser alerts and records their messages.',
172
+ };
150
173
  export function helperApi(harnessSource) {
151
174
  const names = new Set();
152
175
  const cjs = /module\.exports\s*=\s*\{([\s\S]*?)\}/.exec(harnessSource);
@@ -166,10 +189,16 @@ export function helperApi(harnessSource) {
166
189
  }
167
190
  const lines = harnessSource.split('\n');
168
191
  const out = [];
192
+ const steps = harnessSteps(harnessSource);
169
193
  for (const name of names) {
170
194
  const at = lines.findIndex((l) => new RegExp(`^\\s*(?:export\\s+)?(?:async\\s+)?(?:function\\*?\\s+${name}\\s*\\(|(?:const|let|var)\\s+${name}\\s*=)`).test(l));
171
195
  if (at === -1) {
172
- out.push(`- ${name}`);
196
+ // Re-exported from TestForge's runtime (find: uiHelpers.find): its contract, not just its name.
197
+ out.push(RUNTIME_HELPERS[name] ? `- ${RUNTIME_HELPERS[name]}` : `- ${name}`);
198
+ continue;
199
+ }
200
+ if (name === 'openStep' && steps.length) {
201
+ out.push(`- ${lines[at].trim().replace(/^export\s+/, '').replace(/\s*\{\s*\}?\s*$/, '')} — step is EXACTLY one of: ${steps.join(', ')}. Any other name does not exist. Resolves true when the step is open, else a string saying why.`);
173
202
  continue;
174
203
  }
175
204
  let sig = lines[at].trim().replace(/^export\s+/, '').replace(/\s*\{\s*\}?\s*$/, '');
@@ -278,6 +307,60 @@ export function saveMemory(journey, docHash, memory, helpers, extra) {
278
307
  }
279
308
  return path;
280
309
  }
310
+ /**
311
+ * Put what the files really test into the journey's memory and its plan
312
+ * check: which file tests each rule, which rules cannot be tested here and
313
+ * why, and which have no test. Returns the one-line summary.
314
+ */
315
+ export function recordWritten(journey, files) {
316
+ const dir = journeyDir(journey);
317
+ const path = join(dir, 'rules.json');
318
+ const testedBy = new Map();
319
+ const why = new Map();
320
+ const missing = new Set();
321
+ for (const f of files) {
322
+ for (const id of f.tested)
323
+ testedBy.set(id, [...(testedBy.get(id) ?? []), f.slug]);
324
+ for (const c of f.cannotTest)
325
+ why.set(c.id, `cannot be tested here: ${c.why}`);
326
+ for (const id of f.missing)
327
+ missing.add(id);
328
+ }
329
+ const planned = new Set(files.flatMap((f) => [...f.tested, ...f.cannotTest.map((c) => c.id), ...f.missing]));
330
+ try {
331
+ const record = JSON.parse(readFileSync(path, 'utf-8'));
332
+ for (const r of record.rules) {
333
+ if (!planned.has(r.id))
334
+ continue;
335
+ r.testedBy = testedBy.get(r.id) ?? [];
336
+ r.skipped = why.get(r.id) ?? (missing.has(r.id) ? 'no test was written for it' : undefined);
337
+ }
338
+ writeFileSync(path, JSON.stringify(record, null, 2), 'utf-8');
339
+ }
340
+ catch {
341
+ // no memory file: the summary below still reports it
342
+ }
343
+ const dropped = files.reduce((n, f) => n + f.dropped.length, 0);
344
+ const lines = [
345
+ `${testedBy.size}/${planned.size} planned rules have a test`,
346
+ why.size ? `${why.size} cannot be tested here (${[...why.keys()].join(', ')})` : '',
347
+ missing.size ? `NO TEST for ${[...missing].join(', ')}` : '',
348
+ dropped ? `${dropped} test(s) taken out because they could not check their rule` : '',
349
+ ].filter(Boolean);
350
+ const check = join(dir, 'plan-check.md');
351
+ try {
352
+ const before = existsSync(check) ? readFileSync(check, 'utf-8').replace(/\n## Written\n[\s\S]*$/, '') : '';
353
+ const detail = [
354
+ ...[...why].map(([id, w]) => `- ${id}: ${w}`),
355
+ ...[...missing].map((id) => `- ${id}: no test was written for it`),
356
+ ];
357
+ writeFileSync(check, `${before.trimEnd()}\n\n## Written\n\n${lines.join('; ')}.\n${detail.length ? `\n${detail.join('\n')}\n` : ''}`, 'utf-8');
358
+ }
359
+ catch {
360
+ // best-effort
361
+ }
362
+ return lines.join('; ');
363
+ }
281
364
  /**
282
365
  * Rules of this journey that another journey already tests, found without the
283
366
  * AI: the same tag and nearly the same words (≥ 85 % of the words shared).