@activate-core/test-forge 2.7.1 → 2.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,31 @@
1
1
  # Changelog
2
2
 
3
+ ## 2.9.0
4
+
5
+ 2.8.0 was not published to npm; its changes (below) ship in 2.9.0.
6
+
7
+ - A Jest testkit's `call()` is TestForge's own code (`reporters/http.cjs`, `makeCall`), no longer written by the AI in each project: an `Authorization` header a test passes (an old token, or `''` for none) is sent exactly as given, `role: null` sends none, a refused token is minted again once, and a 429 is cleared through `resetRateLimits()` or waited out for its Retry-After (2 s when none), up to 60 s. A testkit that does not use it is refused at generate time. Tests of "no login" and "after sign-out" sent a signed-in request before.
8
+ - Rows a scenario inserts get the state the product derives from them (observers, model events and the jobs they queue, e.g. a busy-time calendar from working hours and orders), computed the way that code does and undone with the row. `DERIVED_STATE` lists what is rebuilt, keyed by the real table name; `setup()` checks those tables exist (`verifyDerivedState`) and stops the run on a wrong name. Without it every hours or availability check saw every operator as free.
9
+ - Doc first, code later: a journey written before its code gets its actions all the same (full signature and named results from the doc, a body that waits). Its tests call them and assert on the named results. When the code comes, `testforge update` writes only the waiting actions from the code; the tests are not touched and stop waiting by themselves.
10
+ - `test.each(table)('title', fn)` is a test of its rule. An earlier version did not see it, read the rule as untested and appended the test again on every generate; the same test under one title is now kept once (no AI).
11
+ - A kept file whose tests were taken out to be written again gets back the earlier test of every rule whose new test did not come; a file written again whole is taken only if it still tests every rule the earlier one did.
12
+ - A blank name used inside a template literal (`` `${URL}?${PARAM}=…` ``) counts as a use: a test, or a helper outside the tests, that builds a request that way now waits instead of sending a placeholder.
13
+ - Jest areas get an actions module (`helpers/actions.js`): one function per step a person takes through the product, written from the handler and the product's own client (`relatedDirs`), so tests no longer guess a request's shape (path parameters, nesting, JSON string fields, multi-call payments). An action the product lacks makes its tests wait; a refused action is filed as a failed step (not a failed rule), recorded in `testforge/actions-failed.json`, and built again on the next `generate`. `generate --rewrite-waiting` writes the waiting tests again with the actions once the module is new or rebuilt.
14
+ - Clearing the API's rate limiter removes no data, so it needs only a non-production target, not `TESTFORGE_ALLOW_RESET`. A row a test inserts gets the values the product checks before it will use it (a price above the minimum, an enabled flag).
15
+
16
+ ### Upgrading
17
+ - Existing testkits are rebuilt once on the next `generate` (contract changed), and each Jest journey's actions module is built once (one call per journey doc).
18
+
19
+ ## 2.8.0
20
+
21
+ - Every command that calls the AI first checks, with no AI call, that the configured AI CLI is on PATH and runs; a missing or signed-out CLI gets a plain message instead of a process error.
22
+ - A journey doc the AI returns inside a ```markdown fence is written without the closing fence and whatever was said after it.
23
+ - UI tests never select by markup: CSS classes, tag paths, positions and XPath are refused in new tests (an id or name attribute the code facts show stays the last resort for an input with no label).
24
+
25
+ - `testforge update` never redrafts the journey docs because the code changed: it refreshes the project map and the names the tests wait on (`interface.json`) only. A changed reference is reported, not acted on. The docs are redrafted only with `update --refresh-docs` or `update --reference <x>`.
26
+ - Docs TestForge writes (discover, check) are recorded in `testforge/doc-origin.json`. When a doc change came from TestForge's drafting rather than a person, generate never renames, removes or rewrites a test that passed (`testforge/passed.json`); a person's edit to the doc, or `--force`, still can.
27
+ - The reporters file a refused connection (`ECONNREFUSED`) and a 429 from the testkit as the environment, not a product failure.
28
+
3
29
  ## 2.7.1
4
30
 
5
31
  - New code never makes generate write a test again. With the doc unchanged and every rule tested, `generate --refresh-code` reads the code facts again and fills the names the tests wait on from them (two small calls); the test files are not rewritten. `--rewrite-waiting` writes the waiting tests again when asked by name.
package/README.md CHANGED
@@ -205,6 +205,27 @@ otherwise — and checks stored state only through the testkit; `Type: web`
205
205
  becomes a Playwright test. Other areas use the framework set in config. It also
206
206
  generates a seed file that provisions the test data each test needs.
207
207
 
208
+ In a Jest area every step a person takes through the product (book, pay, accept
209
+ an offer…) is one function of `helpers/actions.js`, next to the testkit. It is
210
+ written once per journey doc by an agent that reads the handler and the
211
+ product's own client (set `relatedDirs` in `testforge.config.json` to the web or
212
+ mobile repo), and tests call it instead of writing the request themselves. An
213
+ action the product does not offer makes its tests wait. When the product refuses
214
+ an action's own request, the run files that test as a failed step, not a failed
215
+ rule, records it in `testforge/actions-failed.json`, and the next `generate`
216
+ builds that action again from the code.
217
+
218
+ A doc written before its code works the same way: its actions are written with
219
+ the doc's inputs and results and a body that waits, the tests call them, and
220
+ when the code comes `testforge update` writes those bodies from the code. The
221
+ tests themselves are never rewritten for it.
222
+
223
+ A Jest testkit's `call()` is TestForge's own (`reporters/http.cjs`): a header a
224
+ test passes, such as an old token or none, is sent as given, and a 429 from the
225
+ API's rate limiter is cleared or waited out. The testkit supplies only the
226
+ product's own parts: the URL, how a role gets a token, and how to clear the
227
+ limiter.
228
+
208
229
  **generate** hands each file it writes to the project's own ESLint with
209
230
  `--fix` before moving on. Formatting differences are corrected in place — CI
210
231
  runs lint before it runs a test, and a file that is correct but indented
package/dist/cli/index.js CHANGED
@@ -33,6 +33,7 @@ import { checkExistingDoc, runDiscover } from '../core/docs/discover.js';
33
33
  import { runMap } from '../core/project/map.js';
34
34
  import { runInit } from '../core/project/init.js';
35
35
  import { runSetup } from '../core/project/setup.js';
36
+ import { friendlyAiError, preflightAi } from '../core/project/preflight.js';
36
37
  import { runUpdate } from '../core/project/update.js';
37
38
  import { runEnvTemplate } from '../core/project/env-template.js';
38
39
  import { loadConfig } from '../core/project/config.js';
@@ -88,7 +89,7 @@ function printUsage() {
88
89
  console.log('Usage:');
89
90
  console.log(' testforge setup Run init + map + discover in order');
90
91
  console.log(' testforge setup --reference <file|url> Base the docs on a reference');
91
- console.log(' testforge setup --source map Base the docs on testforge.md', ' testforge update --refresh-docs Re-derive docs from the reference');
92
+ console.log(' testforge setup --source map Base the docs on testforge.md', ' testforge update Code changed: refresh the map and the names tests wait on (docs and tests unchanged)', ' testforge update --refresh-docs Redraft the docs from the reference, then regenerate what changed', ' testforge update --reference <x> Redraft the docs from reference <x> (it becomes the project\'s)');
92
93
  console.log(' testforge generate Generate tests from the journey docs');
93
94
  console.log(' testforge generate --basic Use the basic generator only');
94
95
  console.log(' testforge generate --force Regenerate everything');
@@ -122,8 +123,13 @@ async function main() {
122
123
  const moved = command && command !== 'help' && command !== '--help' ? migrateState() : [];
123
124
  if (moved.length)
124
125
  console.log(`Moved TestForge's state into testforge/ (${moved.length} item(s)); nothing was rebuilt.\n`);
126
+ // Commands that call the AI check it is there first (no AI call, no tokens).
127
+ const usesAi = ['update', 'setup', 'init', 'map', 'review', 'discover', 'generate'].includes(command ?? '')
128
+ && !(command === 'generate' && process.argv.includes('--basic'));
129
+ if (usesAi)
130
+ preflightAi();
125
131
  if (command === 'update') {
126
- await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
132
+ await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs'), reference: flagValue('--reference') });
127
133
  }
128
134
  else if (command === 'setup') {
129
135
  await runSetup('.', sourceOptions());
@@ -366,6 +372,7 @@ async function main() {
366
372
  }
367
373
  }
368
374
  main().catch((err) => {
369
- console.error(err instanceof Error ? err.message : String(err));
375
+ // An AI CLI that is missing or signed out gets a plain message, not a process error.
376
+ console.error(friendlyAiError(err) ?? (err instanceof Error ? err.message : String(err)));
370
377
  process.exit(1);
371
378
  });
@@ -15,6 +15,7 @@
15
15
  * keeps this simple and works for projects of any size.
16
16
  */
17
17
  import { existsSync, readdirSync, readFileSync, writeFileSync, mkdirSync, renameSync, } from 'node:fs';
18
+ import { recordDrafted } from './doc-origin.js';
18
19
  import { join, relative, sep, basename } from 'node:path';
19
20
  import { notAJourneyDoc } from './parser.js';
20
21
  import { loadConfig, splitCommand, withEditTools, withNoTools } from '../project/config.js';
@@ -149,6 +150,18 @@ function snapshotDocs(docsDir) {
149
150
  * Returns the AI's text summary of what it created.
150
151
  */
151
152
  export async function runDiscover(srcDir, docsDir, opts = {}, journey) {
153
+ // Every doc this run writes is recorded as TestForge's: a later generate
154
+ // then knows the change came from drafting, not from a person, and keeps
155
+ // the tests that already passed.
156
+ const before = snapshotDocs(docsDir);
157
+ try {
158
+ return await discoverDocs(srcDir, docsDir, opts, journey);
159
+ }
160
+ finally {
161
+ recordDrafted([...snapshotDocs(docsDir)].filter(([rel, text]) => before.get(rel) !== text));
162
+ }
163
+ }
164
+ async function discoverDocs(srcDir, docsDir, opts = {}, journey) {
152
165
  const source = await resolveDocsSource(opts);
153
166
  // One journey from a reference: no comparison of the whole product, the
154
167
  // reference read once and cut to this journey, drafted and reviewed in one
@@ -354,5 +367,7 @@ export async function checkExistingDoc(docsDir, journey, srcDir) {
354
367
  console.log(`\nChecking ${rel}${reference ? ` against ${reference}` : ' (no reference: its own [spec] lines stand)'}...`);
355
368
  const markdown = checkAndFinish(doc, text, section, excerpts, others);
356
369
  writeFileSync(join(docsDir, rel), markdown, 'utf-8');
370
+ if (markdown !== doc)
371
+ recordDrafted([[rel, markdown]]);
357
372
  return `Checked ${join(docsDir, rel)}. Review the added criteria, then run \`testforge generate --journey ${kebab(journey)}\`.`;
358
373
  }
@@ -0,0 +1,42 @@
1
+ /**
2
+ * testforge — who last wrote a journey doc: TestForge or a person.
3
+ *
4
+ * A doc is the source of truth for what the tests assert. A change a person
5
+ * makes to it is a decision about behaviour; a change TestForge makes while
6
+ * drafting (discover, check) is not, and must never quietly rewrite a test
7
+ * that already passed. Every doc TestForge writes is recorded here by the
8
+ * hash of what it wrote; a doc whose content no longer matches that hash was
9
+ * last changed by a person.
10
+ */
11
+ import { createHash } from 'node:crypto';
12
+ import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
13
+ import { dirname, join } from 'node:path';
14
+ import { STATE_DIR } from '../paths.js';
15
+ export const DOC_ORIGIN_PATH = join(STATE_DIR, 'doc-origin.json');
16
+ const hashOf = (text) => createHash('sha1').update(text).digest('hex').slice(0, 16);
17
+ const key = (rel) => rel.split('\\').join('/');
18
+ function read(path = DOC_ORIGIN_PATH) {
19
+ try {
20
+ return JSON.parse(readFileSync(path, 'utf-8'));
21
+ }
22
+ catch {
23
+ return {};
24
+ }
25
+ }
26
+ /** Record the docs TestForge just wrote ({ path relative to the docs dir: content }). */
27
+ export function recordDrafted(docs, path = DOC_ORIGIN_PATH) {
28
+ if (!docs.length)
29
+ return;
30
+ const all = read(path);
31
+ for (const [rel, text] of docs)
32
+ all[key(rel)] = hashOf(text);
33
+ try {
34
+ mkdirSync(dirname(path), { recursive: true });
35
+ writeFileSync(path, `${JSON.stringify(all, null, 2)}\n`, 'utf-8');
36
+ }
37
+ catch { /* best effort: without a record a change counts as a person's */ }
38
+ }
39
+ /** Is `content` exactly what TestForge last wrote to this doc (so no person has changed it since)? */
40
+ export function draftedByTestForge(rel, content, path = DOC_ORIGIN_PATH) {
41
+ return read(path)[key(rel)] === hashOf(content);
42
+ }
@@ -253,6 +253,23 @@ export function draftJourneyDoc(prompt) {
253
253
  const { program, args } = withNoTools(splitCommand(loadConfig().aiCommand));
254
254
  const out = runAiSync(program, args, { input: prompt, encoding: 'utf-8', maxBuffer: 10 * 1024 * 1024 }).trim();
255
255
  const area = /^AREA:\s*([\w-]+)/m.exec(out)?.[1];
256
+ return { area, markdown: journeyDocText(out) };
257
+ }
258
+ /**
259
+ * The journey doc in an AI answer: from "# Journey:" on, without the talk
260
+ * before it, and — when the doc came inside a ```markdown fence — without the
261
+ * closing fence and whatever was said after it.
262
+ */
263
+ export function journeyDocText(out) {
256
264
  const at = out.indexOf('# Journey:');
257
- return { area, markdown: at >= 0 ? `${out.slice(at).trim()}\n` : '' };
265
+ if (at < 0)
266
+ return '';
267
+ const fenced = /```[\w-]*[ \t]*\n\s*$/.test(out.slice(Math.max(0, at - 40), at));
268
+ let doc = out.slice(at);
269
+ if (fenced) {
270
+ const close = doc.search(/\n[ \t]*```[ \t]*(?:\n|$)/);
271
+ if (close >= 0)
272
+ doc = doc.slice(0, close);
273
+ }
274
+ return `${doc.trim()}\n`;
258
275
  }
@@ -216,6 +216,14 @@ export function checkTestGates(content, _ext, ctx) {
216
216
  if (ctx.framework === 'playwright' && ctx.forbidPlaceholders && /\bwaitForTimeout\s*\(/.test(content)) {
217
217
  reasons.push('page.waitForTimeout() is not used: wait for the element, response or URL the test needs (expect(...).toBeVisible(), page.waitForResponse(), expect(page).toHaveURL())');
218
218
  }
219
+ // A selector tied to markup (CSS class or id, a tag path, XPath) breaks when
220
+ // the page is restyled: elements are found the way a person finds them.
221
+ if (ctx.framework === 'playwright' && ctx.forbidPlaceholders) {
222
+ const fragile = [...content.matchAll(/\.locator\(\s*(['"`])((?:xpath=|css=)?(?:\/\/|\.[A-Za-z_-]|[#\w\]=-]*\s*[>~+]\s|[a-z][\w-]*\s*(?:\.|:(?!has\())|[a-z][\w-]*\s+[a-z.#\[]|[^'"`]*:nth-)[^'"`]*)\1/g)].map((m) => m[2]);
223
+ if (fragile.length) {
224
+ reasons.push(`uses a CSS/XPath selector (${fragile.slice(0, 2).map((f) => `"${f.slice(0, 40)}"`).join(', ')}): find elements with kit.find(page, kind, name) or page.getByRole / getByLabel / getByText / getByTestId (a testid only if <code_facts> show it)`);
225
+ }
226
+ }
219
227
  // F19 — a long test timeout needs a reason. A 240 s budget hides a test that
220
228
  // is stuck; the cap makes it fail early with its real error.
221
229
  if (ctx.maxTestTimeoutMs) {
@@ -0,0 +1,200 @@
1
+ /**
2
+ * testforge — the actions a journey's people take, written once from the code.
3
+ *
4
+ * A test asserts what the doc says; how a step reaches the product (path
5
+ * parameters, nesting, a JSON string field, two calls for one payment) is the
6
+ * product's business. When every test writes that request itself it guesses,
7
+ * and a wrong guess makes the test wait or fail for nothing. So each step a
8
+ * person takes is one function in the area's actions module, built by an agent
9
+ * that reads the handler and the product's own client, and every test calls it.
10
+ *
11
+ * The module is rebuilt when its contract changes, when a journey doc it was
12
+ * built for changes, and when a run found one of its actions refused
13
+ * (testforge/actions-failed.json, written by the reporters).
14
+ */
15
+ import { createHash } from 'node:crypto';
16
+ import { existsSync, readFileSync, rmSync } from 'node:fs';
17
+ import { dirname, join, resolve } from 'node:path';
18
+ import { loadConfig, splitCommand, withReadOnlyTools } from '../project/config.js';
19
+ import { STATE_DIR } from '../paths.js';
20
+ import { cleanOutput, runWithStdin } from './ai-generator.js';
21
+ export const ACTIONS_CONTRACT = [
22
+ `ACTIONS. One CommonJS module next to the testkit. Every step a person takes`,
23
+ `through the product goes through it, so the shape of each request is read from`,
24
+ `the code once instead of guessed in every test.`,
25
+ `A1. Which actions: each step the journey doc describes a person (or the`,
26
+ ` product on a person's behalf) doing through the product: sign up, add an`,
27
+ ` address, book, pay, accept or decline an offer, cancel, reschedule, rate,`,
28
+ ` switch a setting. One exported async function per action, named for it in`,
29
+ ` camelCase (book, payForBooking, acceptOffer). What only a test does`,
30
+ ` (scenario rows, reading the database) stays in the testkit.`,
31
+ `A2. How: read the handler the action reaches (route, validation, request`,
32
+ ` class) and the product's own client code where this repo or a related repo`,
33
+ ` has it (web, mobile). Send exactly what that client sends: path parameters,`,
34
+ ` nesting, fields sent as JSON strings, headers, and every call the client`,
35
+ ` makes for that one action, in order (a payment that first creates an`,
36
+ ` intent, then confirms it). A value the client already holds from an`,
37
+ ` earlier screen (the service picked, its price) is fetched the way that`,
38
+ ` screen fetches it. A payment provider's sandbox values are its published`,
39
+ ` test values.`,
40
+ `A3. Signature: name(inputs, { role, body, allowRefusal } = {}). inputs are`,
41
+ ` business values: the ids of the people and records involved, a time, a`,
42
+ ` service name. body is merged last into the request, for a test that sends`,
43
+ ` something unusual. role defaults to the person the doc says acts.`,
44
+ `A4. Every request goes through the testkit's call(); no HTTP or sign-in code`,
45
+ ` of its own.`,
46
+ `A5. Result: { status, body, ...named }. named holds what the doc speaks of`,
47
+ ` after the step, under the doc's own words in camelCase (orderId,`,
48
+ ` orderStatus, operatorId, total), read from where the serializer really`,
49
+ ` writes it, nested objects included. Tests assert on named, so a test`,
50
+ ` written before the code needs no change when the code comes.`,
51
+ `A6. A refusal: when the product answers outside 2xx and the caller did not`,
52
+ ` pass allowRefusal: true, throw refused(name, method, path, res). With`,
53
+ ` allowRefusal the result is returned as it is, for a test of a refusal.`,
54
+ `A7. An action the doc describes that the product does not offer yet (no route`,
55
+ ` or handler for it, or no code at all): write it all the same, with its full`,
56
+ ` signature, comment and named results from the doc, and a body of`,
57
+ ` return waiting(name, '<what was looked for>')(). Never invent a route.`,
58
+ ` When the code comes, only that body is written; the signature stays.`,
59
+ `A10. A value an earlier screen would give (a service, a slot) that the`,
60
+ ` product does not list: throw refused() for that earlier request, naming`,
61
+ ` what was looked for. Never send a placeholder (id 0, an empty value).`,
62
+ `A8. refused and waiting come from`,
63
+ ` require('@activate-core/test-forge/reporters/interface.cjs').`,
64
+ `A9. Above each function, one comment line: what it does, the route(s) it`,
65
+ ` calls (or "not in the product yet"), and the named results it returns.`,
66
+ ` The test writer sees only that line and the signature.`,
67
+ ].join('\n');
68
+ const CONTRACT_HASH = createHash('sha1').update(ACTIONS_CONTRACT).digest('hex').slice(0, 12);
69
+ const STAMP_RE = /^\/\/ testforge-actions-contract: (\w+)(?: journeys: (\S*))?\n/;
70
+ export const ACTIONS_FAILED_PATH = join(STATE_DIR, 'actions-failed.json');
71
+ const hashOf = (text) => createHash('sha1').update(text).digest('hex').slice(0, 12);
72
+ /** The actions module sits next to the testkit, with its extension. */
73
+ export function actionsPath(harnessPath) {
74
+ return join(dirname(harnessPath), `actions${/\.ts$/.test(harnessPath) ? '.ts' : '.js'}`);
75
+ }
76
+ /** The journeys (id → doc hash) a module was built for, and whether its contract is current. */
77
+ export function readStamp(source) {
78
+ const m = STAMP_RE.exec(source);
79
+ if (!m)
80
+ return { current: false, journeys: {} };
81
+ const journeys = {};
82
+ for (const pair of (m[2] ?? '').split(',').filter(Boolean)) {
83
+ const [id, h] = pair.split('=');
84
+ if (id && h)
85
+ journeys[id] = h;
86
+ }
87
+ return { current: m[1] === CONTRACT_HASH, journeys };
88
+ }
89
+ /** The module with a stamp recording its contract and the journeys (with their doc) it serves. */
90
+ export function stampActions(source, journeys) {
91
+ const list = Object.entries(journeys).sort().map(([id, h]) => `${id}=${h}`).join(',');
92
+ return `// testforge-actions-contract: ${CONTRACT_HASH} journeys: ${list}\n${source.replace(new RegExp(STAMP_RE.source, 'gm'), '')}`;
93
+ }
94
+ /** Actions a run found refused: { name: what the product answered }. */
95
+ export function readActionFailures(path = ACTIONS_FAILED_PATH) {
96
+ try {
97
+ return JSON.parse(readFileSync(path, 'utf-8'));
98
+ }
99
+ catch {
100
+ return {};
101
+ }
102
+ }
103
+ export function clearActionFailures(path = ACTIONS_FAILED_PATH) {
104
+ rmSync(path, { force: true });
105
+ }
106
+ /** Exported names of a CommonJS or ESM module. */
107
+ export function exportedNames(src) {
108
+ const names = new Set();
109
+ const cjs = /module\.exports\s*=\s*\{([\s\S]*?)\}/.exec(src);
110
+ if (cjs)
111
+ for (const n of cjs[1].split(',')) {
112
+ const m = /^\s*(\w+)/.exec(n);
113
+ if (m)
114
+ names.add(m[1]);
115
+ }
116
+ for (const m of src.matchAll(/^export\s+(?:async\s+)?(?:function\*?|const|let|class)\s+(\w+)/gm))
117
+ names.add(m[1]);
118
+ return [...names];
119
+ }
120
+ /**
121
+ * Why the module must be built now for this journey, or undefined when it is
122
+ * current: a missing module, an older contract, a doc it has not seen in this
123
+ * version, or one of its actions refused in the last run.
124
+ */
125
+ export function actionsStale(path, journeyId, doc) {
126
+ if (!existsSync(path))
127
+ return 'no actions module yet';
128
+ const src = readFileSync(path, 'utf-8');
129
+ const stamp = readStamp(src);
130
+ if (!stamp.current)
131
+ return 'its contract predates this TestForge version';
132
+ if (stamp.journeys[journeyId] !== hashOf(doc))
133
+ return `the ${journeyId} doc is new to it or changed`;
134
+ const names = exportedNames(src);
135
+ const failed = Object.keys(readActionFailures()).filter((n) => names.includes(n));
136
+ if (failed.length)
137
+ return `the last run had ${failed.join(', ')} refused`;
138
+ return undefined;
139
+ }
140
+ export function journeysWith(path, journeyId, doc) {
141
+ const before = existsSync(path) ? readStamp(readFileSync(path, 'utf-8')).journeys : {};
142
+ return { ...before, [journeyId]: hashOf(doc) };
143
+ }
144
+ export function buildActionsPrompt(path, harnessPath, docPaths) {
145
+ let previous = '';
146
+ try {
147
+ previous = readFileSync(path, 'utf-8');
148
+ }
149
+ catch { /* first build */ }
150
+ const failures = readActionFailures();
151
+ const failed = Object.entries(failures).filter(([n]) => exportedNames(previous).includes(n));
152
+ const related = (loadConfig().relatedDirs ?? []).map((d) => resolve(d));
153
+ return [
154
+ `You are a senior SDET writing the actions module of a test suite. You are inside`,
155
+ `the repository with read-only tools (Read, Grep, Glob, LS). Read before you`,
156
+ `write: every action is a claim about how the product is driven, and one built`,
157
+ `on a guess makes every test that uses it fail as if the product were broken.`,
158
+ ``,
159
+ `The module will be saved at exactly ${path}. The testkit is ${harnessPath}:`,
160
+ `read it, and require it with a path relative to the module.`,
161
+ `The journey doc(s) to cover: ${docPaths.join(', ')}. Read their steps.`,
162
+ ...(related.length
163
+ ? [`The product's other repos (its web or mobile client among them) are readable at: ${related.join(', ')}.`]
164
+ : [`No other repo of the product is configured ("relatedDirs" in testforge.config.json): read the client code`, `this repo has, if any, and the handlers.`]),
165
+ ``,
166
+ ACTIONS_CONTRACT,
167
+ ``,
168
+ ...(previous
169
+ ? [
170
+ `<existing_actions>`,
171
+ previous,
172
+ `</existing_actions>`,
173
+ `Tests already call these. Keep every export with the same name, parameters`,
174
+ `and named results, and add the actions the steps need. An action whose body is`,
175
+ `waiting(...) gets a real body where the code now has its handler.`,
176
+ ``,
177
+ ]
178
+ : []),
179
+ ...(failed.length
180
+ ? [
181
+ `The last run had these actions refused by the product. Read the handler and`,
182
+ `the client again and fix the request; if the action is right and the product`,
183
+ `is at fault, keep it as it is:`,
184
+ ...failed.map(([n, why]) => `- ${n}: ${why}`),
185
+ ``,
186
+ ]
187
+ : []),
188
+ `Output only the CommonJS code: no prose and no fences, and the first`,
189
+ `characters must be code. Do not write any file.`,
190
+ ].join('\n');
191
+ }
192
+ export function generateActionsAsync(path, harnessPath, docPaths) {
193
+ const config = loadConfig();
194
+ const { program, args } = withReadOnlyTools(splitCommand(config.aiCommand));
195
+ return runWithStdin(program, args, buildActionsPrompt(path, harnessPath, docPaths), config.aiTimeoutMs ?? 300_000).then(cleanOutput);
196
+ }
197
+ /** Does the module still have actions the product did not offer when it was built? */
198
+ export function hasWaitingActions(source) {
199
+ return /\bwaiting\(\s*['"`]/.test(source);
200
+ }
@@ -52,6 +52,34 @@ export function onlyNewTests(answer, file, rules) {
52
52
  .map((b) => answer.slice(b.start, b.end).replace(/^\s*\n/, ''))
53
53
  .join('\n\n');
54
54
  }
55
+ /**
56
+ * A kept file's earlier tests for the rules that lost their only test: the
57
+ * ones taken out to be written again whose new tests did not come back, or
58
+ * were refused. They go back with the need() lines they use. A waiting test
59
+ * beats no test.
60
+ */
61
+ export function putBack(out, previous, taken, rules) {
62
+ const now = findTestBlocks(out);
63
+ const have = new Set(now.flatMap((b) => b.rules));
64
+ const titles = new Set(now.map((b) => b.title));
65
+ const back = taken.filter((b) => !titles.has(b.title) && b.rules.some((r) => rules.includes(r) && !have.has(r)));
66
+ if (!back.length)
67
+ return { content: out, rules: [] };
68
+ const texts = back.map((b) => previous.slice(b.start, b.end).replace(/^\s*\n/, ''));
69
+ let content = appendTests(out, texts.join('\n\n'));
70
+ const declared = new Set([...content.matchAll(/^[ \t]*(?:const|let)\s+([A-Za-z_$][\w$]*)\s*=/gm)].map((m) => m[1]));
71
+ const lines = previous.split('\n').filter((l) => {
72
+ const m = /^[ \t]*(?:const|let)\s+([A-Za-z_$][\w$]*)\s*=\s*need\(/.exec(l);
73
+ return m && !declared.has(m[1]) && texts.some((t) => new RegExp(`\\b${m[1].replace(/\$/g, '\\$')}\\b`).test(t));
74
+ });
75
+ if (lines.length) {
76
+ // After the last need() or require() line at the top, where the file keeps them.
77
+ const top = [...content.matchAll(/^(?:const|let)\s+[^\n]*\b(?:need|require)\([^\n]*\n/gm)].pop();
78
+ const at = top ? (top.index ?? 0) + top[0].length : 0;
79
+ content = content.slice(0, at) + lines.join('\n') + '\n' + content.slice(at);
80
+ }
81
+ return { content, rules: [...new Set(back.flatMap((b) => b.rules.filter((r) => rules.includes(r) && !have.has(r))))] };
82
+ }
55
83
  /** Is the file complete as it is? */
56
84
  export function fileIsComplete(content, rules, steps, knownTexts) {
57
85
  const { cov, problems } = review(content, rules, steps, knownTexts);
@@ -141,6 +169,21 @@ export async function completeFile(content, ctx) {
141
169
  }
142
170
  }
143
171
  }
172
+ // Some new tests came back and some did not (refused, or taken out by the
173
+ // second check): the rules left with no test get their earlier test back.
174
+ if (ctx.kept && !failedRedo && firstDrop.length) {
175
+ const back = putBack(out, content, firstDrop.map((p) => p.block), ctx.rules);
176
+ if (back.rules.length && !ctx.check(back.content)) {
177
+ notes.push(`put back the earlier test(s) for ${back.rules.join(', ')}: their new tests did not pass the checks`);
178
+ out = back.content;
179
+ for (const p of firstDrop)
180
+ if (p.rules.some((r) => back.rules.includes(r)))
181
+ dropped.delete(p.title);
182
+ }
183
+ else if (back.rules.length) {
184
+ failedRedo = true; // the earlier tests do not fit back in: keep the file as it was
185
+ }
186
+ }
144
187
  // A kept file whose tests were taken out to be written again, and the new
145
188
  // ones did not come: it stays as it was. A waiting test beats no test.
146
189
  if (ctx.kept && failedRedo && redo.length + firstDrop.length > 0) {
@@ -37,7 +37,7 @@ export const HARNESS_CONTRACT = [
37
37
  `- one resolver per seeded entity that tests read back (H3)`,
38
38
  `- unique(label?): string, and created(kind, id, cleanup?, role?): void (H8)`,
39
39
  `- distinctRunAccount(): boolean (H8)`,
40
- `- resetRateLimits(): Promise<void> (H7)`,
40
+ `- resetRateLimits(): Promise<number>, the entries it cleared (H7)`,
41
41
  `- REQUIRED_PACKAGES: string[] (H6)`,
42
42
  `- a 'run' role alongside the fixture roles (H8)`,
43
43
  `- insert(table, row): Promise<id>, one row of scenario data written straight`,
@@ -48,14 +48,19 @@ export const HARNESS_CONTRACT = [
48
48
  `Jest areas only`,
49
49
  `- call(method, path, { body?, headers?, role? }):`,
50
50
  ` Promise<{ status, headers, body }>`,
51
- ` Every API test sends its requests through it, with the role's token. It`,
52
- ` drives the server in-process with the framework's own inject when the`,
53
- ` server is a Node app this package can require; otherwise (a server in`,
54
- ` another language, or one this package cannot load) it sends HTTP to the`,
55
- ` running server at the base URL named in the env file, with Node's fetch or`,
56
- ` an HTTP client the project already has. Never require '@playwright/test':`,
57
- ` it is not installed in backend packages, and a top-level require breaks`,
58
- ` every suite.`,
51
+ ` Every API test sends its requests through it. It is TestForge's own, made`,
52
+ ` with makeCall; never write call(), its token handling or its retries`,
53
+ ` yourself:`,
54
+ ` const { makeCall, verifyDerivedState } =`,
55
+ ` require('@activate-core/test-forge/reporters/http.cjs');`,
56
+ ` const call = makeCall({ url, ensureToken, resetRateLimits, refused, send });`,
57
+ ` url(path) gives the full URL (the product's prefix rules); ensureToken(role,`,
58
+ ` { force }) a token the product accepts; refused(status) is true for the`,
59
+ ` statuses that mean "token not accepted" (default 401). Pass send(method,`,
60
+ ` url, { headers, body }) only for a Node server this package can require and`,
61
+ ` drive in-process with its framework's inject; otherwise leave it out (HTTP to`,
62
+ ` the base URL in the env file). Never require '@playwright/test': it is not`,
63
+ ` installed in backend packages, and a top-level require breaks every suite.`,
59
64
  `Playwright areas only`,
60
65
  `- login(page, role?): Promise<void>, a session the app really accepts (H2).`,
61
66
  ` It calls suppressPopups first.`,
@@ -143,9 +148,11 @@ export const HARNESS_CONTRACT = [
143
148
  ` - suppressPopups(page) uses an init script to pre-set the flags that stop`,
144
149
  ` promotions, consent, onboarding and permission prompts, before the app's`,
145
150
  ` code runs and across navigations.`,
146
- ` - resetRateLimits() clears only the limiter's entries, and only when they`,
147
- ` live in a store the testkit can reach locally. It is gated like reset(),`,
148
- ` and setup() calls it.`,
151
+ ` - resetRateLimits() clears only the limiter's entries (read in the code how`,
152
+ ` the limiter names them, prefix included), and only when they live in a`,
153
+ ` store the testkit can reach. It removes no data, so it needs only a target`,
154
+ ` that is clearly not production (H5), not TESTFORGE_ALLOW_RESET. It`,
155
+ ` returns how many entries it cleared and never throws; setup() calls it.`,
149
156
  ` - bypassVerification answers the final verify request at url with the`,
150
157
  ` app's real success shape, carrying a real token for an env account of`,
151
158
  ` that role. It answers sendUrl the same way when the environment cannot`,
@@ -189,6 +196,24 @@ export const HARNESS_CONTRACT = [
189
196
  ` - insert fills every NOT NULL column without a default that row leaves out`,
190
197
  ` with a value of its type (read the columns from the database), returns`,
191
198
  ` the new id, and remembers the row.`,
199
+ ` - A row the product checks before it will use it (a service needs a price`,
200
+ ` above the minimum, a record needs to be enabled): read those checks in the`,
201
+ ` code and have insert set each checked column the row leaves out to a value`,
202
+ ` that passes it, per table. A row a test writes is then usable, the way a`,
203
+ ` real one is.`,
204
+ ` - Derived state. Some tables the product fills by itself when a row is`,
205
+ ` saved: model events and observers, and the jobs they queue (a busy-time`,
206
+ ` calendar built from working hours and orders, counters, a search table).`,
207
+ ` insert() and update() write straight to the database and fire none of`,
208
+ ` them, so read the observers, events and jobs of every table a scenario`,
209
+ ` writes. After such a write, insert and update bring the derived rows to`,
210
+ ` what the product would have made, computing them the way that code does,`,
211
+ ` and remember them so they are undone with the row. Export`,
212
+ ` DERIVED_STATE: { <table written>: '<what is rebuilt, from which code>' },`,
213
+ ` keyed by the table name the model really uses (its $table, its`,
214
+ ` tableName), and have setup() call verifyDerivedState(DERIVED_STATE, q)`,
215
+ ` once connected, where q(sql) runs SQL on the test database.`,
216
+ ` A scenario built without this tests nothing: every operator looks free.`,
192
217
  ` - update remembers the previous values of the rows it changes.`,
193
218
  ` - Each test starts from a clean slate: the rows a test writes, and what it`,
194
219
  ` updates, are undone right after that test (last first), so one test's`,
@@ -314,9 +339,10 @@ export function generateHarness(harnessPath, framework) {
314
339
  return cleanOutput(output);
315
340
  }
316
341
  /** Async twin, so a harness can generate alongside other work. */
317
- export function generateHarnessAsync(harnessPath, framework) {
342
+ export function generateHarnessAsync(harnessPath, framework, refusal) {
318
343
  const config = loadConfig();
319
- const prompt = buildHarnessPrompt(harnessPath, framework);
344
+ const prompt = buildHarnessPrompt(harnessPath, framework) +
345
+ (refusal ? `\n\nYour previous testkit was refused: ${refusal.reason}. Write it again with that fixed.` : '');
320
346
  const { program, args } = withReadOnlyTools(splitCommand(config.aiCommand));
321
347
  // Same runner as every other async AI call: one timeout guard, one usage count, one trace.
322
348
  return runWithStdin(program, args, prompt, config.aiTimeoutMs ?? 300_000).then(cleanOutput);
@@ -507,3 +533,20 @@ export function fixturesDoc() {
507
533
  ``,
508
534
  ].join('\n');
509
535
  }
536
+ /**
537
+ * Why a generated testkit is not taken, or null: a Jest testkit must build
538
+ * call() with TestForge's makeCall (the header and 429 rules are code, not
539
+ * something each testkit is trusted to write), and one that derives rows must
540
+ * check its table names at setup.
541
+ */
542
+ export function harnessGate(source, framework) {
543
+ if (framework !== 'jest')
544
+ return null;
545
+ if (!/reporters\/http\.cjs/.test(source) || !/\bmakeCall\s*\(/.test(source)) {
546
+ return "call() must be made with makeCall from require('@activate-core/test-forge/reporters/http.cjs'), not written by hand";
547
+ }
548
+ if (/\bDERIVED_STATE\b/.test(source) && !/\bverifyDerivedState\s*\(/.test(source)) {
549
+ return 'setup() must call verifyDerivedState(DERIVED_STATE, q) so a wrong table name stops the run instead of deriving nothing';
550
+ }
551
+ return null;
552
+ }