@activate-core/test-forge 2.7.0 → 2.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +21 -0
- package/dist/cli/index.js +14 -4
- package/dist/core/docs/discover.js +15 -0
- package/dist/core/docs/doc-origin.js +42 -0
- package/dist/core/docs/reference.js +18 -1
- package/dist/core/gates/validate.js +8 -0
- package/dist/core/generate/actions.js +200 -0
- package/dist/core/generate/ai-generator.js +41 -0
- package/dist/core/generate/ai-usage.js +1 -0
- package/dist/core/generate/complete-file.js +62 -2
- package/dist/core/generate/harness.js +57 -14
- package/dist/core/generate/run-generate.js +203 -27
- package/dist/core/generate/test-blocks.js +123 -2
- package/dist/core/generate/update-journey.js +23 -3
- package/dist/core/project/preflight.js +69 -0
- package/dist/core/project/update.js +120 -14
- package/dist/core/prompts/blocks.js +17 -0
- package/package.json +1 -1
- package/reporters/_shared.cjs +33 -2
- package/reporters/http.cjs +96 -0
- package/reporters/interface.cjs +25 -2
- package/reporters/jest-reporter.cjs +3 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,38 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.9.0
|
|
4
|
+
|
|
5
|
+
2.8.0 was not published to npm; its changes (below) ship in 2.9.0.
|
|
6
|
+
|
|
7
|
+
- A Jest testkit's `call()` is TestForge's own code (`reporters/http.cjs`, `makeCall`), no longer written by the AI in each project: an `Authorization` header a test passes (an old token, or `''` for none) is sent exactly as given, `role: null` sends none, a refused token is minted again once, and a 429 is cleared through `resetRateLimits()` or waited out for its Retry-After (2 s when none), up to 60 s. A testkit that does not use it is refused at generate time. Tests of "no login" and "after sign-out" sent a signed-in request before.
|
|
8
|
+
- Rows a scenario inserts get the state the product derives from them (observers, model events and the jobs they queue, e.g. a busy-time calendar from working hours and orders), computed the way that code does and undone with the row. `DERIVED_STATE` lists what is rebuilt, keyed by the real table name; `setup()` checks those tables exist (`verifyDerivedState`) and stops the run on a wrong name. Without it every hours or availability check saw every operator as free.
|
|
9
|
+
- Doc first, code later: a journey written before its code gets its actions all the same (full signature and named results from the doc, a body that waits). Its tests call them and assert on the named results. When the code comes, `testforge update` writes only the waiting actions from the code; the tests are not touched and stop waiting by themselves.
|
|
10
|
+
- `test.each(table)('title', fn)` is a test of its rule. An earlier version did not see it, read the rule as untested and appended the test again on every generate; the same test under one title is now kept once (no AI).
|
|
11
|
+
- A kept file whose tests were taken out to be written again gets back the earlier test of every rule whose new test did not come; a file written again whole is taken only if it still tests every rule the earlier one did.
|
|
12
|
+
- A blank name used inside a template literal (`` `${URL}?${PARAM}=…` ``) counts as a use: a test, or a helper outside the tests, that builds a request that way now waits instead of sending a placeholder.
|
|
13
|
+
- Jest areas get an actions module (`helpers/actions.js`): one function per step a person takes through the product, written from the handler and the product's own client (`relatedDirs`), so tests no longer guess a request's shape (path parameters, nesting, JSON string fields, multi-call payments). An action the product lacks makes its tests wait; a refused action is filed as a failed step (not a failed rule), recorded in `testforge/actions-failed.json`, and built again on the next `generate`. `generate --rewrite-waiting` writes the waiting tests again with the actions once the module is new or rebuilt.
|
|
14
|
+
- Clearing the API's rate limiter removes no data, so it needs only a non-production target, not `TESTFORGE_ALLOW_RESET`. A row a test inserts gets the values the product checks before it will use it (a price above the minimum, an enabled flag).
|
|
15
|
+
|
|
16
|
+
### Upgrading
|
|
17
|
+
- Existing testkits are rebuilt once on the next `generate` (contract changed), and each Jest journey's actions module is built once (one call per journey doc).
|
|
18
|
+
|
|
19
|
+
## 2.8.0
|
|
20
|
+
|
|
21
|
+
- Every command that calls the AI first checks, with no AI call, that the configured AI CLI is on PATH and runs; a missing or signed-out CLI gets a plain message instead of a process error.
|
|
22
|
+
- A journey doc the AI returns inside a ```markdown fence is written without the closing fence and whatever was said after it.
|
|
23
|
+
- UI tests never select by markup: CSS classes, tag paths, positions and XPath are refused in new tests (an id or name attribute the code facts show stays the last resort for an input with no label).
|
|
24
|
+
|
|
25
|
+
- `testforge update` never redrafts the journey docs because the code changed: it refreshes the project map and the names the tests wait on (`interface.json`) only. A changed reference is reported, not acted on. The docs are redrafted only with `update --refresh-docs` or `update --reference <x>`.
|
|
26
|
+
- Docs TestForge writes (discover, check) are recorded in `testforge/doc-origin.json`. When a doc change came from TestForge's drafting rather than a person, generate never renames, removes or rewrites a test that passed (`testforge/passed.json`); a person's edit to the doc, or `--force`, still can.
|
|
27
|
+
- The reporters file a refused connection (`ECONNREFUSED`) and a 429 from the testkit as the environment, not a product failure.
|
|
28
|
+
|
|
29
|
+
## 2.7.1
|
|
30
|
+
|
|
31
|
+
- New code never makes generate write a test again. With the doc unchanged and every rule tested, `generate --refresh-code` reads the code facts again and fills the names the tests wait on from them (two small calls); the test files are not rewritten. `--rewrite-waiting` writes the waiting tests again when asked by name.
|
|
32
|
+
- kit.insert's id is used as the id it is, with no AI: `op[ID_FIELD]` (ID_FIELD = need('inserted row id field')) becomes `op`, and the blank name goes.
|
|
33
|
+
- With `--rewrite-waiting`: a file whose blank names sit in its hooks or helpers has only that setup written again (its tests stay); a file whose blank names sit in a parametrised table or loop is written again whole, once per doc and code, never one holding a passed test.
|
|
34
|
+
- A refused answer keeps the tests in it that pass the checks on their own.
|
|
35
|
+
|
|
3
36
|
## 2.7.0
|
|
4
37
|
|
|
5
38
|
- The names new tests still wait on are looked up once, at generate time, against the journey's code facts; a value is kept only if the code has it.
|
package/README.md
CHANGED
|
@@ -205,6 +205,27 @@ otherwise — and checks stored state only through the testkit; `Type: web`
|
|
|
205
205
|
becomes a Playwright test. Other areas use the framework set in config. It also
|
|
206
206
|
generates a seed file that provisions the test data each test needs.
|
|
207
207
|
|
|
208
|
+
In a Jest area every step a person takes through the product (book, pay, accept
|
|
209
|
+
an offer…) is one function of `helpers/actions.js`, next to the testkit. It is
|
|
210
|
+
written once per journey doc by an agent that reads the handler and the
|
|
211
|
+
product's own client (set `relatedDirs` in `testforge.config.json` to the web or
|
|
212
|
+
mobile repo), and tests call it instead of writing the request themselves. An
|
|
213
|
+
action the product does not offer makes its tests wait. When the product refuses
|
|
214
|
+
an action's own request, the run files that test as a failed step, not a failed
|
|
215
|
+
rule, records it in `testforge/actions-failed.json`, and the next `generate`
|
|
216
|
+
builds that action again from the code.
|
|
217
|
+
|
|
218
|
+
A doc written before its code works the same way: its actions are written with
|
|
219
|
+
the doc's inputs and results and a body that waits, the tests call them, and
|
|
220
|
+
when the code comes `testforge update` writes those bodies from the code. The
|
|
221
|
+
tests themselves are never rewritten for it.
|
|
222
|
+
|
|
223
|
+
A Jest testkit's `call()` is TestForge's own (`reporters/http.cjs`): a header a
|
|
224
|
+
test passes, such as an old token or none, is sent as given, and a 429 from the
|
|
225
|
+
API's rate limiter is cleared or waited out. The testkit supplies only the
|
|
226
|
+
product's own parts: the URL, how a role gets a token, and how to clear the
|
|
227
|
+
limiter.
|
|
228
|
+
|
|
208
229
|
**generate** hands each file it writes to the project's own ESLint with
|
|
209
230
|
`--fix` before moving on. Formatting differences are corrected in place — CI
|
|
210
231
|
runs lint before it runs a test, and a file that is correct but indented
|
package/dist/cli/index.js
CHANGED
|
@@ -33,6 +33,7 @@ import { checkExistingDoc, runDiscover } from '../core/docs/discover.js';
|
|
|
33
33
|
import { runMap } from '../core/project/map.js';
|
|
34
34
|
import { runInit } from '../core/project/init.js';
|
|
35
35
|
import { runSetup } from '../core/project/setup.js';
|
|
36
|
+
import { friendlyAiError, preflightAi } from '../core/project/preflight.js';
|
|
36
37
|
import { runUpdate } from '../core/project/update.js';
|
|
37
38
|
import { runEnvTemplate } from '../core/project/env-template.js';
|
|
38
39
|
import { loadConfig } from '../core/project/config.js';
|
|
@@ -88,12 +89,12 @@ function printUsage() {
|
|
|
88
89
|
console.log('Usage:');
|
|
89
90
|
console.log(' testforge setup Run init + map + discover in order');
|
|
90
91
|
console.log(' testforge setup --reference <file|url> Base the docs on a reference');
|
|
91
|
-
console.log(' testforge setup --source map Base the docs on testforge.md', ' testforge update --refresh-docs
|
|
92
|
+
console.log(' testforge setup --source map Base the docs on testforge.md', ' testforge update Code changed: refresh the map and the names tests wait on (docs and tests unchanged)', ' testforge update --refresh-docs Redraft the docs from the reference, then regenerate what changed', ' testforge update --reference <x> Redraft the docs from reference <x> (it becomes the project\'s)');
|
|
92
93
|
console.log(' testforge generate Generate tests from the journey docs');
|
|
93
94
|
console.log(' testforge generate --basic Use the basic generator only');
|
|
94
95
|
console.log(' testforge generate --force Regenerate everything');
|
|
95
96
|
console.log(' testforge generate --no-wait Stop when the AI hits its usage limit (default: wait for the reset and resume)');
|
|
96
|
-
console.log(' testforge generate --plan-only Plan and check each journey (testforge/journeys/<area>/<id>/plan-check.md), write no tests', ' testforge generate --journey <name> --refresh-code Read the code again and rewrite the tests still waiting on names (
|
|
97
|
+
console.log(' testforge generate --plan-only Plan and check each journey (testforge/journeys/<area>/<id>/plan-check.md), write no tests', ' testforge generate --journey <name> --refresh-code Read the code again and fill the names the tests wait on (tests are not written again)', ' testforge generate --journey <name> --rewrite-waiting Also write again the tests still waiting on names (costs AI calls)');
|
|
97
98
|
console.log(' testforge generate --journey <name> Generate just ONE journey');
|
|
98
99
|
console.log(' testforge generate --area <area> Generate a whole area (e.g. api)');
|
|
99
100
|
console.log(' testforge discover Draft journey docs (asks: map or reference)');
|
|
@@ -122,8 +123,13 @@ async function main() {
|
|
|
122
123
|
const moved = command && command !== 'help' && command !== '--help' ? migrateState() : [];
|
|
123
124
|
if (moved.length)
|
|
124
125
|
console.log(`Moved TestForge's state into testforge/ (${moved.length} item(s)); nothing was rebuilt.\n`);
|
|
126
|
+
// Commands that call the AI check it is there first (no AI call, no tokens).
|
|
127
|
+
const usesAi = ['update', 'setup', 'init', 'map', 'review', 'discover', 'generate'].includes(command ?? '')
|
|
128
|
+
&& !(command === 'generate' && process.argv.includes('--basic'));
|
|
129
|
+
if (usesAi)
|
|
130
|
+
preflightAi();
|
|
125
131
|
if (command === 'update') {
|
|
126
|
-
await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
|
|
132
|
+
await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs'), reference: flagValue('--reference') });
|
|
127
133
|
}
|
|
128
134
|
else if (command === 'setup') {
|
|
129
135
|
await runSetup('.', sourceOptions());
|
|
@@ -250,6 +256,7 @@ async function main() {
|
|
|
250
256
|
: undefined, // undefined -> config.waitForLimit (default on)
|
|
251
257
|
planOnly: process.argv.includes('--plan-only'),
|
|
252
258
|
refreshCode: process.argv.includes('--refresh-code'),
|
|
259
|
+
rewriteWaiting: process.argv.includes('--rewrite-waiting'),
|
|
253
260
|
});
|
|
254
261
|
if (results.length === 0) {
|
|
255
262
|
console.log('No journey files found in testforge/docs/. Add a .md file and try again.');
|
|
@@ -260,6 +267,8 @@ async function main() {
|
|
|
260
267
|
// A row with no test dir is the run-level notice (AI unavailable), not an
|
|
261
268
|
// unchanged journey — do not label it as one.
|
|
262
269
|
console.log(r.testDir ? ` [skip] ${r.journeyName} (unchanged)` : ` [skip] ${r.journeyName}`);
|
|
270
|
+
for (const w of r.warnings ?? [])
|
|
271
|
+
console.log(` ! ${w}`);
|
|
263
272
|
}
|
|
264
273
|
else {
|
|
265
274
|
const tag = r.status === 'ai' ? '[AI]' : '[basic]';
|
|
@@ -363,6 +372,7 @@ async function main() {
|
|
|
363
372
|
}
|
|
364
373
|
}
|
|
365
374
|
main().catch((err) => {
|
|
366
|
-
|
|
375
|
+
// An AI CLI that is missing or signed out gets a plain message, not a process error.
|
|
376
|
+
console.error(friendlyAiError(err) ?? (err instanceof Error ? err.message : String(err)));
|
|
367
377
|
process.exit(1);
|
|
368
378
|
});
|
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
* keeps this simple and works for projects of any size.
|
|
16
16
|
*/
|
|
17
17
|
import { existsSync, readdirSync, readFileSync, writeFileSync, mkdirSync, renameSync, } from 'node:fs';
|
|
18
|
+
import { recordDrafted } from './doc-origin.js';
|
|
18
19
|
import { join, relative, sep, basename } from 'node:path';
|
|
19
20
|
import { notAJourneyDoc } from './parser.js';
|
|
20
21
|
import { loadConfig, splitCommand, withEditTools, withNoTools } from '../project/config.js';
|
|
@@ -149,6 +150,18 @@ function snapshotDocs(docsDir) {
|
|
|
149
150
|
* Returns the AI's text summary of what it created.
|
|
150
151
|
*/
|
|
151
152
|
export async function runDiscover(srcDir, docsDir, opts = {}, journey) {
|
|
153
|
+
// Every doc this run writes is recorded as TestForge's: a later generate
|
|
154
|
+
// then knows the change came from drafting, not from a person, and keeps
|
|
155
|
+
// the tests that already passed.
|
|
156
|
+
const before = snapshotDocs(docsDir);
|
|
157
|
+
try {
|
|
158
|
+
return await discoverDocs(srcDir, docsDir, opts, journey);
|
|
159
|
+
}
|
|
160
|
+
finally {
|
|
161
|
+
recordDrafted([...snapshotDocs(docsDir)].filter(([rel, text]) => before.get(rel) !== text));
|
|
162
|
+
}
|
|
163
|
+
}
|
|
164
|
+
async function discoverDocs(srcDir, docsDir, opts = {}, journey) {
|
|
152
165
|
const source = await resolveDocsSource(opts);
|
|
153
166
|
// One journey from a reference: no comparison of the whole product, the
|
|
154
167
|
// reference read once and cut to this journey, drafted and reviewed in one
|
|
@@ -354,5 +367,7 @@ export async function checkExistingDoc(docsDir, journey, srcDir) {
|
|
|
354
367
|
console.log(`\nChecking ${rel}${reference ? ` against ${reference}` : ' (no reference: its own [spec] lines stand)'}...`);
|
|
355
368
|
const markdown = checkAndFinish(doc, text, section, excerpts, others);
|
|
356
369
|
writeFileSync(join(docsDir, rel), markdown, 'utf-8');
|
|
370
|
+
if (markdown !== doc)
|
|
371
|
+
recordDrafted([[rel, markdown]]);
|
|
357
372
|
return `Checked ${join(docsDir, rel)}. Review the added criteria, then run \`testforge generate --journey ${kebab(journey)}\`.`;
|
|
358
373
|
}
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* testforge — who last wrote a journey doc: TestForge or a person.
|
|
3
|
+
*
|
|
4
|
+
* A doc is the source of truth for what the tests assert. A change a person
|
|
5
|
+
* makes to it is a decision about behaviour; a change TestForge makes while
|
|
6
|
+
* drafting (discover, check) is not, and must never quietly rewrite a test
|
|
7
|
+
* that already passed. Every doc TestForge writes is recorded here by the
|
|
8
|
+
* hash of what it wrote; a doc whose content no longer matches that hash was
|
|
9
|
+
* last changed by a person.
|
|
10
|
+
*/
|
|
11
|
+
import { createHash } from 'node:crypto';
|
|
12
|
+
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
|
13
|
+
import { dirname, join } from 'node:path';
|
|
14
|
+
import { STATE_DIR } from '../paths.js';
|
|
15
|
+
export const DOC_ORIGIN_PATH = join(STATE_DIR, 'doc-origin.json');
|
|
16
|
+
const hashOf = (text) => createHash('sha1').update(text).digest('hex').slice(0, 16);
|
|
17
|
+
const key = (rel) => rel.split('\\').join('/');
|
|
18
|
+
function read(path = DOC_ORIGIN_PATH) {
|
|
19
|
+
try {
|
|
20
|
+
return JSON.parse(readFileSync(path, 'utf-8'));
|
|
21
|
+
}
|
|
22
|
+
catch {
|
|
23
|
+
return {};
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
/** Record the docs TestForge just wrote ({ path relative to the docs dir: content }). */
|
|
27
|
+
export function recordDrafted(docs, path = DOC_ORIGIN_PATH) {
|
|
28
|
+
if (!docs.length)
|
|
29
|
+
return;
|
|
30
|
+
const all = read(path);
|
|
31
|
+
for (const [rel, text] of docs)
|
|
32
|
+
all[key(rel)] = hashOf(text);
|
|
33
|
+
try {
|
|
34
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
35
|
+
writeFileSync(path, `${JSON.stringify(all, null, 2)}\n`, 'utf-8');
|
|
36
|
+
}
|
|
37
|
+
catch { /* best effort: without a record a change counts as a person's */ }
|
|
38
|
+
}
|
|
39
|
+
/** Is `content` exactly what TestForge last wrote to this doc (so no person has changed it since)? */
|
|
40
|
+
export function draftedByTestForge(rel, content, path = DOC_ORIGIN_PATH) {
|
|
41
|
+
return read(path)[key(rel)] === hashOf(content);
|
|
42
|
+
}
|
|
@@ -253,6 +253,23 @@ export function draftJourneyDoc(prompt) {
|
|
|
253
253
|
const { program, args } = withNoTools(splitCommand(loadConfig().aiCommand));
|
|
254
254
|
const out = runAiSync(program, args, { input: prompt, encoding: 'utf-8', maxBuffer: 10 * 1024 * 1024 }).trim();
|
|
255
255
|
const area = /^AREA:\s*([\w-]+)/m.exec(out)?.[1];
|
|
256
|
+
return { area, markdown: journeyDocText(out) };
|
|
257
|
+
}
|
|
258
|
+
/**
|
|
259
|
+
* The journey doc in an AI answer: from "# Journey:" on, without the talk
|
|
260
|
+
* before it, and — when the doc came inside a ```markdown fence — without the
|
|
261
|
+
* closing fence and whatever was said after it.
|
|
262
|
+
*/
|
|
263
|
+
export function journeyDocText(out) {
|
|
256
264
|
const at = out.indexOf('# Journey:');
|
|
257
|
-
|
|
265
|
+
if (at < 0)
|
|
266
|
+
return '';
|
|
267
|
+
const fenced = /```[\w-]*[ \t]*\n\s*$/.test(out.slice(Math.max(0, at - 40), at));
|
|
268
|
+
let doc = out.slice(at);
|
|
269
|
+
if (fenced) {
|
|
270
|
+
const close = doc.search(/\n[ \t]*```[ \t]*(?:\n|$)/);
|
|
271
|
+
if (close >= 0)
|
|
272
|
+
doc = doc.slice(0, close);
|
|
273
|
+
}
|
|
274
|
+
return `${doc.trim()}\n`;
|
|
258
275
|
}
|
|
@@ -216,6 +216,14 @@ export function checkTestGates(content, _ext, ctx) {
|
|
|
216
216
|
if (ctx.framework === 'playwright' && ctx.forbidPlaceholders && /\bwaitForTimeout\s*\(/.test(content)) {
|
|
217
217
|
reasons.push('page.waitForTimeout() is not used: wait for the element, response or URL the test needs (expect(...).toBeVisible(), page.waitForResponse(), expect(page).toHaveURL())');
|
|
218
218
|
}
|
|
219
|
+
// A selector tied to markup (CSS class or id, a tag path, XPath) breaks when
|
|
220
|
+
// the page is restyled: elements are found the way a person finds them.
|
|
221
|
+
if (ctx.framework === 'playwright' && ctx.forbidPlaceholders) {
|
|
222
|
+
const fragile = [...content.matchAll(/\.locator\(\s*(['"`])((?:xpath=|css=)?(?:\/\/|\.[A-Za-z_-]|[#\w\]=-]*\s*[>~+]\s|[a-z][\w-]*\s*(?:\.|:(?!has\())|[a-z][\w-]*\s+[a-z.#\[]|[^'"`]*:nth-)[^'"`]*)\1/g)].map((m) => m[2]);
|
|
223
|
+
if (fragile.length) {
|
|
224
|
+
reasons.push(`uses a CSS/XPath selector (${fragile.slice(0, 2).map((f) => `"${f.slice(0, 40)}"`).join(', ')}): find elements with kit.find(page, kind, name) or page.getByRole / getByLabel / getByText / getByTestId (a testid only if <code_facts> show it)`);
|
|
225
|
+
}
|
|
226
|
+
}
|
|
219
227
|
// F19 — a long test timeout needs a reason. A 240 s budget hides a test that
|
|
220
228
|
// is stuck; the cap makes it fail early with its real error.
|
|
221
229
|
if (ctx.maxTestTimeoutMs) {
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* testforge — the actions a journey's people take, written once from the code.
|
|
3
|
+
*
|
|
4
|
+
* A test asserts what the doc says; how a step reaches the product (path
|
|
5
|
+
* parameters, nesting, a JSON string field, two calls for one payment) is the
|
|
6
|
+
* product's business. When every test writes that request itself it guesses,
|
|
7
|
+
* and a wrong guess makes the test wait or fail for nothing. So each step a
|
|
8
|
+
* person takes is one function in the area's actions module, built by an agent
|
|
9
|
+
* that reads the handler and the product's own client, and every test calls it.
|
|
10
|
+
*
|
|
11
|
+
* The module is rebuilt when its contract changes, when a journey doc it was
|
|
12
|
+
* built for changes, and when a run found one of its actions refused
|
|
13
|
+
* (testforge/actions-failed.json, written by the reporters).
|
|
14
|
+
*/
|
|
15
|
+
import { createHash } from 'node:crypto';
|
|
16
|
+
import { existsSync, readFileSync, rmSync } from 'node:fs';
|
|
17
|
+
import { dirname, join, resolve } from 'node:path';
|
|
18
|
+
import { loadConfig, splitCommand, withReadOnlyTools } from '../project/config.js';
|
|
19
|
+
import { STATE_DIR } from '../paths.js';
|
|
20
|
+
import { cleanOutput, runWithStdin } from './ai-generator.js';
|
|
21
|
+
export const ACTIONS_CONTRACT = [
|
|
22
|
+
`ACTIONS. One CommonJS module next to the testkit. Every step a person takes`,
|
|
23
|
+
`through the product goes through it, so the shape of each request is read from`,
|
|
24
|
+
`the code once instead of guessed in every test.`,
|
|
25
|
+
`A1. Which actions: each step the journey doc describes a person (or the`,
|
|
26
|
+
` product on a person's behalf) doing through the product: sign up, add an`,
|
|
27
|
+
` address, book, pay, accept or decline an offer, cancel, reschedule, rate,`,
|
|
28
|
+
` switch a setting. One exported async function per action, named for it in`,
|
|
29
|
+
` camelCase (book, payForBooking, acceptOffer). What only a test does`,
|
|
30
|
+
` (scenario rows, reading the database) stays in the testkit.`,
|
|
31
|
+
`A2. How: read the handler the action reaches (route, validation, request`,
|
|
32
|
+
` class) and the product's own client code where this repo or a related repo`,
|
|
33
|
+
` has it (web, mobile). Send exactly what that client sends: path parameters,`,
|
|
34
|
+
` nesting, fields sent as JSON strings, headers, and every call the client`,
|
|
35
|
+
` makes for that one action, in order (a payment that first creates an`,
|
|
36
|
+
` intent, then confirms it). A value the client already holds from an`,
|
|
37
|
+
` earlier screen (the service picked, its price) is fetched the way that`,
|
|
38
|
+
` screen fetches it. A payment provider's sandbox values are its published`,
|
|
39
|
+
` test values.`,
|
|
40
|
+
`A3. Signature: name(inputs, { role, body, allowRefusal } = {}). inputs are`,
|
|
41
|
+
` business values: the ids of the people and records involved, a time, a`,
|
|
42
|
+
` service name. body is merged last into the request, for a test that sends`,
|
|
43
|
+
` something unusual. role defaults to the person the doc says acts.`,
|
|
44
|
+
`A4. Every request goes through the testkit's call(); no HTTP or sign-in code`,
|
|
45
|
+
` of its own.`,
|
|
46
|
+
`A5. Result: { status, body, ...named }. named holds what the doc speaks of`,
|
|
47
|
+
` after the step, under the doc's own words in camelCase (orderId,`,
|
|
48
|
+
` orderStatus, operatorId, total), read from where the serializer really`,
|
|
49
|
+
` writes it, nested objects included. Tests assert on named, so a test`,
|
|
50
|
+
` written before the code needs no change when the code comes.`,
|
|
51
|
+
`A6. A refusal: when the product answers outside 2xx and the caller did not`,
|
|
52
|
+
` pass allowRefusal: true, throw refused(name, method, path, res). With`,
|
|
53
|
+
` allowRefusal the result is returned as it is, for a test of a refusal.`,
|
|
54
|
+
`A7. An action the doc describes that the product does not offer yet (no route`,
|
|
55
|
+
` or handler for it, or no code at all): write it all the same, with its full`,
|
|
56
|
+
` signature, comment and named results from the doc, and a body of`,
|
|
57
|
+
` return waiting(name, '<what was looked for>')(). Never invent a route.`,
|
|
58
|
+
` When the code comes, only that body is written; the signature stays.`,
|
|
59
|
+
`A10. A value an earlier screen would give (a service, a slot) that the`,
|
|
60
|
+
` product does not list: throw refused() for that earlier request, naming`,
|
|
61
|
+
` what was looked for. Never send a placeholder (id 0, an empty value).`,
|
|
62
|
+
`A8. refused and waiting come from`,
|
|
63
|
+
` require('@activate-core/test-forge/reporters/interface.cjs').`,
|
|
64
|
+
`A9. Above each function, one comment line: what it does, the route(s) it`,
|
|
65
|
+
` calls (or "not in the product yet"), and the named results it returns.`,
|
|
66
|
+
` The test writer sees only that line and the signature.`,
|
|
67
|
+
].join('\n');
|
|
68
|
+
const CONTRACT_HASH = createHash('sha1').update(ACTIONS_CONTRACT).digest('hex').slice(0, 12);
|
|
69
|
+
const STAMP_RE = /^\/\/ testforge-actions-contract: (\w+)(?: journeys: (\S*))?\n/;
|
|
70
|
+
export const ACTIONS_FAILED_PATH = join(STATE_DIR, 'actions-failed.json');
|
|
71
|
+
const hashOf = (text) => createHash('sha1').update(text).digest('hex').slice(0, 12);
|
|
72
|
+
/** The actions module sits next to the testkit, with its extension. */
|
|
73
|
+
export function actionsPath(harnessPath) {
|
|
74
|
+
return join(dirname(harnessPath), `actions${/\.ts$/.test(harnessPath) ? '.ts' : '.js'}`);
|
|
75
|
+
}
|
|
76
|
+
/** The journeys (id → doc hash) a module was built for, and whether its contract is current. */
|
|
77
|
+
export function readStamp(source) {
|
|
78
|
+
const m = STAMP_RE.exec(source);
|
|
79
|
+
if (!m)
|
|
80
|
+
return { current: false, journeys: {} };
|
|
81
|
+
const journeys = {};
|
|
82
|
+
for (const pair of (m[2] ?? '').split(',').filter(Boolean)) {
|
|
83
|
+
const [id, h] = pair.split('=');
|
|
84
|
+
if (id && h)
|
|
85
|
+
journeys[id] = h;
|
|
86
|
+
}
|
|
87
|
+
return { current: m[1] === CONTRACT_HASH, journeys };
|
|
88
|
+
}
|
|
89
|
+
/** The module with a stamp recording its contract and the journeys (with their doc) it serves. */
|
|
90
|
+
export function stampActions(source, journeys) {
|
|
91
|
+
const list = Object.entries(journeys).sort().map(([id, h]) => `${id}=${h}`).join(',');
|
|
92
|
+
return `// testforge-actions-contract: ${CONTRACT_HASH} journeys: ${list}\n${source.replace(new RegExp(STAMP_RE.source, 'gm'), '')}`;
|
|
93
|
+
}
|
|
94
|
+
/** Actions a run found refused: { name: what the product answered }. */
|
|
95
|
+
export function readActionFailures(path = ACTIONS_FAILED_PATH) {
|
|
96
|
+
try {
|
|
97
|
+
return JSON.parse(readFileSync(path, 'utf-8'));
|
|
98
|
+
}
|
|
99
|
+
catch {
|
|
100
|
+
return {};
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
export function clearActionFailures(path = ACTIONS_FAILED_PATH) {
|
|
104
|
+
rmSync(path, { force: true });
|
|
105
|
+
}
|
|
106
|
+
/** Exported names of a CommonJS or ESM module. */
|
|
107
|
+
export function exportedNames(src) {
|
|
108
|
+
const names = new Set();
|
|
109
|
+
const cjs = /module\.exports\s*=\s*\{([\s\S]*?)\}/.exec(src);
|
|
110
|
+
if (cjs)
|
|
111
|
+
for (const n of cjs[1].split(',')) {
|
|
112
|
+
const m = /^\s*(\w+)/.exec(n);
|
|
113
|
+
if (m)
|
|
114
|
+
names.add(m[1]);
|
|
115
|
+
}
|
|
116
|
+
for (const m of src.matchAll(/^export\s+(?:async\s+)?(?:function\*?|const|let|class)\s+(\w+)/gm))
|
|
117
|
+
names.add(m[1]);
|
|
118
|
+
return [...names];
|
|
119
|
+
}
|
|
120
|
+
/**
|
|
121
|
+
* Why the module must be built now for this journey, or undefined when it is
|
|
122
|
+
* current: a missing module, an older contract, a doc it has not seen in this
|
|
123
|
+
* version, or one of its actions refused in the last run.
|
|
124
|
+
*/
|
|
125
|
+
export function actionsStale(path, journeyId, doc) {
|
|
126
|
+
if (!existsSync(path))
|
|
127
|
+
return 'no actions module yet';
|
|
128
|
+
const src = readFileSync(path, 'utf-8');
|
|
129
|
+
const stamp = readStamp(src);
|
|
130
|
+
if (!stamp.current)
|
|
131
|
+
return 'its contract predates this TestForge version';
|
|
132
|
+
if (stamp.journeys[journeyId] !== hashOf(doc))
|
|
133
|
+
return `the ${journeyId} doc is new to it or changed`;
|
|
134
|
+
const names = exportedNames(src);
|
|
135
|
+
const failed = Object.keys(readActionFailures()).filter((n) => names.includes(n));
|
|
136
|
+
if (failed.length)
|
|
137
|
+
return `the last run had ${failed.join(', ')} refused`;
|
|
138
|
+
return undefined;
|
|
139
|
+
}
|
|
140
|
+
export function journeysWith(path, journeyId, doc) {
|
|
141
|
+
const before = existsSync(path) ? readStamp(readFileSync(path, 'utf-8')).journeys : {};
|
|
142
|
+
return { ...before, [journeyId]: hashOf(doc) };
|
|
143
|
+
}
|
|
144
|
+
export function buildActionsPrompt(path, harnessPath, docPaths) {
|
|
145
|
+
let previous = '';
|
|
146
|
+
try {
|
|
147
|
+
previous = readFileSync(path, 'utf-8');
|
|
148
|
+
}
|
|
149
|
+
catch { /* first build */ }
|
|
150
|
+
const failures = readActionFailures();
|
|
151
|
+
const failed = Object.entries(failures).filter(([n]) => exportedNames(previous).includes(n));
|
|
152
|
+
const related = (loadConfig().relatedDirs ?? []).map((d) => resolve(d));
|
|
153
|
+
return [
|
|
154
|
+
`You are a senior SDET writing the actions module of a test suite. You are inside`,
|
|
155
|
+
`the repository with read-only tools (Read, Grep, Glob, LS). Read before you`,
|
|
156
|
+
`write: every action is a claim about how the product is driven, and one built`,
|
|
157
|
+
`on a guess makes every test that uses it fail as if the product were broken.`,
|
|
158
|
+
``,
|
|
159
|
+
`The module will be saved at exactly ${path}. The testkit is ${harnessPath}:`,
|
|
160
|
+
`read it, and require it with a path relative to the module.`,
|
|
161
|
+
`The journey doc(s) to cover: ${docPaths.join(', ')}. Read their steps.`,
|
|
162
|
+
...(related.length
|
|
163
|
+
? [`The product's other repos (its web or mobile client among them) are readable at: ${related.join(', ')}.`]
|
|
164
|
+
: [`No other repo of the product is configured ("relatedDirs" in testforge.config.json): read the client code`, `this repo has, if any, and the handlers.`]),
|
|
165
|
+
``,
|
|
166
|
+
ACTIONS_CONTRACT,
|
|
167
|
+
``,
|
|
168
|
+
...(previous
|
|
169
|
+
? [
|
|
170
|
+
`<existing_actions>`,
|
|
171
|
+
previous,
|
|
172
|
+
`</existing_actions>`,
|
|
173
|
+
`Tests already call these. Keep every export with the same name, parameters`,
|
|
174
|
+
`and named results, and add the actions the steps need. An action whose body is`,
|
|
175
|
+
`waiting(...) gets a real body where the code now has its handler.`,
|
|
176
|
+
``,
|
|
177
|
+
]
|
|
178
|
+
: []),
|
|
179
|
+
...(failed.length
|
|
180
|
+
? [
|
|
181
|
+
`The last run had these actions refused by the product. Read the handler and`,
|
|
182
|
+
`the client again and fix the request; if the action is right and the product`,
|
|
183
|
+
`is at fault, keep it as it is:`,
|
|
184
|
+
...failed.map(([n, why]) => `- ${n}: ${why}`),
|
|
185
|
+
``,
|
|
186
|
+
]
|
|
187
|
+
: []),
|
|
188
|
+
`Output only the CommonJS code: no prose and no fences, and the first`,
|
|
189
|
+
`characters must be code. Do not write any file.`,
|
|
190
|
+
].join('\n');
|
|
191
|
+
}
|
|
192
|
+
export function generateActionsAsync(path, harnessPath, docPaths) {
|
|
193
|
+
const config = loadConfig();
|
|
194
|
+
const { program, args } = withReadOnlyTools(splitCommand(config.aiCommand));
|
|
195
|
+
return runWithStdin(program, args, buildActionsPrompt(path, harnessPath, docPaths), config.aiTimeoutMs ?? 300_000).then(cleanOutput);
|
|
196
|
+
}
|
|
197
|
+
/** Does the module still have actions the product did not offer when it was built? */
|
|
198
|
+
export function hasWaitingActions(source) {
|
|
199
|
+
return /\bwaiting\(\s*['"`]/.test(source);
|
|
200
|
+
}
|
|
@@ -966,3 +966,44 @@ export function cleanOutput(text) {
|
|
|
966
966
|
.join('\n')
|
|
967
967
|
.trim();
|
|
968
968
|
}
|
|
969
|
+
/**
|
|
970
|
+
* Write a file's setup again (its hooks and helpers), not its tests: the
|
|
971
|
+
* setup waited on names the code facts answer, or built data through routes
|
|
972
|
+
* the facts do not show. One small call; the tests stay as they are.
|
|
973
|
+
* Returns the new blocks in order, or null when the answer does not have one
|
|
974
|
+
* per block.
|
|
975
|
+
*/
|
|
976
|
+
export async function rewriteSetupWithAI(input) {
|
|
977
|
+
const mark = (n) => `// ==== SETUP ${n} ====`;
|
|
978
|
+
const prompt = [
|
|
979
|
+
`You are a senior SDET rewriting the setup of one Jest test file: its hooks and helpers, not its tests.`,
|
|
980
|
+
`The tests below this setup stay exactly as they are and use what it sets.`,
|
|
981
|
+
``,
|
|
982
|
+
`Rules`,
|
|
983
|
+
`- Make the data the setup needs with kit.insert(table, row) and kit.update(table, match, values),`,
|
|
984
|
+
` using the tables and columns <code_facts> shows. kit.insert returns the new row's id itself.`,
|
|
985
|
+
`- Send a request only through a route <code_facts> shows, with its fields as written there.`,
|
|
986
|
+
`- Never call need(), and never use a constant that is not in <file_top> or set by this setup.`,
|
|
987
|
+
`- Keep each block what it is (the same hook, or a helper with the same name and parameters), and`,
|
|
988
|
+
` keep every variable it sets (ctx.*, outer let variables, maps) with the same name and meaning.`,
|
|
989
|
+
`- No comments except what was there.`,
|
|
990
|
+
``,
|
|
991
|
+
`Output: for each block, the line ${mark(1)} (with its number) and then its code. Nothing else.`,
|
|
992
|
+
``,
|
|
993
|
+
`<testkit>`, input.helpers, `</testkit>`,
|
|
994
|
+
``,
|
|
995
|
+
...(input.facts ? [`<code_facts>`, input.facts, `</code_facts>`, ``] : []),
|
|
996
|
+
`<file_top path="${input.filePath}">`, input.top, `</file_top>`,
|
|
997
|
+
``,
|
|
998
|
+
...input.blocks.flatMap((b, i) => [mark(i + 1), b.text, ``]),
|
|
999
|
+
].join('\n');
|
|
1000
|
+
const config = loadConfig();
|
|
1001
|
+
const { program, args } = withNoTools(splitCommand(config.aiCommand), 'Answer only with the setup blocks asked for.');
|
|
1002
|
+
// Split on the markers first: the comment cleaner would take them out.
|
|
1003
|
+
const out = await runWithStdin(program, args, prompt, config.aiTimeoutMs);
|
|
1004
|
+
const parts = out.split(/^\/\/ ==== SETUP (\d+) ====[ \t]*$/m);
|
|
1005
|
+
const got = [];
|
|
1006
|
+
for (let i = 1; i < parts.length; i += 2)
|
|
1007
|
+
got[Number(parts[i]) - 1] = parts[i + 1].replace(/^\s*```\w*\s*$/gm, '').trim();
|
|
1008
|
+
return got.length === input.blocks.length && got.every((g) => g && g.length > 0) ? got : null;
|
|
1009
|
+
}
|
|
@@ -82,6 +82,7 @@ const STAGES = [
|
|
|
82
82
|
[/^=== REPAIR ===$/m, 'repair'],
|
|
83
83
|
[/^You are a senior QA engineer\. Generate a complete/, 'write'],
|
|
84
84
|
[/^You are a senior QA engineer writing one .* test file/, 'write'],
|
|
85
|
+
[/^You are a senior SDET rewriting the setup/, 'setup'],
|
|
85
86
|
[/^You are a senior test architect/, 'plan'],
|
|
86
87
|
[/^You are a senior engineer establishing the MECHANICS/, 'grounding'],
|
|
87
88
|
[/^You review ONE claimed product defect/, 'defect-review'],
|
|
@@ -52,6 +52,34 @@ export function onlyNewTests(answer, file, rules) {
|
|
|
52
52
|
.map((b) => answer.slice(b.start, b.end).replace(/^\s*\n/, ''))
|
|
53
53
|
.join('\n\n');
|
|
54
54
|
}
|
|
55
|
+
/**
|
|
56
|
+
* A kept file's earlier tests for the rules that lost their only test: the
|
|
57
|
+
* ones taken out to be written again whose new tests did not come back, or
|
|
58
|
+
* were refused. They go back with the need() lines they use. A waiting test
|
|
59
|
+
* beats no test.
|
|
60
|
+
*/
|
|
61
|
+
export function putBack(out, previous, taken, rules) {
|
|
62
|
+
const now = findTestBlocks(out);
|
|
63
|
+
const have = new Set(now.flatMap((b) => b.rules));
|
|
64
|
+
const titles = new Set(now.map((b) => b.title));
|
|
65
|
+
const back = taken.filter((b) => !titles.has(b.title) && b.rules.some((r) => rules.includes(r) && !have.has(r)));
|
|
66
|
+
if (!back.length)
|
|
67
|
+
return { content: out, rules: [] };
|
|
68
|
+
const texts = back.map((b) => previous.slice(b.start, b.end).replace(/^\s*\n/, ''));
|
|
69
|
+
let content = appendTests(out, texts.join('\n\n'));
|
|
70
|
+
const declared = new Set([...content.matchAll(/^[ \t]*(?:const|let)\s+([A-Za-z_$][\w$]*)\s*=/gm)].map((m) => m[1]));
|
|
71
|
+
const lines = previous.split('\n').filter((l) => {
|
|
72
|
+
const m = /^[ \t]*(?:const|let)\s+([A-Za-z_$][\w$]*)\s*=\s*need\(/.exec(l);
|
|
73
|
+
return m && !declared.has(m[1]) && texts.some((t) => new RegExp(`\\b${m[1].replace(/\$/g, '\\$')}\\b`).test(t));
|
|
74
|
+
});
|
|
75
|
+
if (lines.length) {
|
|
76
|
+
// After the last need() or require() line at the top, where the file keeps them.
|
|
77
|
+
const top = [...content.matchAll(/^(?:const|let)\s+[^\n]*\b(?:need|require)\([^\n]*\n/gm)].pop();
|
|
78
|
+
const at = top ? (top.index ?? 0) + top[0].length : 0;
|
|
79
|
+
content = content.slice(0, at) + lines.join('\n') + '\n' + content.slice(at);
|
|
80
|
+
}
|
|
81
|
+
return { content, rules: [...new Set(back.flatMap((b) => b.rules.filter((r) => rules.includes(r) && !have.has(r))))] };
|
|
82
|
+
}
|
|
55
83
|
/** Is the file complete as it is? */
|
|
56
84
|
export function fileIsComplete(content, rules, steps, knownTexts) {
|
|
57
85
|
const { cov, problems } = review(content, rules, steps, knownTexts);
|
|
@@ -102,8 +130,25 @@ export async function completeFile(content, ctx) {
|
|
|
102
130
|
calls = 1;
|
|
103
131
|
try {
|
|
104
132
|
const tests = await ctx.write(ctx.journey, ctx.path, head, title, '', ctx.harnessPath, ctx.transport, factsSlice(ctx.facts, slice.controls), { journey: slice.text, helpers: ctx.helpers, rules: left }, describeProblems(firstDrop, first.cov));
|
|
105
|
-
const
|
|
106
|
-
|
|
133
|
+
const fix = ctx.fix ?? ((c) => c);
|
|
134
|
+
let candidate = fix(appendTests(out, onlyNewTests(tests, out, left)));
|
|
135
|
+
let refused = ctx.check(candidate);
|
|
136
|
+
if (refused) {
|
|
137
|
+
// One bad test refuses the whole answer: keep the tests that pass the
|
|
138
|
+
// checks on their own, so the good ones are not thrown away with it.
|
|
139
|
+
const answer = onlyNewTests(tests, out, left);
|
|
140
|
+
let kept = out;
|
|
141
|
+
for (const b of findTestBlocks(answer)) {
|
|
142
|
+
const next = fix(appendTests(kept, answer.slice(b.start, b.end)));
|
|
143
|
+
if (!ctx.check(next))
|
|
144
|
+
kept = next;
|
|
145
|
+
}
|
|
146
|
+
if (kept !== out) {
|
|
147
|
+
notes.push(`some tests written again for ${left.join(', ')} were refused (${refused.split('\n')[0].slice(0, 160)}); the others were kept`);
|
|
148
|
+
candidate = kept;
|
|
149
|
+
refused = null;
|
|
150
|
+
}
|
|
151
|
+
}
|
|
107
152
|
if (refused) {
|
|
108
153
|
notes.push(`tests written again for ${left.join(', ')} were refused (${refused.split('\n')[0]})`);
|
|
109
154
|
failedRedo = true;
|
|
@@ -124,6 +169,21 @@ export async function completeFile(content, ctx) {
|
|
|
124
169
|
}
|
|
125
170
|
}
|
|
126
171
|
}
|
|
172
|
+
// Some new tests came back and some did not (refused, or taken out by the
|
|
173
|
+
// second check): the rules left with no test get their earlier test back.
|
|
174
|
+
if (ctx.kept && !failedRedo && firstDrop.length) {
|
|
175
|
+
const back = putBack(out, content, firstDrop.map((p) => p.block), ctx.rules);
|
|
176
|
+
if (back.rules.length && !ctx.check(back.content)) {
|
|
177
|
+
notes.push(`put back the earlier test(s) for ${back.rules.join(', ')}: their new tests did not pass the checks`);
|
|
178
|
+
out = back.content;
|
|
179
|
+
for (const p of firstDrop)
|
|
180
|
+
if (p.rules.some((r) => back.rules.includes(r)))
|
|
181
|
+
dropped.delete(p.title);
|
|
182
|
+
}
|
|
183
|
+
else if (back.rules.length) {
|
|
184
|
+
failedRedo = true; // the earlier tests do not fit back in: keep the file as it was
|
|
185
|
+
}
|
|
186
|
+
}
|
|
127
187
|
// A kept file whose tests were taken out to be written again, and the new
|
|
128
188
|
// ones did not come: it stays as it was. A waiting test beats no test.
|
|
129
189
|
if (ctx.kept && failedRedo && redo.length + firstDrop.length > 0) {
|