@activate-core/test-forge 0.1.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +58 -2
- package/dist/cli/index.js +107 -62
- package/dist/core/ai-generator.js +79 -26
- package/dist/core/config.js +80 -5
- package/dist/core/discover.js +45 -7
- package/dist/core/env-template.js +91 -5
- package/dist/core/index.js +1 -1
- package/dist/core/init.js +3 -3
- package/dist/core/map.js +3 -3
- package/dist/core/prompt.js +70 -0
- package/dist/core/reference.js +162 -0
- package/dist/core/run-generate.js +98 -17
- package/dist/core/seed-generator.js +23 -10
- package/dist/core/setup.js +5 -4
- package/dist/core/source-of-truth.js +84 -0
- package/dist/core/update.js +5 -3
- package/dist/core/validate.js +82 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -40,7 +40,7 @@ testforge update # re-check and refresh only what changed
|
|
|
40
40
|
| Command | What it does |
|
|
41
41
|
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
42
42
|
| `testforge setup` | One-time setup: runs init, map, and discover in order, with progress. |
|
|
43
|
-
| `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated.
|
|
43
|
+
| `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. |
|
|
44
44
|
| `testforge update` | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |
|
|
45
45
|
|
|
46
46
|
### Advanced (run a single step)
|
|
@@ -73,6 +73,57 @@ generates a seed file that provisions the test data each test needs.
|
|
|
73
73
|
**update** hashes your source code and compares it to the last run. No
|
|
74
74
|
change means no work and no AI calls; a change triggers a refresh.
|
|
75
75
|
|
|
76
|
+
## Source of truth
|
|
77
|
+
|
|
78
|
+
The journey docs in `docs/` are the source of truth for the tests —
|
|
79
|
+
`generate` reads the docs and nothing else. So where the docs come from
|
|
80
|
+
decides everything.
|
|
81
|
+
|
|
82
|
+
By default the docs are derived from your **code** (the AI reads the
|
|
83
|
+
project, writes `testforge.md`, then drafts the docs from it). That is
|
|
84
|
+
fast, but it inherits whatever the code does: if a flow is buggy or a
|
|
85
|
+
scenario is mishandled, the AI faithfully documents the wrong behavior,
|
|
86
|
+
and the tests then lock that wrong behavior in.
|
|
87
|
+
|
|
88
|
+
So when the docs are drafted — during `testforge setup` (or `testforge
|
|
89
|
+
discover`) — testforge asks what they should be based on:
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
What should the journey docs be based on?
|
|
93
|
+
[1] testforge.md (what the AI derived from your code) (default)
|
|
94
|
+
[2] An external reference (a file/PDF in scope, a website, or a Notion link)
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Choose **[2]** to give testforge an authoritative reference of the
|
|
98
|
+
_expected_ behavior — a spec or requirements doc in the repo, a PDF, a
|
|
99
|
+
website, or a Notion page. testforge then:
|
|
100
|
+
|
|
101
|
+
1. reads the reference and compares it against `testforge.md`,
|
|
102
|
+
2. writes a full comparison to **`testforge.diff.md`** and prints a short
|
|
103
|
+
summary of the differences,
|
|
104
|
+
3. asks you to confirm, and on confirmation **drafts the docs from the
|
|
105
|
+
reference** — expected behavior comes from the reference, while routes,
|
|
106
|
+
fields, models, and env var names come from the code/map,
|
|
107
|
+
4. `generate` then turns those docs into tests as usual.
|
|
108
|
+
|
|
109
|
+
The reference becomes the source of truth — expected behavior from a human
|
|
110
|
+
or client, not just what the code happens to do. The choice is remembered
|
|
111
|
+
in `.testforge/source.json`, so a later `testforge update` refreshes the
|
|
112
|
+
docs from the same source without asking again. Your review of
|
|
113
|
+
`testforge.diff.md` stays as an audit of what differed and why.
|
|
114
|
+
|
|
115
|
+
**Non-interactive use (CI, scripts):**
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
testforge setup --source map # base the docs on testforge.md, no prompt
|
|
119
|
+
testforge setup --reference ./spec.md # base the docs on a reference, no prompts
|
|
120
|
+
testforge setup --reference https://… # a URL or Notion link works too
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
The same flags work on `testforge discover`. With no flags and no terminal
|
|
124
|
+
attached, testforge does not block — it falls back to `testforge.md` and
|
|
125
|
+
prints a note.
|
|
126
|
+
|
|
76
127
|
## Running the generated tests
|
|
77
128
|
|
|
78
129
|
testforge generates the tests and seed files — you run them with your
|
|
@@ -159,7 +210,12 @@ The test runner produces its own pass/fail report.
|
|
|
159
210
|
|
|
160
211
|
- **aiCommand** — the CLI used to call the AI. Any tool that reads a
|
|
161
212
|
prompt from stdin and prints the reply works, so testforge is not tied
|
|
162
|
-
to one provider.
|
|
213
|
+
to one provider. testforge appends the right flags per phase: the
|
|
214
|
+
setup phases (init/map/discover) run with edit permission because the
|
|
215
|
+
model writes files, while the generation phases (tests/seeds) run with
|
|
216
|
+
no tools (`--allowedTools ""` for Claude) so the model just prints code
|
|
217
|
+
instead of trying to create files — which would otherwise leak "the
|
|
218
|
+
write wasn't approved…" prose into the output.
|
|
163
219
|
- **testOutput** — where tests go, per area. Areas are not fixed — use
|
|
164
220
|
whatever your project has (`api`, `web`, `ops`, `lambda`, ...).
|
|
165
221
|
- **framework** — which test framework each area uses.
|
package/dist/cli/index.js
CHANGED
|
@@ -3,9 +3,16 @@
|
|
|
3
3
|
* testforge — command-line entry point.
|
|
4
4
|
*
|
|
5
5
|
* Commands:
|
|
6
|
-
* testforge
|
|
7
|
-
*
|
|
8
|
-
*
|
|
6
|
+
* testforge setup init + map + discover. discover asks
|
|
7
|
+
* whether the docs come from testforge.md
|
|
8
|
+
* or an external reference (the source of
|
|
9
|
+
* truth for every test).
|
|
10
|
+
* testforge setup --reference <x> Base the docs on reference <x> (a file
|
|
11
|
+
* path, URL, or Notion link) — no prompt.
|
|
12
|
+
* testforge setup --source map Base the docs on testforge.md — no prompt.
|
|
13
|
+
* testforge generate Generate tests from the journey docs.
|
|
14
|
+
* testforge generate --basic Use the built-in basic generator only.
|
|
15
|
+
* testforge generate --force Regenerate everything, ignoring the cache.
|
|
9
16
|
*/
|
|
10
17
|
import { VERSION } from '../core/index.js';
|
|
11
18
|
import { runGenerate } from '../core/run-generate.js';
|
|
@@ -18,74 +25,112 @@ import { runEnvTemplate } from '../core/env-template.js';
|
|
|
18
25
|
const command = process.argv[2];
|
|
19
26
|
const useBasic = process.argv.includes('--basic');
|
|
20
27
|
const force = process.argv.includes('--force');
|
|
21
|
-
|
|
22
|
-
|
|
28
|
+
/** Read the value that follows a `--flag <value>` option, if present. */
|
|
29
|
+
function flagValue(name) {
|
|
30
|
+
const i = process.argv.indexOf(name);
|
|
31
|
+
if (i === -1)
|
|
32
|
+
return undefined;
|
|
33
|
+
const value = process.argv[i + 1];
|
|
34
|
+
if (value === undefined || value.startsWith('--'))
|
|
35
|
+
return undefined;
|
|
36
|
+
return value;
|
|
23
37
|
}
|
|
24
|
-
|
|
25
|
-
|
|
38
|
+
/** Source-of-truth options shared by setup and discover. */
|
|
39
|
+
function sourceOptions() {
|
|
40
|
+
return { reference: flagValue('--reference'), source: flagValue('--source') };
|
|
26
41
|
}
|
|
27
|
-
|
|
28
|
-
console.log('Inspecting your project to set up testforge...\n');
|
|
29
|
-
const output = runInit();
|
|
30
|
-
console.log(output);
|
|
31
|
-
console.log('\ntestforge.config.json written. Review it, then run `testforge map`.');
|
|
32
|
-
}
|
|
33
|
-
else if (command === 'map') {
|
|
34
|
-
console.log('Building a high-level map of your project with AI...');
|
|
35
|
-
console.log('This scans your source code and writes testforge.md\n');
|
|
36
|
-
const output = runMap('.');
|
|
37
|
-
console.log(output);
|
|
38
|
-
console.log('\nProject map written to testforge.md');
|
|
39
|
-
}
|
|
40
|
-
else if (command === 'discover') {
|
|
41
|
-
console.log('Scanning your project with AI to draft journey files...');
|
|
42
|
-
console.log('This reads your source code and writes drafts into docs/.\n');
|
|
43
|
-
const output = runDiscover('.', 'docs');
|
|
44
|
-
console.log(output);
|
|
45
|
-
console.log('\nDraft journeys written to docs/. Please review and edit them,');
|
|
46
|
-
console.log('then run `testforge generate`.');
|
|
47
|
-
}
|
|
48
|
-
else if (command === 'env') {
|
|
49
|
-
runEnvTemplate('docs', '.env.testing.example');
|
|
50
|
-
}
|
|
51
|
-
else if (command === 'generate') {
|
|
52
|
-
const useAI = !useBasic;
|
|
53
|
-
console.log(useAI
|
|
54
|
-
? 'Generating tests with AI for changed journeys...\n'
|
|
55
|
-
: 'Generating tests with the basic generator...\n');
|
|
56
|
-
const results = runGenerate('docs', useAI, force);
|
|
57
|
-
if (results.length === 0) {
|
|
58
|
-
console.log('No journey files found in docs/. Add a .md file and try again.');
|
|
59
|
-
}
|
|
60
|
-
else {
|
|
61
|
-
for (const r of results) {
|
|
62
|
-
if (r.status === 'skipped') {
|
|
63
|
-
console.log(` [skip] ${r.journeyName} (unchanged)`);
|
|
64
|
-
}
|
|
65
|
-
else {
|
|
66
|
-
const tag = r.status === 'ai' ? '[AI]' : '[basic]';
|
|
67
|
-
console.log(` ${tag} ${r.journeyName}`);
|
|
68
|
-
console.log(` test: ${r.testPath}`);
|
|
69
|
-
if (r.status === 'ai')
|
|
70
|
-
console.log(` seed: ${r.seedPath}`);
|
|
71
|
-
}
|
|
72
|
-
}
|
|
73
|
-
const generated = results.filter((r) => r.status !== 'skipped').length;
|
|
74
|
-
const skipped = results.filter((r) => r.status === 'skipped').length;
|
|
75
|
-
console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged).`);
|
|
76
|
-
}
|
|
77
|
-
}
|
|
78
|
-
else {
|
|
42
|
+
function printUsage() {
|
|
79
43
|
console.log(`testforge v${VERSION}`);
|
|
80
44
|
console.log('');
|
|
81
45
|
console.log('Usage:');
|
|
82
|
-
console.log(' testforge
|
|
46
|
+
console.log(' testforge setup Run init + map + discover in order');
|
|
47
|
+
console.log(' testforge setup --reference <file|url> Base the docs on a reference');
|
|
48
|
+
console.log(' testforge setup --source map Base the docs on testforge.md');
|
|
49
|
+
console.log(' testforge generate Generate tests from the journey docs');
|
|
83
50
|
console.log(' testforge generate --basic Use the basic generator only');
|
|
84
51
|
console.log(' testforge generate --force Regenerate everything');
|
|
85
|
-
console.log(' testforge discover
|
|
52
|
+
console.log(' testforge discover Draft journey docs (asks: map or reference)');
|
|
86
53
|
console.log(' testforge map Scan project, write journey + data map');
|
|
87
54
|
console.log(' testforge init Inspect project and write config');
|
|
88
|
-
console.log(' testforge setup Run init + map + discover in order');
|
|
89
55
|
console.log(' testforge update Re-check and update only what changed');
|
|
90
56
|
console.log(' testforge env Write .env.testing.example from docs');
|
|
91
57
|
}
|
|
58
|
+
async function main() {
|
|
59
|
+
if (command === 'update') {
|
|
60
|
+
await runUpdate('.');
|
|
61
|
+
}
|
|
62
|
+
else if (command === 'setup') {
|
|
63
|
+
await runSetup('.', sourceOptions());
|
|
64
|
+
}
|
|
65
|
+
else if (command === 'init') {
|
|
66
|
+
console.log('Inspecting your project to set up testforge...\n');
|
|
67
|
+
const output = runInit();
|
|
68
|
+
console.log(output);
|
|
69
|
+
console.log('\ntestforge.config.json written. Review it, then run `testforge map`.');
|
|
70
|
+
}
|
|
71
|
+
else if (command === 'map') {
|
|
72
|
+
console.log('Building a high-level map of your project with AI...');
|
|
73
|
+
console.log('This scans your source code and writes testforge.md\n');
|
|
74
|
+
const output = runMap('.');
|
|
75
|
+
console.log(output);
|
|
76
|
+
console.log('\nProject map written to testforge.md');
|
|
77
|
+
}
|
|
78
|
+
else if (command === 'discover') {
|
|
79
|
+
console.log('Drafting journey files with AI...');
|
|
80
|
+
console.log('This writes drafts into docs/ from your chosen source.\n');
|
|
81
|
+
const output = await runDiscover('.', 'docs', sourceOptions());
|
|
82
|
+
console.log(output);
|
|
83
|
+
console.log('\nDraft journeys written to docs/. Please review and edit them,');
|
|
84
|
+
console.log('then run `testforge generate`.');
|
|
85
|
+
}
|
|
86
|
+
else if (command === 'env') {
|
|
87
|
+
runEnvTemplate('docs', '.env.testing.example');
|
|
88
|
+
}
|
|
89
|
+
else if (command === 'generate') {
|
|
90
|
+
const useAI = !useBasic;
|
|
91
|
+
console.log(useAI
|
|
92
|
+
? 'Generating tests with AI for changed journeys...\n'
|
|
93
|
+
: 'Generating tests with the basic generator...\n');
|
|
94
|
+
const results = runGenerate('docs', useAI, force);
|
|
95
|
+
if (results.length === 0) {
|
|
96
|
+
console.log('No journey files found in docs/. Add a .md file and try again.');
|
|
97
|
+
}
|
|
98
|
+
else {
|
|
99
|
+
for (const r of results) {
|
|
100
|
+
if (r.status === 'skipped') {
|
|
101
|
+
console.log(` [skip] ${r.journeyName} (unchanged)`);
|
|
102
|
+
}
|
|
103
|
+
else {
|
|
104
|
+
const tag = r.status === 'ai' ? '[AI]' : '[basic]';
|
|
105
|
+
console.log(` ${tag} ${r.journeyName}`);
|
|
106
|
+
console.log(` test: ${r.testPath}`);
|
|
107
|
+
if (r.status === 'ai') {
|
|
108
|
+
if (r.seedStatus === 'ok') {
|
|
109
|
+
console.log(` seed: ${r.seedPath}`);
|
|
110
|
+
}
|
|
111
|
+
else if (r.seedStatus === 'failed') {
|
|
112
|
+
console.log(` seed: (not written — generation failed)`);
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
for (const w of r.warnings ?? []) {
|
|
117
|
+
console.log(` ! ${w}`);
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
const generated = results.filter((r) => r.status !== 'skipped').length;
|
|
121
|
+
const skipped = results.filter((r) => r.status === 'skipped').length;
|
|
122
|
+
const seedFailures = results.filter((r) => r.seedStatus === 'failed').length;
|
|
123
|
+
console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged).`);
|
|
124
|
+
if (seedFailures > 0) {
|
|
125
|
+
console.log(`Warning: ${seedFailures} seed file(s) could not be generated — see notes above.`);
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
else {
|
|
130
|
+
printUsage();
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
main().catch((err) => {
|
|
134
|
+
console.error(err instanceof Error ? err.message : String(err));
|
|
135
|
+
process.exit(1);
|
|
136
|
+
});
|
|
@@ -12,10 +12,10 @@
|
|
|
12
12
|
*/
|
|
13
13
|
import { execFileSync } from 'node:child_process';
|
|
14
14
|
import { readFileSync, existsSync } from 'node:fs';
|
|
15
|
-
import { loadConfig, splitCommand } from './config.js';
|
|
15
|
+
import { loadConfig, splitCommand, withTextOnlyTools } from './config.js';
|
|
16
16
|
const MAP_PATH = 'testforge.md';
|
|
17
17
|
/** Build the prompt we send to the AI for one journey. */
|
|
18
|
-
export function buildPrompt(journey, map) {
|
|
18
|
+
export function buildPrompt(journey, map, testPath) {
|
|
19
19
|
const config = loadConfig();
|
|
20
20
|
const fw = config.framework?.[journey.type];
|
|
21
21
|
const isApi = journey.type === 'api' || fw === 'jest' || fw === 'supertest';
|
|
@@ -42,6 +42,16 @@ export function buildPrompt(journey, map) {
|
|
|
42
42
|
`useful). For payment/webhook journeys, test idempotency, amount/`,
|
|
43
43
|
`price tampering, and signature/verification where relevant.`,
|
|
44
44
|
``,
|
|
45
|
+
`IMPORTANT — do not presume the app is vulnerable. For every security`,
|
|
46
|
+
`probe, assert the SECURE expectation (e.g. auth bypass returns 401/403,`,
|
|
47
|
+
`a tampered amount is rejected, a duplicate charge is idempotent). Only`,
|
|
48
|
+
`assert that a vulnerability exists when the journey doc EXPLICITLY`,
|
|
49
|
+
`documents it as a known/accepted behavior. If you cannot tell from the`,
|
|
50
|
+
`journey or map whether a defense exists, write the check as`,
|
|
51
|
+
`test.todo(...) or it.skip(...) with a "// TODO:" note — never assume a`,
|
|
52
|
+
`guard is missing and produce a failing test that falsely brands the`,
|
|
53
|
+
`product as vulnerable.`,
|
|
54
|
+
``,
|
|
45
55
|
isApi
|
|
46
56
|
? [
|
|
47
57
|
`Because this is an API journey, write Jest tests that call the`,
|
|
@@ -60,6 +70,17 @@ export function buildPrompt(journey, map) {
|
|
|
60
70
|
`they are real.`,
|
|
61
71
|
].join('\n'),
|
|
62
72
|
``,
|
|
73
|
+
...(testPath
|
|
74
|
+
? [
|
|
75
|
+
`This test file will be saved at EXACTLY this path:`,
|
|
76
|
+
` ${testPath}`,
|
|
77
|
+
`Compute every require()/import path (to app source, helpers, the`,
|
|
78
|
+
`seed file, etc.) RELATIVE TO THAT EXACT location — count the`,
|
|
79
|
+
`directory depth from that path, do not assume a conventional`,
|
|
80
|
+
`depth. Getting the relative depth wrong breaks the whole file.`,
|
|
81
|
+
``,
|
|
82
|
+
]
|
|
83
|
+
: []),
|
|
63
84
|
`This test relies on seeded test data (a separate seed file provisions`,
|
|
64
85
|
`the database before tests run). Assume the seed has created the`,
|
|
65
86
|
`records described in the journey's Data section, using stable test`,
|
|
@@ -96,10 +117,12 @@ export function buildPrompt(journey, map) {
|
|
|
96
117
|
* journey. Reads the project map (testforge.md) for data-layer
|
|
97
118
|
* context. Returns the test file content.
|
|
98
119
|
*/
|
|
99
|
-
export function generateWithAI(journey) {
|
|
120
|
+
export function generateWithAI(journey, testPath) {
|
|
100
121
|
const map = existsSync(MAP_PATH) ? readFileSync(MAP_PATH, 'utf-8') : '';
|
|
101
|
-
const prompt = buildPrompt(journey, map);
|
|
102
|
-
|
|
122
|
+
const prompt = buildPrompt(journey, map, testPath);
|
|
123
|
+
// Generation is pure text — run with NO tools so the model just
|
|
124
|
+
// prints code instead of trying to write files.
|
|
125
|
+
const { program, args } = withTextOnlyTools(splitCommand(loadConfig().aiCommand));
|
|
103
126
|
const output = execFileSync(program, args, {
|
|
104
127
|
input: prompt,
|
|
105
128
|
encoding: 'utf-8',
|
|
@@ -107,35 +130,65 @@ export function generateWithAI(journey) {
|
|
|
107
130
|
});
|
|
108
131
|
return cleanOutput(output);
|
|
109
132
|
}
|
|
133
|
+
/** Does this line look like the start (or body) of real code? */
|
|
134
|
+
function looksLikeCode(line) {
|
|
135
|
+
const t = line.trim();
|
|
136
|
+
return (t.startsWith('//') ||
|
|
137
|
+
t.startsWith('/*') ||
|
|
138
|
+
t.startsWith('*') ||
|
|
139
|
+
t.startsWith('import ') ||
|
|
140
|
+
t.startsWith('export ') ||
|
|
141
|
+
t.startsWith('const ') ||
|
|
142
|
+
t.startsWith('let ') ||
|
|
143
|
+
t.startsWith('var ') ||
|
|
144
|
+
t.startsWith('function ') ||
|
|
145
|
+
t.startsWith('async ') ||
|
|
146
|
+
t.startsWith('require(') ||
|
|
147
|
+
t.startsWith("'use strict'") ||
|
|
148
|
+
t.startsWith('"use strict"'));
|
|
149
|
+
}
|
|
110
150
|
/**
|
|
111
151
|
* Extract clean code from the AI's reply.
|
|
112
152
|
*
|
|
113
153
|
* 1. If there's a fenced code block, take its contents.
|
|
114
|
-
* 2. Otherwise,
|
|
115
|
-
*
|
|
116
|
-
*
|
|
117
|
-
*
|
|
154
|
+
* 2. Otherwise, keep only the span between the first and the last
|
|
155
|
+
* code-ish line — this strips BOTH leading prose ("Here is the
|
|
156
|
+
* code:") and trailing prose ("Note: I attempted to write the
|
|
157
|
+
* file...") that the model sometimes adds despite instructions.
|
|
158
|
+
* 3. If no code can be detected at all, throw so the caller can
|
|
159
|
+
* retry or skip — we never return junk to be written to disk.
|
|
160
|
+
*
|
|
161
|
+
* Exported so the seed generator can share the exact same cleaning
|
|
162
|
+
* logic instead of a weaker fence-only strip.
|
|
118
163
|
*/
|
|
119
|
-
function cleanOutput(text) {
|
|
164
|
+
export function cleanOutput(text) {
|
|
120
165
|
// 1. Prefer a fenced code block if present.
|
|
121
|
-
const fence = text.match(/```[a-zA-Z]*\n([\s\S]*?)
|
|
166
|
+
const fence = text.match(/```[a-zA-Z]*\n([\s\S]*?)```/);
|
|
122
167
|
if (fence) {
|
|
123
|
-
|
|
168
|
+
const inner = fence[1].trim();
|
|
169
|
+
if (!inner) {
|
|
170
|
+
throw new Error('AI output contained an empty code fence');
|
|
171
|
+
}
|
|
172
|
+
return inner;
|
|
124
173
|
}
|
|
125
|
-
// 2. No fence —
|
|
174
|
+
// 2. No fence — locate the first line that looks like real code.
|
|
126
175
|
const lines = text.split('\n');
|
|
127
|
-
const codeStart = lines.findIndex(
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
176
|
+
const codeStart = lines.findIndex(looksLikeCode);
|
|
177
|
+
if (codeStart === -1) {
|
|
178
|
+
throw new Error('AI output contained no recognizable code');
|
|
179
|
+
}
|
|
180
|
+
// Trim trailing prose: walk back from the end to the last line that
|
|
181
|
+
// is code-ish or ends with a code terminator (; } ) ]). A trailing
|
|
182
|
+
// sentence ("Note: ...") ends with a word/period, so it is dropped.
|
|
183
|
+
let codeEnd = codeStart;
|
|
184
|
+
for (let i = lines.length - 1; i >= codeStart; i--) {
|
|
185
|
+
const t = lines[i].trim();
|
|
186
|
+
if (t === '')
|
|
187
|
+
continue;
|
|
188
|
+
if (looksLikeCode(t) || /[;})\]]$/.test(t)) {
|
|
189
|
+
codeEnd = i;
|
|
190
|
+
break;
|
|
191
|
+
}
|
|
139
192
|
}
|
|
140
|
-
return
|
|
193
|
+
return lines.slice(codeStart, codeEnd + 1).join('\n').trim();
|
|
141
194
|
}
|
package/dist/core/config.js
CHANGED
|
@@ -14,7 +14,16 @@
|
|
|
14
14
|
*
|
|
15
15
|
* If no config file exists, we default to "claude -p" so users with
|
|
16
16
|
* Claude installed work out of the box. Users of other tools (Gemini,
|
|
17
|
-
* Codex, etc.) set aiCommand to their own CLI
|
|
17
|
+
* Codex, etc.) set aiCommand to their own CLI — it just needs to read a
|
|
18
|
+
* prompt on stdin and print the reply to stdout.
|
|
19
|
+
*
|
|
20
|
+
* The command is a BASE. Each phase appends the flags it needs (these
|
|
21
|
+
* are tool-specific; for Claude): the setup phases (init/map/discover)
|
|
22
|
+
* add `--permission-mode acceptEdits` because the model reads/writes
|
|
23
|
+
* files, while the pure-text generation phases add `--allowedTools ""`
|
|
24
|
+
* so the model has NO tools and just prints code — it can't try (and
|
|
25
|
+
* fail) to write a file and leak "the write wasn't approved..." prose
|
|
26
|
+
* into the output. See withTextOnlyTools / withEditTools.
|
|
18
27
|
*/
|
|
19
28
|
import { readFileSync, existsSync } from 'node:fs';
|
|
20
29
|
import { join } from 'node:path';
|
|
@@ -44,14 +53,80 @@ export function loadConfig() {
|
|
|
44
53
|
}
|
|
45
54
|
/**
|
|
46
55
|
* Split an AI command string into a program and its arguments.
|
|
47
|
-
*
|
|
56
|
+
*
|
|
57
|
+
* Quote-aware: single or double quotes group a token and can produce an
|
|
58
|
+
* EMPTY-string argument, which matters for invocations like
|
|
59
|
+
* `claude -p --allowedTools ""` (disable all tools). A naive
|
|
60
|
+
* whitespace split would turn "" into the literal two-char token `""`.
|
|
61
|
+
*
|
|
62
|
+
* Example: `claude -p --allowedTools ""`
|
|
63
|
+
* -> { program: "claude", args: ["-p", "--allowedTools", ""] }
|
|
48
64
|
*/
|
|
49
65
|
export function splitCommand(command) {
|
|
50
|
-
const
|
|
51
|
-
|
|
52
|
-
|
|
66
|
+
const tokens = [];
|
|
67
|
+
let current = '';
|
|
68
|
+
let quote = null;
|
|
69
|
+
let started = false; // did the current token have any chars (incl. empty quotes)?
|
|
70
|
+
for (const ch of command.trim()) {
|
|
71
|
+
if (quote) {
|
|
72
|
+
if (ch === quote) {
|
|
73
|
+
quote = null; // closing quote — token may be empty
|
|
74
|
+
}
|
|
75
|
+
else {
|
|
76
|
+
current += ch;
|
|
77
|
+
}
|
|
78
|
+
continue;
|
|
79
|
+
}
|
|
80
|
+
if (ch === '"' || ch === "'") {
|
|
81
|
+
quote = ch;
|
|
82
|
+
started = true;
|
|
83
|
+
continue;
|
|
84
|
+
}
|
|
85
|
+
if (/\s/.test(ch)) {
|
|
86
|
+
if (started) {
|
|
87
|
+
tokens.push(current);
|
|
88
|
+
current = '';
|
|
89
|
+
started = false;
|
|
90
|
+
}
|
|
91
|
+
continue;
|
|
92
|
+
}
|
|
93
|
+
current += ch;
|
|
94
|
+
started = true;
|
|
95
|
+
}
|
|
96
|
+
if (started)
|
|
97
|
+
tokens.push(current);
|
|
98
|
+
const program = tokens[0] ?? '';
|
|
99
|
+
const args = tokens.slice(1);
|
|
53
100
|
return { program, args };
|
|
54
101
|
}
|
|
102
|
+
/** Is this command Claude's CLI (so Claude-specific flags apply)? */
|
|
103
|
+
function isClaude(program) {
|
|
104
|
+
const base = program.split(/[\\/]/).pop() ?? program;
|
|
105
|
+
return base === 'claude';
|
|
106
|
+
}
|
|
107
|
+
/**
|
|
108
|
+
* Args for a pure-text generation call (test/seed): no tools at all, so
|
|
109
|
+
* the model prints code instead of attempting file writes. Only applies
|
|
110
|
+
* the Claude flag when the base command is Claude; other CLIs get the
|
|
111
|
+
* command unchanged (configure their own text-only mode if needed).
|
|
112
|
+
*/
|
|
113
|
+
export function withTextOnlyTools(cmd) {
|
|
114
|
+
if (!isClaude(cmd.program))
|
|
115
|
+
return cmd;
|
|
116
|
+
return { program: cmd.program, args: [...cmd.args, '--allowedTools', ''] };
|
|
117
|
+
}
|
|
118
|
+
/**
|
|
119
|
+
* Args for a phase where the model must read/write files (init, map,
|
|
120
|
+
* discover). Mirrors withTextOnlyTools but grants edit permission.
|
|
121
|
+
*/
|
|
122
|
+
export function withEditTools(cmd) {
|
|
123
|
+
if (!isClaude(cmd.program))
|
|
124
|
+
return cmd;
|
|
125
|
+
return {
|
|
126
|
+
program: cmd.program,
|
|
127
|
+
args: [...cmd.args, '--permission-mode', 'acceptEdits'],
|
|
128
|
+
};
|
|
129
|
+
}
|
|
55
130
|
/**
|
|
56
131
|
* Decide where a journey's test file should go, based on its type
|
|
57
132
|
* and the config. Falls back to "generated-tests/<area>" when the
|
package/dist/core/discover.js
CHANGED
|
@@ -15,7 +15,10 @@
|
|
|
15
15
|
* keeps this simple and works for projects of any size.
|
|
16
16
|
*/
|
|
17
17
|
import { execFileSync } from 'node:child_process';
|
|
18
|
-
import { loadConfig, splitCommand } from './config.js';
|
|
18
|
+
import { loadConfig, splitCommand, withEditTools } from './config.js';
|
|
19
|
+
import { promptConfirm } from './prompt.js';
|
|
20
|
+
import { resolveDocsSource, saveSourceChoice, } from './source-of-truth.js';
|
|
21
|
+
import { analyzeReference, draftDocsFromReference, DIFF_PATH, } from './reference.js';
|
|
19
22
|
/** Build the discovery prompt. */
|
|
20
23
|
export function buildDiscoverPrompt(srcDir, docsDir) {
|
|
21
24
|
return [
|
|
@@ -83,21 +86,56 @@ export function buildDiscoverPrompt(srcDir, docsDir) {
|
|
|
83
86
|
].join('\n');
|
|
84
87
|
}
|
|
85
88
|
/**
|
|
86
|
-
*
|
|
87
|
-
*
|
|
88
|
-
* summary of what it created).
|
|
89
|
+
* Draft the docs from the project's own code (the default). Invokes the
|
|
90
|
+
* AI CLI with permission to read source and write markdown files.
|
|
89
91
|
*/
|
|
90
|
-
|
|
92
|
+
function runCodeDiscover(srcDir, docsDir) {
|
|
91
93
|
const prompt = buildDiscoverPrompt(srcDir, docsDir);
|
|
92
94
|
// Use the configured AI command. Note: discovery needs the AI to
|
|
93
95
|
// read source files and write markdown files. The flags for that
|
|
94
96
|
// are tool-specific; for Claude we add acceptEdits. Other tools may
|
|
95
97
|
// need different setup (see README).
|
|
96
|
-
const { program, args } = splitCommand(loadConfig().aiCommand);
|
|
97
|
-
const output = execFileSync(program,
|
|
98
|
+
const { program, args } = withEditTools(splitCommand(loadConfig().aiCommand));
|
|
99
|
+
const output = execFileSync(program, args, {
|
|
98
100
|
input: prompt,
|
|
99
101
|
encoding: 'utf-8',
|
|
100
102
|
maxBuffer: 10 * 1024 * 1024,
|
|
101
103
|
});
|
|
102
104
|
return output.trim();
|
|
103
105
|
}
|
|
106
|
+
/**
|
|
107
|
+
* Run discovery. First decides the source of truth for the docs (the
|
|
108
|
+
* code-derived map, or an external reference — see source-of-truth.ts),
|
|
109
|
+
* then drafts the journey docs from it. When a reference is chosen it is
|
|
110
|
+
* compared against the map first (writing testforge.diff.md), and, for a
|
|
111
|
+
* fresh interactive choice, confirmed before the docs are rewritten.
|
|
112
|
+
*
|
|
113
|
+
* Async because the source-of-truth barrier may prompt the developer.
|
|
114
|
+
* Returns the AI's text summary of what it created.
|
|
115
|
+
*/
|
|
116
|
+
export async function runDiscover(srcDir, docsDir, opts = {}) {
|
|
117
|
+
const source = await resolveDocsSource(opts);
|
|
118
|
+
if (source.mode === 'reference' && source.reference) {
|
|
119
|
+
console.log(`\nComparing the reference against testforge.md: ${source.reference}`);
|
|
120
|
+
console.log('(reading the reference and the map — this may take a moment)\n');
|
|
121
|
+
const summary = analyzeReference(source.reference, docsDir);
|
|
122
|
+
console.log(summary);
|
|
123
|
+
console.log(`\nFull comparison written to ${DIFF_PATH}.`);
|
|
124
|
+
// Only a freshly-typed interactive choice gets a confirmation step;
|
|
125
|
+
// an explicit flag or a remembered choice proceeds directly.
|
|
126
|
+
if (source.origin === 'prompt') {
|
|
127
|
+
const ok = await promptConfirm('\nDraft the docs from this reference (make it the source of truth)?', true);
|
|
128
|
+
if (!ok) {
|
|
129
|
+
console.log('Basing docs on testforge.md instead.\n');
|
|
130
|
+
saveSourceChoice({ mode: 'map' });
|
|
131
|
+
return runCodeDiscover(srcDir, docsDir);
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
console.log('\nDrafting docs from the reference...\n');
|
|
135
|
+
const out = draftDocsFromReference(srcDir, docsDir, source.reference);
|
|
136
|
+
saveSourceChoice({ mode: 'reference', reference: source.reference });
|
|
137
|
+
return out;
|
|
138
|
+
}
|
|
139
|
+
saveSourceChoice({ mode: 'map' });
|
|
140
|
+
return runCodeDiscover(srcDir, docsDir);
|
|
141
|
+
}
|