tryscript 0.1.5 → 0.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +41 -8
- package/dist/bin.cjs +356 -13
- package/dist/bin.cjs.map +1 -1
- package/dist/bin.mjs +357 -14
- package/dist/bin.mjs.map +1 -1
- package/dist/index.cjs +10 -2
- package/dist/index.d.cts +148 -2
- package/dist/index.d.mts +148 -2
- package/dist/index.mjs +2 -2
- package/dist/{src-D-bd-j9T.cjs → src-BIZMxxIt.cjs} +487 -12
- package/dist/src-BIZMxxIt.cjs.map +1 -0
- package/dist/{src-CC3xA1cp.mjs → src-BQxIhzgF.mjs} +441 -14
- package/dist/src-BQxIhzgF.mjs.map +1 -0
- package/docs/tryscript-reference.md +116 -5
- package/package.json +4 -2
- package/dist/src-CC3xA1cp.mjs.map +0 -1
- package/dist/src-D-bd-j9T.cjs.map +0 -1
package/README.md
CHANGED
|
@@ -1,25 +1,43 @@
|
|
|
1
1
|
# tryscript
|
|
2
2
|
|
|
3
|
-
[](https://x.com/ojoshe)
|
|
4
|
+
[](https://github.com/jlevy/tryscript/actions/runs/22267140455)
|
|
5
|
+
[](https://github.com/jlevy/tryscript/actions/runs/22267140455)
|
|
5
6
|
[](https://www.npmjs.com/package/tryscript)
|
|
6
|
-
[](https://x.com/ojoshe)
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
**Powerful, agent-friendly testing of CLI applications via golden tests**
|
|
9
9
|
|
|
10
10
|
> [!NOTE]
|
|
11
11
|
> 100% of the code and specs in this repository were written by Claude Code.
|
|
12
12
|
> The design and management and prompting was by me ([jlevy](https://github.com/jlevy)) supported by the workflows, agent rules,
|
|
13
|
-
> and other research docs in [
|
|
13
|
+
> and other research docs in [tbd](https://github.com/jlevy/tbd).
|
|
14
14
|
>
|
|
15
|
-
>
|
|
15
|
+
> I find the code quality higher than most agent-written code I've
|
|
16
16
|
> seen because of the spec-driven process.
|
|
17
17
|
> You can review the architecture doc and all of the specs all of the specs in [docs/project](docs/project).
|
|
18
18
|
> The general research, guideline, and rules docs I use are in [docs/general](docs/general).
|
|
19
19
|
|
|
20
|
+
## Why?
|
|
21
|
+
|
|
22
|
+
Write CLI tests as Markdown. tryscript runs commands, captures output, and compares against expected results:
|
|
23
|
+
|
|
24
|
+
- Tests are clear and maintainable for agents and humans: tests become documentation; documentation becomes tests
|
|
25
|
+
- Inner state and working can be exposed for greater test coverage at no extra cost
|
|
26
|
+
- Things are quick to implement or test using arbitrary shell commands
|
|
27
|
+
|
|
28
|
+
This began as a TypeScript port of [trycmd](https://github.com/assert-rs/trycmd) but I (well, Claude and friends)
|
|
29
|
+
have since enhanced it to be more agent-friendly and self-documenting as a CLI.
|
|
30
|
+
|
|
31
|
+
For a bit more philosophy on why golden tests are so useful, you (or your friendly agent)
|
|
32
|
+
should read [tbd](https://github.com/jlevy/tbd)’s guidelines doc:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
npx --yes get-tbd@latest guidelines golden-testing-guidelines
|
|
36
|
+
```
|
|
37
|
+
|
|
20
38
|
## What It Does
|
|
21
39
|
|
|
22
|
-
|
|
40
|
+
An example test:
|
|
23
41
|
|
|
24
42
|
````markdown
|
|
25
43
|
---
|
|
@@ -68,6 +86,16 @@ $ my-cli process data.json > output.txt && grep "success" output.txt
|
|
|
68
86
|
|
|
69
87
|
The `[..]` matches any text on that line. The `...` matches zero or more lines. These "elision patterns" let tests handle dynamic output gracefully. Any shell command works - pipes, redirects, environment variables, etc.
|
|
70
88
|
|
|
89
|
+
### Wildcard Categories
|
|
90
|
+
|
|
91
|
+
Tryscript supports three categories of wildcards, in order of preference:
|
|
92
|
+
|
|
93
|
+
1. **Named patterns** (`[HASH]`, `[VERSION]`, `[CWD]`, etc.) -- Typed dynamic values with specific meaning. Preferred when the output has a known structure.
|
|
94
|
+
2. **Unknown wildcards** (`[??]`, `???`) -- Temporary placeholders for output you haven't filled in yet. Intended to be expanded with `--expand` before finalizing tests.
|
|
95
|
+
3. **Generic wildcards** (`[..]`, `...`) -- Intentional omission of unpredictable or irrelevant output. Use when the exact value doesn't matter for the test.
|
|
96
|
+
|
|
97
|
+
Use `--expand` to automatically fill in unknown wildcards with actual output after a successful run.
|
|
98
|
+
|
|
71
99
|
## Quick Start
|
|
72
100
|
|
|
73
101
|
```bash
|
|
@@ -90,7 +118,8 @@ npx tryscript run --update tests/
|
|
|
90
118
|
## Features
|
|
91
119
|
|
|
92
120
|
- **Markdown format** - Tests are readable documentation
|
|
93
|
-
- **Elision patterns** - Handle variable output: `[..]`, `...`, `[CWD]`, `[ROOT]`, `[EXE]`
|
|
121
|
+
- **Elision patterns** - Handle variable output: `[..]`, `...`, `[??]`, `???`, `[CWD]`, `[ROOT]`, `[EXE]`
|
|
122
|
+
- **Wildcard expansion** - Fill in `[??]`/`???` placeholders with actual output via `--expand`
|
|
94
123
|
- **Custom patterns** - Define regex patterns for timestamps, versions, UUIDs
|
|
95
124
|
- **Update mode** - Regenerate expected output with `--update`
|
|
96
125
|
- **Sandbox mode** - Isolate tests in temp directories
|
|
@@ -113,6 +142,10 @@ For complete syntax reference, run `tryscript docs` or see the [reference docume
|
|
|
113
142
|
| Option | Description |
|
|
114
143
|
| --- | --- |
|
|
115
144
|
| `--update` | Update test files with actual output |
|
|
145
|
+
| `--expand` | Expand unknown wildcards (`???`/`[??]`) with actual output |
|
|
146
|
+
| `--expand-generic` | Expand unknown + generic wildcards |
|
|
147
|
+
| `--expand-all` | Expand all wildcards (including named patterns) |
|
|
148
|
+
| `--capture-log <path>` | Write wildcard capture log to YAML file |
|
|
116
149
|
| `--fail-fast` | Stop on first failure |
|
|
117
150
|
| `--filter <regex>` | Filter tests by name |
|
|
118
151
|
| `--verbose` | Show detailed output |
|
package/dist/bin.cjs
CHANGED
|
@@ -1,20 +1,20 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
3
|
|
|
4
|
-
const require_src = require('./src-
|
|
4
|
+
const require_src = require('./src-BIZMxxIt.cjs');
|
|
5
5
|
let node_url = require("node:url");
|
|
6
6
|
let node_fs = require("node:fs");
|
|
7
7
|
let node_path = require("node:path");
|
|
8
8
|
let node_child_process = require("node:child_process");
|
|
9
9
|
let node_fs_promises = require("node:fs/promises");
|
|
10
10
|
let node_os = require("node:os");
|
|
11
|
+
let atomically = require("atomically");
|
|
11
12
|
let commander = require("commander");
|
|
12
13
|
let fast_glob = require("fast-glob");
|
|
13
14
|
fast_glob = require_src.__toESM(fast_glob);
|
|
14
15
|
let picocolors = require("picocolors");
|
|
15
16
|
picocolors = require_src.__toESM(picocolors);
|
|
16
17
|
let diff = require("diff");
|
|
17
|
-
let atomically = require("atomically");
|
|
18
18
|
|
|
19
19
|
//#region src/cli/lib/shared.ts
|
|
20
20
|
/**
|
|
@@ -136,9 +136,10 @@ function reportSummary(summary, _options) {
|
|
|
136
136
|
async function updateTestFile(file, results) {
|
|
137
137
|
let content = file.rawContent;
|
|
138
138
|
const changes = [];
|
|
139
|
-
const
|
|
139
|
+
const resultByBlock = new Map(results.map((result) => [result.block, result]));
|
|
140
|
+
const blocksWithResults = [...file.blocks].map((block) => ({
|
|
140
141
|
block,
|
|
141
|
-
result:
|
|
142
|
+
result: resultByBlock.get(block)
|
|
142
143
|
})).reverse();
|
|
143
144
|
for (const { block, result } of blocksWithResults) {
|
|
144
145
|
if (!result) continue;
|
|
@@ -161,15 +162,248 @@ async function updateTestFile(file, results) {
|
|
|
161
162
|
* Build an updated console block with new expected output.
|
|
162
163
|
*/
|
|
163
164
|
function buildUpdatedBlock(block, result) {
|
|
164
|
-
const
|
|
165
|
+
const fence = "`".repeat(/^(`+)/.exec(block.rawContent)?.[1]?.length ?? 3);
|
|
166
|
+
const commandLines = block.command.split("\n").map((line, i) => {
|
|
165
167
|
return i === 0 ? `$ ${line}` : `> ${line}`;
|
|
166
|
-
})
|
|
168
|
+
});
|
|
169
|
+
const lines = [`${fence}console`, ...commandLines];
|
|
167
170
|
const trimmedOutput = result.actualOutput.trimEnd();
|
|
168
171
|
if (trimmedOutput) lines.push(trimmedOutput);
|
|
169
|
-
lines.push(`? ${result.actualExitCode}`,
|
|
172
|
+
lines.push(`? ${result.actualExitCode}`, fence);
|
|
170
173
|
return lines.join("\n");
|
|
171
174
|
}
|
|
172
175
|
|
|
176
|
+
//#endregion
|
|
177
|
+
//#region src/lib/lcov.ts
|
|
178
|
+
/**
|
|
179
|
+
* LCOV parsing, merging, and writing utilities.
|
|
180
|
+
*
|
|
181
|
+
* LCOV format reference:
|
|
182
|
+
* - SF: Source file path
|
|
183
|
+
* - DA:linenum,hitcount - Line data
|
|
184
|
+
* - FN:linenum,funcname - Function definition
|
|
185
|
+
* - FNDA:hitcount,funcname - Function hit data
|
|
186
|
+
* - FNF: Functions found count
|
|
187
|
+
* - FNH: Functions hit count
|
|
188
|
+
* - BRF: Branches found count
|
|
189
|
+
* - BRH: Branches hit count
|
|
190
|
+
* - BRDA:line,block,branch,taken - Branch data
|
|
191
|
+
* - LF: Lines found count
|
|
192
|
+
* - LH: Lines hit count
|
|
193
|
+
* - end_of_record - End of file record
|
|
194
|
+
*/
|
|
195
|
+
/**
|
|
196
|
+
* Parse LCOV content into structured data.
|
|
197
|
+
*/
|
|
198
|
+
function parseLcov(content) {
|
|
199
|
+
const files = /* @__PURE__ */ new Map();
|
|
200
|
+
let currentFile = null;
|
|
201
|
+
for (const line of content.split("\n")) {
|
|
202
|
+
const trimmed = line.trim();
|
|
203
|
+
if (trimmed.startsWith("SF:")) {
|
|
204
|
+
const path = trimmed.slice(3);
|
|
205
|
+
currentFile = {
|
|
206
|
+
path,
|
|
207
|
+
lines: /* @__PURE__ */ new Map(),
|
|
208
|
+
functions: /* @__PURE__ */ new Map(),
|
|
209
|
+
branches: []
|
|
210
|
+
};
|
|
211
|
+
files.set(path, currentFile);
|
|
212
|
+
} else if (trimmed.startsWith("DA:") && currentFile) {
|
|
213
|
+
const parts = trimmed.slice(3).split(",");
|
|
214
|
+
const lineNumber = parseInt(parts[0], 10);
|
|
215
|
+
const hitCount = parseInt(parts[1], 10);
|
|
216
|
+
currentFile.lines.set(lineNumber, {
|
|
217
|
+
lineNumber,
|
|
218
|
+
hitCount
|
|
219
|
+
});
|
|
220
|
+
} else if (trimmed.startsWith("FN:") && currentFile) {
|
|
221
|
+
const parts = trimmed.slice(3).split(",");
|
|
222
|
+
const lineNumber = parseInt(parts[0], 10);
|
|
223
|
+
const name = parts.slice(1).join(",");
|
|
224
|
+
if (!currentFile.functions.has(name)) currentFile.functions.set(name, {
|
|
225
|
+
name,
|
|
226
|
+
lineNumber,
|
|
227
|
+
hitCount: 0
|
|
228
|
+
});
|
|
229
|
+
else currentFile.functions.get(name).lineNumber = lineNumber;
|
|
230
|
+
} else if (trimmed.startsWith("FNDA:") && currentFile) {
|
|
231
|
+
const parts = trimmed.slice(5).split(",");
|
|
232
|
+
const hitCount = parseInt(parts[0], 10);
|
|
233
|
+
const name = parts.slice(1).join(",");
|
|
234
|
+
if (currentFile.functions.has(name)) currentFile.functions.get(name).hitCount = hitCount;
|
|
235
|
+
else currentFile.functions.set(name, {
|
|
236
|
+
name,
|
|
237
|
+
lineNumber: 0,
|
|
238
|
+
hitCount
|
|
239
|
+
});
|
|
240
|
+
} else if (trimmed.startsWith("BRDA:") && currentFile) {
|
|
241
|
+
const parts = trimmed.slice(5).split(",");
|
|
242
|
+
currentFile.branches.push({
|
|
243
|
+
line: parseInt(parts[0], 10),
|
|
244
|
+
block: parseInt(parts[1], 10),
|
|
245
|
+
branch: parseInt(parts[2], 10),
|
|
246
|
+
taken: parts[3] === "-" ? -1 : parseInt(parts[3], 10)
|
|
247
|
+
});
|
|
248
|
+
} else if (trimmed === "end_of_record") currentFile = null;
|
|
249
|
+
}
|
|
250
|
+
return { files };
|
|
251
|
+
}
|
|
252
|
+
/**
|
|
253
|
+
* Merge multiple LCOV data structures, taking max hit counts.
|
|
254
|
+
*/
|
|
255
|
+
function mergeLcov(...lcovs) {
|
|
256
|
+
const merged = /* @__PURE__ */ new Map();
|
|
257
|
+
for (const lcov of lcovs) for (const [path, file] of lcov.files) if (!merged.has(path)) merged.set(path, {
|
|
258
|
+
path,
|
|
259
|
+
lines: new Map(file.lines),
|
|
260
|
+
functions: new Map(file.functions),
|
|
261
|
+
branches: [...file.branches]
|
|
262
|
+
});
|
|
263
|
+
else {
|
|
264
|
+
const existing = merged.get(path);
|
|
265
|
+
for (const [lineNum, lineData] of file.lines) {
|
|
266
|
+
const existingLine = existing.lines.get(lineNum);
|
|
267
|
+
if (existingLine) existingLine.hitCount = Math.max(existingLine.hitCount, lineData.hitCount);
|
|
268
|
+
else existing.lines.set(lineNum, { ...lineData });
|
|
269
|
+
}
|
|
270
|
+
for (const [name, funcData] of file.functions) {
|
|
271
|
+
const existingFunc = existing.functions.get(name);
|
|
272
|
+
if (existingFunc) existingFunc.hitCount = Math.max(existingFunc.hitCount, funcData.hitCount);
|
|
273
|
+
else existing.functions.set(name, { ...funcData });
|
|
274
|
+
}
|
|
275
|
+
for (const branch of file.branches) {
|
|
276
|
+
const existingBranch = existing.branches.find((b) => b.line === branch.line && b.block === branch.block && b.branch === branch.branch);
|
|
277
|
+
if (existingBranch) {
|
|
278
|
+
if (branch.taken >= 0) existingBranch.taken = existingBranch.taken >= 0 ? Math.max(existingBranch.taken, branch.taken) : branch.taken;
|
|
279
|
+
} else existing.branches.push({ ...branch });
|
|
280
|
+
}
|
|
281
|
+
}
|
|
282
|
+
return { files: merged };
|
|
283
|
+
}
|
|
284
|
+
/**
|
|
285
|
+
* Convert LCOV data back to LCOV format string.
|
|
286
|
+
*/
|
|
287
|
+
function formatLcov(lcov) {
|
|
288
|
+
const lines = [];
|
|
289
|
+
for (const file of lcov.files.values()) {
|
|
290
|
+
lines.push(`SF:${file.path}`);
|
|
291
|
+
const sortedFunctions = [...file.functions.values()].sort((a, b) => a.lineNumber - b.lineNumber);
|
|
292
|
+
for (const func of sortedFunctions) lines.push(`FN:${func.lineNumber},${func.name}`);
|
|
293
|
+
for (const func of sortedFunctions) lines.push(`FNDA:${func.hitCount},${func.name}`);
|
|
294
|
+
const fnf = file.functions.size;
|
|
295
|
+
const fnh = [...file.functions.values()].filter((f) => f.hitCount > 0).length;
|
|
296
|
+
lines.push(`FNF:${fnf}`);
|
|
297
|
+
lines.push(`FNH:${fnh}`);
|
|
298
|
+
for (const branch of file.branches) {
|
|
299
|
+
const taken = branch.taken < 0 ? "-" : branch.taken.toString();
|
|
300
|
+
lines.push(`BRDA:${branch.line},${branch.block},${branch.branch},${taken}`);
|
|
301
|
+
}
|
|
302
|
+
const brf = file.branches.length;
|
|
303
|
+
const brh = file.branches.filter((b) => b.taken > 0).length;
|
|
304
|
+
lines.push(`BRF:${brf}`);
|
|
305
|
+
lines.push(`BRH:${brh}`);
|
|
306
|
+
const sortedLines = [...file.lines.values()].sort((a, b) => a.lineNumber - b.lineNumber);
|
|
307
|
+
for (const line of sortedLines) lines.push(`DA:${line.lineNumber},${line.hitCount}`);
|
|
308
|
+
const lf = file.lines.size;
|
|
309
|
+
const lh = [...file.lines.values()].filter((l) => l.hitCount > 0).length;
|
|
310
|
+
lines.push(`LF:${lf}`);
|
|
311
|
+
lines.push(`LH:${lh}`);
|
|
312
|
+
lines.push("end_of_record");
|
|
313
|
+
}
|
|
314
|
+
return lines.join("\n") + "\n";
|
|
315
|
+
}
|
|
316
|
+
/**
|
|
317
|
+
* Convert LCOV data to JSON summary format (compatible with istanbul/vitest).
|
|
318
|
+
*/
|
|
319
|
+
function lcovToJsonSummary(lcov) {
|
|
320
|
+
const withPct = (total, covered) => ({
|
|
321
|
+
total,
|
|
322
|
+
covered,
|
|
323
|
+
skipped: 0,
|
|
324
|
+
pct: total > 0 ? parseFloat((covered / total * 100).toFixed(2)) : 100
|
|
325
|
+
});
|
|
326
|
+
const totals = {
|
|
327
|
+
lines: {
|
|
328
|
+
total: 0,
|
|
329
|
+
covered: 0
|
|
330
|
+
},
|
|
331
|
+
functions: {
|
|
332
|
+
total: 0,
|
|
333
|
+
covered: 0
|
|
334
|
+
},
|
|
335
|
+
branches: {
|
|
336
|
+
total: 0,
|
|
337
|
+
covered: 0
|
|
338
|
+
}
|
|
339
|
+
};
|
|
340
|
+
const summary = { total: {
|
|
341
|
+
lines: withPct(0, 0),
|
|
342
|
+
statements: withPct(0, 0),
|
|
343
|
+
functions: withPct(0, 0),
|
|
344
|
+
branches: withPct(0, 0),
|
|
345
|
+
branchesTrue: {
|
|
346
|
+
total: 0,
|
|
347
|
+
covered: 0,
|
|
348
|
+
skipped: 0,
|
|
349
|
+
pct: 100
|
|
350
|
+
}
|
|
351
|
+
} };
|
|
352
|
+
for (const file of lcov.files.values()) {
|
|
353
|
+
const linesTotal = file.lines.size;
|
|
354
|
+
const linesCovered = [...file.lines.values()].filter((l) => l.hitCount > 0).length;
|
|
355
|
+
const funcsTotal = file.functions.size;
|
|
356
|
+
const funcsCovered = [...file.functions.values()].filter((f) => f.hitCount > 0).length;
|
|
357
|
+
const branchesTotal = file.branches.length;
|
|
358
|
+
const branchesCovered = file.branches.filter((b) => b.taken > 0).length;
|
|
359
|
+
summary[file.path] = {
|
|
360
|
+
lines: withPct(linesTotal, linesCovered),
|
|
361
|
+
statements: withPct(linesTotal, linesCovered),
|
|
362
|
+
functions: withPct(funcsTotal, funcsCovered),
|
|
363
|
+
branches: withPct(branchesTotal, branchesCovered)
|
|
364
|
+
};
|
|
365
|
+
totals.lines.total += linesTotal;
|
|
366
|
+
totals.lines.covered += linesCovered;
|
|
367
|
+
totals.functions.total += funcsTotal;
|
|
368
|
+
totals.functions.covered += funcsCovered;
|
|
369
|
+
totals.branches.total += branchesTotal;
|
|
370
|
+
totals.branches.covered += branchesCovered;
|
|
371
|
+
}
|
|
372
|
+
summary.total = {
|
|
373
|
+
lines: withPct(totals.lines.total, totals.lines.covered),
|
|
374
|
+
statements: withPct(totals.lines.total, totals.lines.covered),
|
|
375
|
+
functions: withPct(totals.functions.total, totals.functions.covered),
|
|
376
|
+
branches: withPct(totals.branches.total, totals.branches.covered),
|
|
377
|
+
branchesTrue: {
|
|
378
|
+
total: 0,
|
|
379
|
+
covered: 0,
|
|
380
|
+
skipped: 0,
|
|
381
|
+
pct: 100
|
|
382
|
+
}
|
|
383
|
+
};
|
|
384
|
+
return summary;
|
|
385
|
+
}
|
|
386
|
+
/**
|
|
387
|
+
* Read and parse an LCOV file.
|
|
388
|
+
*/
|
|
389
|
+
function readLcovFile(path) {
|
|
390
|
+
return parseLcov((0, node_fs.readFileSync)(path, "utf8"));
|
|
391
|
+
}
|
|
392
|
+
/**
|
|
393
|
+
* Write LCOV data to a file.
|
|
394
|
+
*/
|
|
395
|
+
function writeLcovFile(path, lcov) {
|
|
396
|
+
(0, node_fs.mkdirSync)((0, node_path.dirname)(path), { recursive: true });
|
|
397
|
+
(0, node_fs.writeFileSync)(path, formatLcov(lcov));
|
|
398
|
+
}
|
|
399
|
+
/**
|
|
400
|
+
* Write JSON summary to a file.
|
|
401
|
+
*/
|
|
402
|
+
function writeJsonSummary(path, summary) {
|
|
403
|
+
(0, node_fs.mkdirSync)((0, node_path.dirname)(path), { recursive: true });
|
|
404
|
+
(0, node_fs.writeFileSync)(path, JSON.stringify(summary, null, 2));
|
|
405
|
+
}
|
|
406
|
+
|
|
173
407
|
//#endregion
|
|
174
408
|
//#region src/lib/coverage.ts
|
|
175
409
|
/**
|
|
@@ -275,6 +509,33 @@ async function cleanupCoverageContext(ctx) {
|
|
|
275
509
|
});
|
|
276
510
|
} catch {}
|
|
277
511
|
}
|
|
512
|
+
/**
|
|
513
|
+
* Merge external LCOV file with generated coverage.
|
|
514
|
+
* Reads the generated lcov.info, merges with external LCOV, and writes back.
|
|
515
|
+
* Also generates coverage-summary.json for badge generation.
|
|
516
|
+
*
|
|
517
|
+
* @returns Object with merged coverage percentages, or null if merge failed
|
|
518
|
+
*/
|
|
519
|
+
function mergeExternalCoverage(reportsDir, externalLcovPath) {
|
|
520
|
+
const generatedLcovPath = (0, node_path.join)(reportsDir, "lcov.info");
|
|
521
|
+
if (!(0, node_fs.existsSync)(externalLcovPath)) {
|
|
522
|
+
console.error(`External LCOV file not found: ${externalLcovPath}`);
|
|
523
|
+
return null;
|
|
524
|
+
}
|
|
525
|
+
if (!(0, node_fs.existsSync)(generatedLcovPath)) {
|
|
526
|
+
console.error(`Generated LCOV file not found: ${generatedLcovPath}`);
|
|
527
|
+
console.error("Make sure \"lcov\" is included in reporters");
|
|
528
|
+
return null;
|
|
529
|
+
}
|
|
530
|
+
const mergedLcov = mergeLcov(readLcovFile(externalLcovPath), readLcovFile(generatedLcovPath));
|
|
531
|
+
writeLcovFile(generatedLcovPath, mergedLcov);
|
|
532
|
+
const summary = lcovToJsonSummary(mergedLcov);
|
|
533
|
+
writeJsonSummary((0, node_path.join)(reportsDir, "coverage-summary.json"), summary);
|
|
534
|
+
return {
|
|
535
|
+
lines: summary.total.lines.pct,
|
|
536
|
+
functions: summary.total.functions.pct
|
|
537
|
+
};
|
|
538
|
+
}
|
|
278
539
|
|
|
279
540
|
//#endregion
|
|
280
541
|
//#region src/cli/commands/run.ts
|
|
@@ -282,10 +543,32 @@ async function cleanupCoverageContext(ctx) {
|
|
|
282
543
|
* Register the run command.
|
|
283
544
|
*/
|
|
284
545
|
function registerRunCommand(program) {
|
|
285
|
-
program.command("run").description("Run golden tests").argument("[files...]", "Test files to run (default: **/*.tryscript.md)").option("--update", "Update golden files with actual output").option("--diff", "Show diff on failure (default: true)").option("--no-diff", "Hide diff on failure").option("--fail-fast", "Stop on first failure").option("--filter <pattern>", "Filter tests by name pattern").option("--verbose", "Show detailed output including passing test output").option("--quiet", "Suppress non-essential output (only show failures)").option("--coverage", "Enable code coverage collection (requires c8)").option("--coverage-dir <dir>", "Coverage output directory (default: coverage-tryscript)").option("--coverage-reporter <reporter...>", "Coverage reporters (default: text, html). Can be specified multiple times.").option("--coverage-exclude <pattern...>", "Patterns to exclude from coverage (c8 --exclude). Can be specified multiple times.").option("--coverage-exclude-node-modules", "Exclude node_modules from coverage (c8 --exclude-node-modules, default: true)").option("--no-coverage-exclude-node-modules", "Include node_modules in coverage (c8 --no-exclude-node-modules)").option("--coverage-exclude-after-remap", "Apply exclude logic after sourcemap remapping (c8 --exclude-after-remap)").option("--coverage-skip-full", "Hide files with 100% coverage (c8 --skip-full)").option("--coverage-allow-external", "Allow files from outside cwd (c8 --allowExternal)").option("--coverage-monocart", "Use monocart for accurate line counts, better for merging with vitest (c8 --experimental-monocart)").action(runCommand$1);
|
|
546
|
+
program.command("run").description("Run golden tests").argument("[files...]", "Test files to run (default: **/*.tryscript.md)").option("--update", "Update golden files with actual output").option("--diff", "Show diff on failure (default: true)").option("--no-diff", "Hide diff on failure").option("--fail-fast", "Stop on first failure").option("--filter <pattern>", "Filter tests by name pattern").option("--verbose", "Show detailed output including passing test output").option("--quiet", "Suppress non-essential output (only show failures)").option("--expand", "Expand unknown wildcards (??? and [??]) with actual output").option("--expand-generic", "Expand unknown and generic wildcards with actual output").option("--expand-all", "Expand all wildcards (including named patterns) with actual output").option("--capture-log <path>", "Write wildcard capture log to YAML file").option("--coverage", "Enable code coverage collection (requires c8)").option("--coverage-dir <dir>", "Coverage output directory (default: coverage-tryscript)").option("--coverage-reporter <reporter...>", "Coverage reporters (default: text, html). Can be specified multiple times.").option("--coverage-exclude <pattern...>", "Patterns to exclude from coverage (c8 --exclude). Can be specified multiple times.").option("--coverage-exclude-node-modules", "Exclude node_modules from coverage (c8 --exclude-node-modules, default: true)").option("--no-coverage-exclude-node-modules", "Include node_modules in coverage (c8 --no-exclude-node-modules)").option("--coverage-exclude-after-remap", "Apply exclude logic after sourcemap remapping (c8 --exclude-after-remap)").option("--coverage-skip-full", "Hide files with 100% coverage (c8 --skip-full)").option("--coverage-allow-external", "Allow files from outside cwd (c8 --allowExternal)").option("--coverage-monocart", "Use monocart for accurate line counts, better for merging with vitest (c8 --experimental-monocart)").option("--merge-lcov <path>", "Merge coverage from an existing LCOV file (e.g., from vitest --coverage)").action(runCommand$1);
|
|
547
|
+
}
|
|
548
|
+
/**
|
|
549
|
+
* Count unknown wildcard tokens (`???` and `[??]`) in expected output.
|
|
550
|
+
*/
|
|
551
|
+
function countUnknownWildcards(expectedOutput) {
|
|
552
|
+
return (expectedOutput.match(/\[\?\?]/g) ?? []).length + (expectedOutput.match(/\?\?\?\n/g) ?? []).length;
|
|
286
553
|
}
|
|
287
554
|
async function runCommand$1(files, options) {
|
|
288
555
|
const startTime = Date.now();
|
|
556
|
+
if ([
|
|
557
|
+
options.expand,
|
|
558
|
+
options.expandGeneric,
|
|
559
|
+
options.expandAll
|
|
560
|
+
].filter(Boolean).length > 1) {
|
|
561
|
+
logError("--expand, --expand-generic, and --expand-all are mutually exclusive");
|
|
562
|
+
process.exit(1);
|
|
563
|
+
}
|
|
564
|
+
let expandLevel;
|
|
565
|
+
if (options.expand) expandLevel = "unknown";
|
|
566
|
+
else if (options.expandGeneric) expandLevel = "generic";
|
|
567
|
+
else if (options.expandAll) expandLevel = "all";
|
|
568
|
+
if (expandLevel && options.update) {
|
|
569
|
+
logError("--expand* flags and --update are mutually exclusive");
|
|
570
|
+
process.exit(1);
|
|
571
|
+
}
|
|
289
572
|
const opts = {
|
|
290
573
|
diff: options.diff !== false,
|
|
291
574
|
verbose: options.verbose ?? false,
|
|
@@ -311,20 +594,32 @@ async function runCommand$1(files, options) {
|
|
|
311
594
|
logError("Coverage requires c8. Install with: npm install -D c8");
|
|
312
595
|
process.exit(1);
|
|
313
596
|
}
|
|
597
|
+
let reporters = options.coverageReporter ?? globalConfig.coverage?.reporters;
|
|
598
|
+
if (options.mergeLcov) {
|
|
599
|
+
if (!reporters) reporters = [
|
|
600
|
+
"text",
|
|
601
|
+
"html",
|
|
602
|
+
"lcov"
|
|
603
|
+
];
|
|
604
|
+
else if (!reporters.includes("lcov")) reporters = [...reporters, "lcov"];
|
|
605
|
+
}
|
|
314
606
|
coverageCtx = await createCoverageContext({
|
|
315
607
|
...globalConfig.coverage,
|
|
316
608
|
reportsDir: options.coverageDir ?? globalConfig.coverage?.reportsDir,
|
|
317
|
-
reporters
|
|
609
|
+
reporters,
|
|
318
610
|
exclude: options.coverageExclude ?? globalConfig.coverage?.exclude,
|
|
319
611
|
excludeNodeModules: options.coverageExcludeNodeModules ?? globalConfig.coverage?.excludeNodeModules,
|
|
320
612
|
excludeAfterRemap: options.coverageExcludeAfterRemap ?? globalConfig.coverage?.excludeAfterRemap,
|
|
321
613
|
skipFull: options.coverageSkipFull ?? globalConfig.coverage?.skipFull,
|
|
322
614
|
allowExternal: options.coverageAllowExternal ?? globalConfig.coverage?.allowExternal,
|
|
323
|
-
monocart: options.coverageMonocart ?? globalConfig.coverage?.monocart
|
|
615
|
+
monocart: options.coverageMonocart ?? globalConfig.coverage?.monocart,
|
|
616
|
+
mergeLcov: options.mergeLcov ?? globalConfig.coverage?.mergeLcov
|
|
324
617
|
});
|
|
325
618
|
coverageEnv = getCoverageEnv(coverageCtx);
|
|
326
619
|
}
|
|
327
620
|
const fileResults = [];
|
|
621
|
+
const fileContexts = /* @__PURE__ */ new Map();
|
|
622
|
+
const filePatterns = /* @__PURE__ */ new Map();
|
|
328
623
|
let shouldStop = false;
|
|
329
624
|
for (const filePath of testFiles) {
|
|
330
625
|
if (shouldStop) break;
|
|
@@ -340,6 +635,7 @@ async function runCommand$1(files, options) {
|
|
|
340
635
|
if (blocksToRun.length === 0) continue;
|
|
341
636
|
const ctx = await require_src.createExecutionContext(config, filePath, coverageEnv);
|
|
342
637
|
const results = [];
|
|
638
|
+
let fileContext;
|
|
343
639
|
try {
|
|
344
640
|
for (const block of blocksToRun) {
|
|
345
641
|
const result = await require_src.runBlock(block, ctx);
|
|
@@ -366,6 +662,12 @@ async function runCommand$1(files, options) {
|
|
|
366
662
|
}
|
|
367
663
|
}
|
|
368
664
|
await require_src.runAfterHook(ctx);
|
|
665
|
+
fileContext = {
|
|
666
|
+
root: ctx.testDir,
|
|
667
|
+
cwd: ctx.cwd
|
|
668
|
+
};
|
|
669
|
+
fileContexts.set(filePath, fileContext);
|
|
670
|
+
filePatterns.set(filePath, config.patterns ?? {});
|
|
369
671
|
} finally {
|
|
370
672
|
await require_src.cleanupExecutionContext(ctx);
|
|
371
673
|
}
|
|
@@ -381,7 +683,14 @@ async function runCommand$1(files, options) {
|
|
|
381
683
|
const { updated, changes } = await updateTestFile(testFile, results);
|
|
382
684
|
if (updated) console.error(colors.warn(` ${status.update} Updated: ${changes.join(", ")}`));
|
|
383
685
|
}
|
|
686
|
+
if (expandLevel && fileContext) {
|
|
687
|
+
const { expanded, expandedCount, changes } = await require_src.expandTestFile(testFile, results, expandLevel, fileContext, config.patterns ?? {});
|
|
688
|
+
if (expanded) console.error(colors.warn(` ${status.update} Expanded ${expandedCount} wildcard(s): ${changes.join(", ")}`));
|
|
689
|
+
}
|
|
384
690
|
}
|
|
691
|
+
let totalUnknownWildcards = 0;
|
|
692
|
+
for (const fr of fileResults) for (const block of fr.file.blocks) totalUnknownWildcards += countUnknownWildcards(block.expectedOutput);
|
|
693
|
+
if (totalUnknownWildcards > 0) logWarn(`${totalUnknownWildcards} unknown wildcard(s) found (??? or [??]). These are temporary and should be expanded. Use --expand to fill them in.`);
|
|
385
694
|
const summary = {
|
|
386
695
|
files: fileResults,
|
|
387
696
|
totalPassed: fileResults.reduce((sum, f) => sum + f.results.filter((r) => r.passed).length, 0),
|
|
@@ -390,11 +699,25 @@ async function runCommand$1(files, options) {
|
|
|
390
699
|
duration: Date.now() - startTime
|
|
391
700
|
};
|
|
392
701
|
reportSummary(summary, opts);
|
|
702
|
+
if (options.captureLog) try {
|
|
703
|
+
await require_src.writeCaptureLog(options.captureLog, fileResults, (file) => fileContexts.get(file.path) ?? {
|
|
704
|
+
root: process.cwd(),
|
|
705
|
+
cwd: process.cwd()
|
|
706
|
+
}, (file) => filePatterns.get(file.path) ?? {});
|
|
707
|
+
console.error(colors.info(`Capture log written to ${options.captureLog}`));
|
|
708
|
+
} catch (error) {
|
|
709
|
+
logError(`Failed to write capture log: ${error instanceof Error ? error.message : String(error)}`);
|
|
710
|
+
}
|
|
393
711
|
if (coverageCtx) {
|
|
394
712
|
console.error("\nGenerating coverage report...");
|
|
395
713
|
try {
|
|
396
714
|
await generateCoverageReport(coverageCtx);
|
|
397
715
|
console.error(colors.success(`Coverage report written to ${coverageCtx.options.reportsDir}/`));
|
|
716
|
+
if (coverageCtx.options.mergeLcov) {
|
|
717
|
+
console.error(`Merging with external coverage: ${coverageCtx.options.mergeLcov}`);
|
|
718
|
+
const merged = mergeExternalCoverage(coverageCtx.options.reportsDir, coverageCtx.options.mergeLcov);
|
|
719
|
+
if (merged) console.error(colors.success(`Merged coverage: ${merged.lines}% lines, ${merged.functions}% functions`));
|
|
720
|
+
}
|
|
398
721
|
} catch (error) {
|
|
399
722
|
logError(`Failed to generate coverage report: ${error instanceof Error ? error.message : String(error)}`);
|
|
400
723
|
} finally {
|
|
@@ -410,7 +733,7 @@ async function runCommand$1(files, options) {
|
|
|
410
733
|
* Register the coverage command.
|
|
411
734
|
*/
|
|
412
735
|
function registerCoverageCommand(program) {
|
|
413
|
-
program.command("coverage").description("Run commands with merged V8 coverage").argument("<commands...>", "Commands to run (each will inherit coverage environment)").option("--reports-dir <dir>", "Coverage output directory (default: coverage)").option("--reporters <reporters>", "Comma-separated coverage reporters (default: text,json,json-summary,lcov,html)").option("--include <patterns>", "Comma-separated patterns to include in coverage").option("--exclude <patterns>", "Comma-separated patterns to exclude from coverage").option("--exclude-node-modules", "Exclude node_modules from coverage (default: true)", true).option("--no-exclude-node-modules", "Include node_modules in coverage").option("--exclude-after-remap", "Apply exclude logic after sourcemap remapping").option("--skip-full", "Hide files with 100% coverage").option("--allow-external", "Allow files from outside cwd").option("--monocart", "Use monocart for accurate line counts (recommended for merging)").option("--src <dir>", "Source directory for sourcemap remapping (default: src)").option("--verbose", "Show coverage summary after each command for debugging").action(coverageCommand);
|
|
736
|
+
program.command("coverage").description("Run commands with merged V8 coverage").argument("<commands...>", "Commands to run (each will inherit coverage environment)").option("--reports-dir <dir>", "Coverage output directory (default: coverage)").option("--reporters <reporters>", "Comma-separated coverage reporters (default: text,json,json-summary,lcov,html)").option("--include <patterns>", "Comma-separated patterns to include in coverage").option("--exclude <patterns>", "Comma-separated patterns to exclude from coverage").option("--exclude-node-modules", "Exclude node_modules from coverage (default: true)", true).option("--no-exclude-node-modules", "Include node_modules in coverage").option("--exclude-after-remap", "Apply exclude logic after sourcemap remapping").option("--skip-full", "Hide files with 100% coverage").option("--allow-external", "Allow files from outside cwd").option("--monocart", "Use monocart for accurate line counts (recommended for merging)").option("--src <dir>", "Source directory for sourcemap remapping (default: src)").option("--verbose", "Show coverage summary after each command for debugging").option("--merge-lcov <path>", "Merge coverage from an existing LCOV file (e.g., from vitest --coverage)").action(coverageCommand);
|
|
414
737
|
}
|
|
415
738
|
/**
|
|
416
739
|
* Run a command with inherited coverage environment.
|
|
@@ -606,12 +929,32 @@ async function coverageCommand(commands, options) {
|
|
|
606
929
|
if (parsedOptions.verbose && stats.fileCount > 0) await generateTextReport(coverageTemp, parsedOptions, command);
|
|
607
930
|
previousFileCount = stats.fileCount;
|
|
608
931
|
}
|
|
609
|
-
console.error(colors.info("\n=== Generating
|
|
932
|
+
console.error(colors.info("\n=== Generating coverage report ==="));
|
|
610
933
|
if (!await generateReport(coverageTemp, parsedOptions)) {
|
|
611
934
|
logError("Failed to generate coverage report");
|
|
612
935
|
process.exit(1);
|
|
613
936
|
}
|
|
614
|
-
|
|
937
|
+
const reportsDir = parsedOptions.reportsDir ?? "coverage";
|
|
938
|
+
if (parsedOptions.mergeLcov) {
|
|
939
|
+
const externalLcovPath = parsedOptions.mergeLcov;
|
|
940
|
+
const generatedLcovPath = (0, node_path.join)(reportsDir, "lcov.info");
|
|
941
|
+
if (!(0, node_fs.existsSync)(externalLcovPath)) {
|
|
942
|
+
logError(`External LCOV file not found: ${externalLcovPath}`);
|
|
943
|
+
process.exit(1);
|
|
944
|
+
}
|
|
945
|
+
if (!(0, node_fs.existsSync)(generatedLcovPath)) {
|
|
946
|
+
logError(`Generated LCOV file not found: ${generatedLcovPath}`);
|
|
947
|
+
logError("Make sure \"lcov\" is included in reporters");
|
|
948
|
+
process.exit(1);
|
|
949
|
+
}
|
|
950
|
+
console.error(colors.info(`\nMerging with external coverage: ${externalLcovPath}`));
|
|
951
|
+
const mergedLcov = mergeLcov(readLcovFile(externalLcovPath), readLcovFile(generatedLcovPath));
|
|
952
|
+
writeLcovFile(generatedLcovPath, mergedLcov);
|
|
953
|
+
const summary = lcovToJsonSummary(mergedLcov);
|
|
954
|
+
writeJsonSummary((0, node_path.join)(reportsDir, "coverage-summary.json"), summary);
|
|
955
|
+
console.error(colors.success(`\nMerged coverage: ${summary.total.lines.pct}% lines, ${summary.total.functions.pct}% functions`));
|
|
956
|
+
}
|
|
957
|
+
console.error(colors.success(`\nCoverage report written to ${reportsDir}/`));
|
|
615
958
|
} finally {
|
|
616
959
|
await (0, node_fs_promises.rm)(coverageTemp, {
|
|
617
960
|
recursive: true,
|