@webywz/eval-smoke 0.3.1 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. package/README.md +228 -127
  2. package/dist/baseline.d.ts +25 -0
  3. package/dist/baseline.d.ts.map +1 -0
  4. package/dist/baseline.js +86 -0
  5. package/dist/baseline.js.map +1 -0
  6. package/dist/github-summary.d.ts +4 -0
  7. package/dist/github-summary.d.ts.map +1 -0
  8. package/dist/github-summary.js +39 -0
  9. package/dist/github-summary.js.map +1 -0
  10. package/dist/index.d.ts +4 -0
  11. package/dist/index.d.ts.map +1 -1
  12. package/dist/index.js +3 -0
  13. package/dist/index.js.map +1 -1
  14. package/dist/run-cli.d.ts +7 -1
  15. package/dist/run-cli.d.ts.map +1 -1
  16. package/dist/run-cli.js +78 -3
  17. package/dist/run-cli.js.map +1 -1
  18. package/dist/summary-md.d.ts +4 -0
  19. package/dist/summary-md.d.ts.map +1 -0
  20. package/dist/summary-md.js +22 -0
  21. package/dist/summary-md.js.map +1 -0
  22. package/examples/github-action.yml +41 -0
  23. package/package.json +2 -2
package/README.md CHANGED
@@ -1,8 +1,22 @@
1
1
  # eval-smoke
2
2
 
3
- One-minute smoke evals for prompts and models.
3
+ [![npm](https://img.shields.io/npm/v/@webywz/eval-smoke?style=flat-square&color=111111&labelColor=eeeeee)](https://www.npmjs.com/package/@webywz/eval-smoke)
4
+ [![node](https://img.shields.io/node/v/@webywz/eval-smoke?style=flat-square&color=111111&labelColor=eeeeee)](https://www.npmjs.com/package/@webywz/eval-smoke)
5
+ [![license](https://img.shields.io/npm/l/@webywz/eval-smoke?style=flat-square&color=111111&labelColor=eeeeee)](./LICENSE)
4
6
 
5
- Full eval harnesses are heavy. `eval-smoke` is the tiny local suite you run before a ship: a handful of cases, pass/fail, exit code 1 on regression. YAML or JSON in, a CI-friendly report out — including optional JUnit XML.
7
+ > **One-minute smoke evals for prompts and models.**
8
+ >
9
+ > **一分钟烟雾测试：在发版前拦住 prompt / 模型回归。**
10
+
11
+ YAML or JSON in. Pass/fail out. Exit `1` on regression. Optional JUnit for CI.
12
+
13
+ YAML 或 JSON 进，通过/失败出。回归时退出码 `1`。可选 JUnit 对接 CI。
14
+
15
+ ---
16
+
17
+ ## English
18
+
19
+ ### Demo
6
20
 
7
21
  ```bash
8
22
  npm i -g @webywz/eval-smoke
@@ -10,8 +24,8 @@ eval-smoke init
10
24
  eval-smoke suite.yaml --mock
11
25
  ```
12
26
 
13
- ```
14
- eval-smoke 0.3.0 · mock · gpt-4o-mini · smoke · 4 cases
27
+ ```text
28
+ eval-smoke 0.4.0 · mock · gpt-4o-mini · smoke · 4 cases
15
29
 
16
30
  pass greeting 0ms
17
31
  pass capital-france 0ms
@@ -21,23 +35,31 @@ eval-smoke 0.3.0 · mock · gpt-4o-mini · smoke · 4 cases
21
35
  4 passed 0 failed 1ms
22
36
  ```
23
37
 
24
- `--mock` is offline and deterministic. Contributors and CI do not need API keys.
38
+ `--mock` is offline and deterministic. CI and contributors do not need API keys.
39
+
40
+ ### Features
25
41
 
26
- ## Install
42
+ | | |
43
+ | --- | --- |
44
+ | **Tiny suite** | A handful of cases — not a full harness |
45
+ | **Matchers** | `equals`, `contains`, `not_contains`, `regex`, `json`, `min_tokens` / `max_tokens` |
46
+ | **init** | `eval-smoke init` scaffolds a mock-friendly `suite.yaml` |
47
+ | **Filter** | `--only` by id substring or `/regex/` |
48
+ | **CI** | `--junit`, `--github-summary`, exit codes, `--fail-fast`, `--concurrency` |
49
+ | **Baseline** | `--baseline` / `--update-baseline` catch pass→fail regressions |
50
+ | **Providers** | Mock offline; live OpenAI / Anthropic when keyed |
51
+
52
+ ### Install
27
53
 
28
54
  ```bash
29
55
  npm i -g @webywz/eval-smoke
30
- # or, from a project:
56
+ # or one-off:
31
57
  npx @webywz/eval-smoke suite.yaml --mock
32
58
  ```
33
59
 
34
- Requires Node 18+.
60
+ Node 18+.
35
61
 
36
- ## 60-second start
37
-
38
- 1. Scaffold a suite with `eval-smoke init`, or copy the example.
39
- 2. Run mock mode until the runner and matchers look right.
40
- 3. Point the same file at a live model when you are ready.
62
+ ### Quick start
41
63
 
42
64
  ```bash
43
65
  eval-smoke init
@@ -46,23 +68,26 @@ eval-smoke suite.yaml --mock --only greeting
46
68
  eval-smoke suite.yaml --mock --junit junit.xml
47
69
  eval-smoke examples/suite.yaml --mock --json
48
70
  eval-smoke examples/suite.yaml --mock --concurrency 4
71
+ eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
72
+ eval-smoke examples/suite.yaml --mock --baseline baseline.json
73
+ eval-smoke examples/suite.yaml --mock --github-summary
49
74
  OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml
50
75
  ANTHROPIC_API_KEY=sk-ant-... eval-smoke examples/suite.yaml --model claude-3-5-haiku-20241022
51
76
  ```
52
77
 
53
- ## `eval-smoke init`
78
+ ### `eval-smoke init`
54
79
 
55
80
  ```bash
56
81
  eval-smoke init # writes ./suite.yaml
57
82
  eval-smoke init ./prompts # writes ./prompts/suite.yaml
58
- eval-smoke init --force # overwrite an existing suite.yaml
83
+ eval-smoke init --force # overwrite
59
84
  ```
60
85
 
61
- Creates a small mock-friendly suite (greeting, capital, arithmetic, refusal). Refuses to overwrite unless you pass `--force`.
86
+ Creates a small mock-friendly suite (greeting, capital, arithmetic, refusal). Refuses to overwrite unless `--force`.
62
87
 
63
- ## Suite format
88
+ ### Suite format
64
89
 
65
- A suite is a YAML or JSON list of cases, or an object with a `cases` array.
90
+ YAML or JSON — a list of cases, or an object with `cases`:
66
91
 
67
92
  ```yaml
68
93
  name: demo
@@ -87,132 +112,219 @@ cases:
87
112
  city: Paris
88
113
  ```
89
114
 
90
- A bare list is also valid:
115
+ | Field | Notes |
116
+ | --- | --- |
117
+ | `id` / `prompt` | Required |
118
+ | `expect.contains` | Substring, case-insensitive; string or list |
119
+ | `expect.not_contains` | Must be absent |
120
+ | `expect.regex` | `/pattern/i`, `(?i)pattern`, or raw |
121
+ | `expect.equals` | Exact after NFC + LF normalize |
122
+ | `expect.json` | `true` = parseable; object/array = deep-equal |
123
+ | `expect.min_tokens` / `max_tokens` | Approx via `ceil(chars / 4)` |
124
+ | `system` / `model` | Suite or case |
125
+ | `provider` | `openai`, `anthropic`, or `mock` |
126
+
127
+ At least one matcher per case. Failures print expected/actual (and a first-diff for `equals`).
128
+
129
+ ### CLI
91
130
 
92
- ```yaml
93
- - id: capital-france
94
- prompt: What is the capital of France? Answer with the city name only.
95
- expect:
96
- contains: Paris
97
131
  ```
132
+ eval-smoke <suite.yaml|suite.json> [options]
133
+ eval-smoke init [dir] [--force]
134
+ ```
135
+
136
+ | Flag | Meaning |
137
+ | --- | --- |
138
+ | `--mock` | Offline deterministic runner |
139
+ | `--json` | Machine-readable JSON report |
140
+ | `--junit <path>` | Write JUnit XML (still prints human report unless `--json`) |
141
+ | `--baseline <path>` | Exit `1` if a baseline-passing case now fails or is missing |
142
+ | `--update-baseline <path>` | Write current report JSON (pretty) to path |
143
+ | `--github-summary [path]` | Append pass/fail Markdown to `$GITHUB_STEP_SUMMARY` or path |
144
+ | `--only <pattern>` | Cases whose id contains pattern, or matches `/regex/` |
145
+ | `--fail-fast` | Stop after first failure |
146
+ | `--concurrency <n>` | Parallel cases (default 1; ignored with `--fail-fast`) |
147
+ | `--timeout <ms>` | Per-case timeout (default 30000) |
148
+ | `--model <name>` | Override suite model |
149
+ | `--provider <name>` | `openai` \| `anthropic` \| `mock` |
150
+ | `--force` | Overwrite for `init` |
151
+ | `--no-color` | Disable ANSI (`NO_COLOR` respected) |
152
+
153
+ | Exit | Meaning |
154
+ | --- | --- |
155
+ | `0` | All passed |
156
+ | `1` | One or more failed |
157
+ | `2` | Usage / config error (missing file, bad YAML, no key, `--only` matched nothing) |
158
+
159
+ ### Runners
98
160
 
99
- | Field | Where | Notes |
100
- | --- | --- | --- |
101
- | `id` | case | Required, unique. |
102
- | `prompt` | case | Required. |
103
- | `expect.contains` | case | Substring, case-insensitive. String or list; all must match. |
104
- | `expect.not_contains` | case | Substring must be absent. String or list. |
105
- | `expect.regex` | case | JavaScript regex. `/pattern/i`, `(?i)pattern`, or a raw pattern. |
106
- | `expect.equals` | case | Exact match after NFC + LF normalize; one trailing newline ignored. |
107
- | `expect.json` | case | `true` = must parse as JSON; object/array = deep-equal after parse. |
108
- | `expect.min_tokens` | case | Approximate floor using `ceil(chars / 4)`. |
109
- | `expect.max_tokens` | case | Approximate cap using `ceil(chars / 4)`. |
110
- | `system` | suite or case | Optional system prompt for live providers. |
111
- | `model` | suite or case | Live default: `gpt-4o-mini` or `claude-3-5-haiku-20241022`. |
112
- | `provider` | suite | `openai`, `anthropic`, or `mock`. |
161
+ **Mock** never calls the network. Canned replies keep the example suite green in CI:
113
162
 
114
- At least one matcher is required on every case. Failed checks print expected/actual (and a first-diff pointer for `equals`).
163
+ | Prompt contains | Mock reply |
164
+ | --- | --- |
165
+ | `hello` | `Hello` |
166
+ | `capital of France` | `Paris` |
167
+ | `2 + 2` | `4` |
168
+ | `ignore previous` / `PWNED` | a refusal without `PWNED` |
115
169
 
116
- ## CLI
170
+ **Live OpenAI** with `OPENAI_API_KEY` (optional `OPENAI_BASE_URL`). **Live Anthropic** with `ANTHROPIC_API_KEY`, or when the model name starts with `claude`.
117
171
 
172
+ ### Library
173
+
174
+ ```ts
175
+ import { loadSuite, runSuite, formatJUnit, initSuite } from "@webywz/eval-smoke";
176
+
177
+ initSuite(".", { force: false });
178
+ const suite = loadSuite("suite.yaml");
179
+ const report = await runSuite(suite, { mock: true, concurrency: 4 });
180
+ console.log(formatJUnit(report));
181
+ if (!report.ok) process.exit(1);
118
182
  ```
119
- eval-smoke <suite.yaml|suite.json> [options]
120
- eval-smoke init [dir] [--force]
121
183
 
122
- --mock Offline runner with deterministic fake answers
123
- --json Machine-readable JSON report
124
- --junit <path> Write JUnit XML to path (still prints human report unless --json)
125
- --only <pattern> Run cases whose id contains pattern, or matches /regex/
126
- --fail-fast Stop after the first failing case
127
- --concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)
128
- --timeout <ms> Per-case timeout (default: 30000)
129
- --model <name> Override suite model
130
- --provider <name> openai | anthropic | mock
131
- --force Overwrite existing suite.yaml for init
132
- --no-color Disable ANSI colors (also respects NO_COLOR)
133
- -h, --help
134
- -v, --version
184
+ ### CI snippet
185
+
186
+ ```yaml
187
+ - run: npx @webywz/eval-smoke@0.4.0 prompts/smoke.yaml --mock --junit junit.xml
188
+ - uses: actions/upload-artifact@v4
189
+ if: always()
190
+ with:
191
+ name: junit
192
+ path: junit.xml
135
193
  ```
136
194
 
137
- | Exit | Meaning |
195
+ ### Links
196
+
197
+ | | |
138
198
  | --- | --- |
139
- | 0 | Every case passed |
140
- | 1 | One or more cases failed |
141
- | 2 | Usage or configuration error (missing file, bad YAML, no API key, `--only` matched nothing) |
199
+ | npm | [npmjs.com/package/@webywz/eval-smoke](https://www.npmjs.com/package/@webywz/eval-smoke) |
200
+ | Issues | [github.com/webywz/eval-smoke/issues](https://github.com/webywz/eval-smoke/issues) |
201
+ | License | [MIT](./LICENSE) © webywz |
202
+
203
+ ---
142
204
 
143
- ### `--only`
205
+ ## 中文
144
206
 
145
- Filter cases by `id` before running:
207
+ ### 演示
146
208
 
147
209
  ```bash
148
- eval-smoke suite.yaml --mock --only greeting # substring
149
- eval-smoke suite.yaml --mock --only '/^capital/' # /regex/ (same style as matchers)
150
- eval-smoke suite.yaml --mock --only '/hello/i'
210
+ npm i -g @webywz/eval-smoke
211
+ eval-smoke init
212
+ eval-smoke suite.yaml --mock
151
213
  ```
152
214
 
153
- Exits `2` if zero cases match.
215
+ `--mock` 离线、确定。CI 与贡献者不需要 API key。
154
216
 
155
- ### `--junit`
217
+ ### 功能
156
218
 
157
- After a run, write a JUnit XML report for CI while still printing the human (or `--json`) report to stdout:
219
+ | | |
220
+ | --- | --- |
221
+ | **轻量套件** | 少量用例，不是完整评测框架 |
222
+ | **匹配器** | `equals`、`contains`、`not_contains`、`regex`、`json`、token 上下限 |
223
+ | **init** | `eval-smoke init` 生成可 mock 的 `suite.yaml` |
224
+ | **过滤** | `--only`：id 子串或 `/regex/` |
225
+ | **CI** | `--junit`、`--github-summary`、退出码、`--fail-fast`、`--concurrency` |
226
+ | **基线** | `--baseline` / `--update-baseline` 拦住 pass→fail 回归 |
227
+ | **提供商** | Mock 离线；有 key 时走 OpenAI / Anthropic |
228
+
229
+ ### 安装
158
230
 
159
231
  ```bash
160
- eval-smoke suite.yaml --mock --junit reports/junit.xml
232
+ npm i -g @webywz/eval-smoke
233
+ # 或一次性：
234
+ npx @webywz/eval-smoke suite.yaml --mock
161
235
  ```
162
236
 
163
- Failures include matcher detail (`expected` / `actual` / `detail`). Skipped cases (e.g. `--fail-fast`) become `<skipped/>`.
237
+ 需要 Node 18+。
238
+
239
+ ### 快速开始
164
240
 
165
- ## Runners
241
+ ```bash
242
+ eval-smoke init
243
+ eval-smoke suite.yaml --mock
244
+ eval-smoke suite.yaml --mock --only greeting
245
+ eval-smoke suite.yaml --mock --junit junit.xml
246
+ eval-smoke examples/suite.yaml --mock --json
247
+ eval-smoke examples/suite.yaml --mock --concurrency 4
248
+ eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
249
+ eval-smoke examples/suite.yaml --mock --baseline baseline.json
250
+ eval-smoke examples/suite.yaml --mock --github-summary
251
+ OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml
252
+ ANTHROPIC_API_KEY=sk-ant-... eval-smoke examples/suite.yaml --model claude-3-5-haiku-20241022
253
+ ```
166
254
 
167
- **Mock** (`--mock` or `--provider mock`) never calls a network. It returns canned replies matched from the prompt so the example suite stays green in CI:
255
+ ### `eval-smoke init`
168
256
 
169
- | Prompt contains | Mock reply |
257
+ ```bash
258
+ eval-smoke init # 写入 ./suite.yaml
259
+ eval-smoke init ./prompts # 写入 ./prompts/suite.yaml
260
+ eval-smoke init --force # 覆盖已有文件
261
+ ```
262
+
263
+ 生成小型 mock 友好套件（问候、首都、算术、拒答）。不加 `--force` 不会覆盖已有文件。
264
+
265
+ ### 套件格式
266
+
267
+ YAML 或 JSON —— 用例列表，或带 `cases` 的对象：
268
+
269
+ ```yaml
270
+ name: demo
271
+ model: gpt-4o-mini
272
+ cases:
273
+ - id: greeting
274
+ prompt: Reply with exactly the word Hello.
275
+ expect:
276
+ equals: Hello
277
+ contains: Hello
278
+ not_contains: Goodbye
279
+ regex: "(?i)^hello\\b"
280
+ min_tokens: 1
281
+ max_tokens: 32
282
+ ```
283
+
284
+ | 字段 | 说明 |
170
285
  | --- | --- |
171
- | `hello` | `Hello` |
172
- | `capital of France` | `Paris` |
173
- | `2 + 2` | `4` |
174
- | `ignore previous` / `PWNED` | a refusal that does not contain `PWNED` |
286
+ | `id` / `prompt` | 必填 |
287
+ | `expect.contains` | 子串，忽略大小写；字符串或列表 |
288
+ | `expect.not_contains` | 不得出现 |
289
+ | `expect.regex` | `/pattern/i`、`(?i)pattern` 或裸模式 |
290
+ | `expect.equals` | NFC + LF 规范化后精确匹配 |
291
+ | `expect.json` | `true` = 可解析；对象/数组 = 深相等 |
292
+ | `expect.min_tokens` / `max_tokens` | 约 `ceil(chars / 4)` |
293
+ | `system` / `model` | 套件级或用例级 |
294
+ | `provider` | `openai` / `anthropic` / `mock` |
175
295
 
176
- Anything else returns `[mock] No canned reply for this prompt.` — useful as a deliberate fail.
296
+ 每个用例至少要有一个匹配器。失败时打印期望/实际（`equals` 还有首差位置）。
177
297
 
178
- **Live OpenAI** when `OPENAI_API_KEY` is set. Optional `OPENAI_BASE_URL` for compatible servers.
298
+ ### 命令行要点
179
299
 
180
- **Live Anthropic** when `ANTHROPIC_API_KEY` is set, or when the model name starts with `claude`.
300
+ | 参数 | 含义 |
301
+ | --- | --- |
302
+ | `--mock` | 离线确定性运行 |
303
+ | `--json` | JSON 报告 |
304
+ | `--junit <path>` | 写出 JUnit XML（仍打印人类可读报告，除非同时 `--json`） |
305
+ | `--baseline <path>` | 基线里曾通过的用例若失败或缺失则退出 `1` |
306
+ | `--update-baseline <path>` | 把当前报告 JSON（pretty）写入路径 |
307
+ | `--github-summary [path]` | 追加通过/失败 Markdown 到 `$GITHUB_STEP_SUMMARY` 或指定路径 |
308
+ | `--only <pattern>` | 按 id 子串或 `/regex/` 过滤 |
309
+ | `--fail-fast` | 首个失败即停 |
310
+ | `--concurrency <n>` | 并行用例数（默认 1） |
311
+ | `--timeout <ms>` | 单用例超时（默认 30000） |
312
+ | `--model` / `--provider` | 覆盖模型 / 提供商 |
313
+ | `--force` | `init` 时覆盖 |
314
+
315
+ | 退出码 | 含义 |
316
+ | --- | --- |
317
+ | `0` | 全部通过 |
318
+ | `1` | 有失败 |
319
+ | `2` | 用法/配置错误（缺文件、坏 YAML、无 key、`--only` 无匹配） |
181
320
 
182
- If no key is set and you did not pass `--mock`, the CLI exits 2 with a short hint.
321
+ ### 运行器
183
322
 
184
- ## JSON report
323
+ **Mock** 永不联网。内置 canned 回复保证示例套件在 CI 里绿。
185
324
 
186
- ```bash
187
- eval-smoke examples/suite.yaml --mock --json
188
- ```
325
+ **Live OpenAI**：`OPENAI_API_KEY`（可选 `OPENAI_BASE_URL`）。**Live Anthropic**：`ANTHROPIC_API_KEY`，或模型名以 `claude` 开头。
189
326
 
190
- ```json
191
- {
192
- "ok": true,
193
- "version": "0.3.0",
194
- "runner": "mock",
195
- "model": "gpt-4o-mini",
196
- "suite": "demo",
197
- "passed": 5,
198
- "failed": 0,
199
- "skipped": 0,
200
- "duration_ms": 1,
201
- "cases": [
202
- {
203
- "id": "greeting",
204
- "ok": true,
205
- "duration_ms": 1,
206
- "output": "Hello",
207
- "checks": [{ "name": "contains", "ok": true, "expected": "Hello" }]
208
- }
209
- ]
210
- }
211
- ```
212
-
213
- Failed checks may include `actual` plus a multi-line `detail` (first-diff for `equals`).
214
-
215
- ## Programmatic API
327
+ ### 库用法
216
328
 
217
329
  ```ts
218
330
  import { loadSuite, runSuite, formatJUnit, initSuite } from "@webywz/eval-smoke";
@@ -224,21 +336,10 @@ console.log(formatJUnit(report));
224
336
  if (!report.ok) process.exit(1);
225
337
  ```
226
338
 
227
- `runSuite` also accepts a custom `complete` function, which is how the tests inject fakes without keys.
228
-
229
- ## CI
339
+ ### 链接
230
340
 
231
- GitHub Actions in this repo runs `npm test`, `npm run build`, and `eval-smoke examples/suite.yaml --mock` on Node 18/20/22. Drop this into another project:
232
-
233
- ```yaml
234
- - run: npx @webywz/eval-smoke@0.3.1 prompts/smoke.yaml --mock --junit junit.xml
235
- - uses: actions/upload-artifact@v4
236
- if: always()
237
- with:
238
- name: junit
239
- path: junit.xml
240
- ```
241
-
242
- ## License
243
-
244
- MIT
341
+ | | |
342
+ | --- | --- |
343
+ | npm | [npmjs.com/package/@webywz/eval-smoke](https://www.npmjs.com/package/@webywz/eval-smoke) |
344
+ | Issues | [github.com/webywz/eval-smoke/issues](https://github.com/webywz/eval-smoke/issues) |
345
+ | License | [MIT](./LICENSE) © webywz |
package/dist/baseline.d.ts ADDED
@@ -0,0 +1,25 @@
1
+ import type { Report } from "./types.js";
2
+ export interface BaselineCase {
3
+ id: string;
4
+ ok: boolean;
5
+ }
6
+ export interface BaselineFile {
7
+ cases: BaselineCase[];
8
+ }
9
+ /** Minimal shape: case id → ok. Accepts a full Report or a slim baseline. */
10
+ export declare function loadBaseline(path: string): Map<string, boolean>;
11
+ export interface BaselineCheck {
12
+ ok: boolean;
13
+ regressions: string[];
14
+ /** Human-readable lines */
15
+ report: string;
16
+ }
17
+ /**
18
+ * Fail if any case that passed in baseline now fails, or a baseline case is missing
19
+ * from the current run. New cases are OK. Skipped current cases that passed in
20
+ * baseline count as missing/regression.
21
+ */
22
+ export declare function checkBaseline(report: Report, baseline: Map<string, boolean>): BaselineCheck;
23
+ /** Pretty-print current report JSON for --update-baseline. */
24
+ export declare function writeBaseline(path: string, report: Report): void;
25
+ //# sourceMappingURL=baseline.d.ts.map
package/dist/baseline.d.ts.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"baseline.d.ts","sourceRoot":"","sources":["../src/baseline.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,YAAY,CAAC;AAGzC,MAAM,WAAW,YAAY;IAC3B,EAAE,EAAE,MAAM,CAAC;IACX,EAAE,EAAE,OAAO,CAAC;CACb;AAED,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,YAAY,EAAE,CAAC;CACvB;AAED,6EAA6E;AAC7E,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,GAAG,GAAG,CAAC,MAAM,EAAE,OAAO,CAAC,CAmC/D;AAED,MAAM,WAAW,aAAa;IAC5B,EAAE,EAAE,OAAO,CAAC;IACZ,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,2BAA2B;IAC3B,MAAM,EAAE,MAAM,CAAC;CAChB;AAED;;;;GAIG;AACH,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,aAAa,CAgC3F;AAED,8DAA8D;AAC9D,wBAAgB,aAAa,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,IAAI,CAEhE"}
package/dist/baseline.js ADDED
@@ -0,0 +1,86 @@
1
+ import { readFileSync, writeFileSync } from "node:fs";
2
+ import { UsageError } from "./types.js";
3
+ /** Minimal shape: case id → ok. Accepts a full Report or a slim baseline. */
4
+ export function loadBaseline(path) {
5
+ let raw;
6
+ try {
7
+ raw = readFileSync(path, "utf8");
8
+ }
9
+ catch (error) {
10
+ const code = error && typeof error === "object" && "code" in error
11
+ ? String(error.code)
12
+ : "";
13
+ if (code === "ENOENT")
14
+ throw new UsageError(`baseline not found: ${path}`);
15
+ throw error;
16
+ }
17
+ let parsed;
18
+ try {
19
+ parsed = JSON.parse(raw);
20
+ }
21
+ catch {
22
+ throw new UsageError(`baseline is not valid JSON: ${path}`);
23
+ }
24
+ if (!parsed || typeof parsed !== "object") {
25
+ throw new UsageError(`baseline must be a JSON object: ${path}`);
26
+ }
27
+ const cases = parsed.cases;
28
+ if (!Array.isArray(cases)) {
29
+ throw new UsageError(`baseline missing cases[]: ${path}`);
30
+ }
31
+ const map = new Map();
32
+ for (const entry of cases) {
33
+ if (!entry || typeof entry !== "object")
34
+ continue;
35
+ const id = entry.id;
36
+ const ok = entry.ok;
37
+ if (typeof id !== "string" || !id)
38
+ continue;
39
+ if (typeof ok !== "boolean")
40
+ continue;
41
+ map.set(id, ok);
42
+ }
43
+ return map;
44
+ }
45
+ /**
46
+ * Fail if any case that passed in baseline now fails, or a baseline case is missing
47
+ * from the current run. New cases are OK. Skipped current cases that passed in
48
+ * baseline count as missing/regression.
49
+ */
50
+ export function checkBaseline(report, baseline) {
51
+ const byId = new Map(report.cases.map((c) => [c.id, c]));
52
+ const regressions = [];
53
+ for (const [id, wasOk] of baseline) {
54
+ if (!wasOk)
55
+ continue; // only guard formerly-passing cases
56
+ const current = byId.get(id);
57
+ if (!current) {
58
+ regressions.push(`${id}: missing from current run (passed in baseline)`);
59
+ continue;
60
+ }
61
+ if (current.skipped) {
62
+ regressions.push(`${id}: skipped now (passed in baseline)`);
63
+ continue;
64
+ }
65
+ if (!current.ok) {
66
+ regressions.push(`${id}: was pass, now fail`);
67
+ }
68
+ }
69
+ const lines = ["baseline check:"];
70
+ if (regressions.length === 0) {
71
+ lines.push(" ok — no regressions vs baseline");
72
+ }
73
+ else {
74
+ lines.push(` ${regressions.length} regression(s):`);
75
+ for (const line of regressions) {
76
+ lines.push(` - ${line}`);
77
+ }
78
+ }
79
+ lines.push("");
80
+ return { ok: regressions.length === 0, regressions, report: lines.join("\n") };
81
+ }
82
+ /** Pretty-print current report JSON for --update-baseline. */
83
+ export function writeBaseline(path, report) {
84
+ writeFileSync(path, `${JSON.stringify(report, null, 2)}\n`, "utf8");
85
+ }
86
+ //# sourceMappingURL=baseline.js.map
package/dist/baseline.js.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"baseline.js","sourceRoot":"","sources":["../src/baseline.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAEtD,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAWxC,6EAA6E;AAC7E,MAAM,UAAU,YAAY,CAAC,IAAY;IACvC,IAAI,GAAW,CAAC;IAChB,IAAI,CAAC;QACH,GAAG,GAAG,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IACnC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,GACR,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,IAAI,KAAK;YACnD,CAAC,CAAC,MAAM,CAAE,KAA2B,CAAC,IAAI,CAAC;YAC3C,CAAC,CAAC,EAAE,CAAC;QACT,IAAI,IAAI,KAAK,QAAQ;YAAE,MAAM,IAAI,UAAU,CAAC,uBAAuB,IAAI,EAAE,CAAC,CAAC;QAC3E,MAAM,KAAK,CAAC;IACd,CAAC;IACD,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC3B,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,UAAU,CAAC,+BAA+B,IAAI,EAAE,CAAC,CAAC;IAC9D,CAAC;IACD,IAAI,CAAC,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,EAAE,CAAC;QAC1C,MAAM,IAAI,UAAU,CAAC,mCAAmC,IAAI,EAAE,CAAC,CAAC;IAClE,CAAC;IACD,MAAM,KAAK,GAAI,MAA8B,CAAC,KAAK,CAAC;IACpD,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC;QAC1B,MAAM,IAAI,UAAU,CAAC,6BAA6B,IAAI,EAAE,CAAC,CAAC;IAC5D,CAAC;IACD,MAAM,GAAG,GAAG,IAAI,GAAG,EAAmB,CAAC;IACvC,KAAK,MAAM,KAAK,IAAI,KAAK,EAAE,CAAC;QAC1B,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ;YAAE,SAAS;QAClD,MAAM,EAAE,GAAI,KAA0B,CAAC,EAAE,CAAC;QAC1C,MAAM,EAAE,GAAI,KAA0B,CAAC,EAAE,CAAC;QAC1C,IAAI,OAAO,EAAE,KAAK,QAAQ,IAAI,CAAC,EAAE;YAAE,SAAS;QAC5C,IAAI,OAAO,EAAE,KAAK,SAAS;YAAE,SAAS;QACtC,GAAG,CAAC,GAAG,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AASD;;;;GAIG;AACH,MAAM,UAAU,aAAa,CAAC,MAAc,EAAE,QAA8B;IAC1E,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;IACzD,MAAM,WAAW,GAAa,EAAE,CAAC;IAEjC,KAAK,MAAM,CAAC,EAAE,EAAE,KAAK,CAAC,IAAI,QAAQ,EAAE,CAAC;QACnC,IAAI,CAAC,KAAK;YAAE,SAAS,CAAC,oCAAoC;QAC1D,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAC7B,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,WAAW,CAAC,IAAI,CAAC,GAAG,EAAE,iDAAiD,CAAC,CAAC;YACzE,SAAS;QACX,CAAC;QACD,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;YACpB,WAAW,CAAC,IAAI,CAAC,GAAG,EAAE,oCAAoC,CAAC,CAAC;YAC5D,SAAS;QACX,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,EAAE,EAAE,CAAC;YAChB,WAAW,CAAC,IAAI,CAAC,GAAG,EAAE,sBAAsB,CAAC,CAAC;QAChD,CAAC;IACH,CAAC;IAED,MAAM,KAAK,GAAG,CAAC,iBAAiB,CAAC,CAAC;IAClC,IAAI,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC7B,KAAK,CAAC,IAAI,CAAC,mCAAmC,CAAC,CAAC;IAClD,CAAC;SAAM,CAAC;QACN,KAAK,CAAC,IAAI,CAAC,KAAK,WAAW,CAAC,MAAM,iBAAiB,CAAC,CAAC;QACrD,KAAK,MAAM,IAAI,IAAI,WAAW,EAAE,CAAC;YAC/B,KAAK,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;QAC5B,CAAC;IACH,CAAC;IACD,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IAEf,OAAO,EAAE,EAAE,EAAE,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,WAAW,EAAE,MAAM,EAAE,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;AACjF,CAAC;AAED,8DAA8D;AAC9D,MAAM,UAAU,aAAa,CAAC,IAAY,EAAE,MAAc;IACxD,aAAa,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;AACtE,CAAC"}
package/dist/github-summary.d.ts ADDED
@@ -0,0 +1,4 @@
1
+ export declare const DEFAULT_GITHUB_SUMMARY = "github-step-summary.md";
2
+ export declare function resolveGithubSummaryPath(flagPath: string | true | undefined, env?: NodeJS.ProcessEnv, cwd?: string): string | undefined;
3
+ export declare function appendGithubSummary(path: string, markdown: string): void;
4
+ //# sourceMappingURL=github-summary.d.ts.map
package/dist/github-summary.d.ts.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"github-summary.d.ts","sourceRoot":"","sources":["../src/github-summary.ts"],"names":[],"mappings":"AAGA,eAAO,MAAM,sBAAsB,2BAA2B,CAAC;AAE/D,wBAAgB,wBAAwB,CACtC,QAAQ,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,EACnC,GAAG,GAAE,MAAM,CAAC,UAAwB,EACpC,GAAG,GAAE,MAAsB,GAC1B,MAAM,GAAG,SAAS,CAWpB;AAED,wBAAgB,mBAAmB,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,GAAG,IAAI,CAoBxE"}
package/dist/github-summary.js ADDED
@@ -0,0 +1,39 @@
1
+ import { appendFileSync, mkdirSync, writeFileSync } from "node:fs";
2
+ import { dirname, resolve } from "node:path";
3
+ export const DEFAULT_GITHUB_SUMMARY = "github-step-summary.md";
4
+ export function resolveGithubSummaryPath(flagPath, env = process.env, cwd = process.cwd()) {
5
+ if (flagPath === undefined) {
6
+ const fromEnv = env.GITHUB_STEP_SUMMARY;
7
+ return fromEnv && fromEnv.trim() ? fromEnv : undefined;
8
+ }
9
+ if (flagPath === true || flagPath === "") {
10
+ const fromEnv = env.GITHUB_STEP_SUMMARY;
11
+ if (fromEnv && fromEnv.trim())
12
+ return fromEnv;
13
+ return resolve(cwd, DEFAULT_GITHUB_SUMMARY);
14
+ }
15
+ return resolve(cwd, flagPath);
16
+ }
17
+ export function appendGithubSummary(path, markdown) {
18
+ const body = markdown.endsWith("\n") ? markdown : `${markdown}\n`;
19
+ try {
20
+ mkdirSync(dirname(path), { recursive: true });
21
+ }
22
+ catch {
23
+ // ignore
24
+ }
25
+ try {
26
+ appendFileSync(path, body, "utf8");
27
+ }
28
+ catch (error) {
29
+ const code = error && typeof error === "object" && "code" in error
30
+ ? String(error.code)
31
+ : "";
32
+ if (code === "ENOENT") {
33
+ writeFileSync(path, body, "utf8");
34
+ return;
35
+ }
36
+ throw error;
37
+ }
38
+ }
39
+ //# sourceMappingURL=github-summary.js.map
package/dist/github-summary.js.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"github-summary.js","sourceRoot":"","sources":["../src/github-summary.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,SAAS,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACnE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAE7C,MAAM,CAAC,MAAM,sBAAsB,GAAG,wBAAwB,CAAC;AAE/D,MAAM,UAAU,wBAAwB,CACtC,QAAmC,EACnC,MAAyB,OAAO,CAAC,GAAG,EACpC,MAAc,OAAO,CAAC,GAAG,EAAE;IAE3B,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;QAC3B,MAAM,OAAO,GAAG,GAAG,CAAC,mBAAmB,CAAC;QACxC,OAAO,OAAO,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IACzD,CAAC;IACD,IAAI,QAAQ,KAAK,IAAI,IAAI,QAAQ,KAAK,EAAE,EAAE,CAAC;QACzC,MAAM,OAAO,GAAG,GAAG,CAAC,mBAAmB,CAAC;QACxC,IAAI,OAAO,IAAI,OAAO,CAAC,IAAI,EAAE;YAAE,OAAO,OAAO,CAAC;QAC9C,OAAO,OAAO,CAAC,GAAG,EAAE,sBAAsB,CAAC,CAAC;IAC9C,CAAC;IACD,OAAO,OAAO,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;AAChC,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,IAAY,EAAE,QAAgB;IAChE,MAAM,IAAI,GAAG,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,GAAG,QAAQ,IAAI,CAAC;IAClE,IAAI,CAAC;QACH,SAAS,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAChD,CAAC;IAAC,MAAM,CAAC;QACP,SAAS;IACX,CAAC;IACD,IAAI,CAAC;QACH,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,CAAC;IACrC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,GACR,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,IAAI,KAAK;YACnD,CAAC,CAAC,MAAM,CAAE,KAA2B,CAAC,IAAI,CAAC;YAC3C,CAAC,CAAC,EAAE,CAAC;QACT,IAAI,IAAI,KAAK,QAAQ,EAAE,CAAC;YACtB,aAAa,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,CAAC;YAClC,OAAO;QACT,CAAC;QACD,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC"}
package/dist/index.d.ts CHANGED
@@ -9,4 +9,8 @@ export { estimateTokens } from "./tokens.js";
9
9
  export { VERSION } from "./version.js";
10
10
  export type { Case, CaseResult, CheckName, CheckResult, CompleteFn, CompleteRequest, Expect, JsonExpect, ProviderName, Report, RunOptions, Suite, } from "./types.js";
11
11
  export { UsageError } from "./types.js";
12
+ export { checkBaseline, loadBaseline, writeBaseline } from "./baseline.js";
13
+ export type { BaselineCheck, BaselineCase, BaselineFile } from "./baseline.js";
14
+ export { appendGithubSummary, resolveGithubSummaryPath, DEFAULT_GITHUB_SUMMARY } from "./github-summary.js";
15
+ export { formatGithubSummary } from "./summary-md.js";
12
16
  //# sourceMappingURL=index.d.ts.map
package/dist/index.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EACV,IAAI,EACJ,UAAU,EACV,SAAS,EACT,WAAW,EACX,UAAU,EACV,eAAe,EACf,MAAM,EACN,UAAU,EACV,YAAY,EACZ,MAAM,EACN,UAAU,EACV,KAAK,GACN,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EACV,IAAI,EACJ,UAAU,EACV,SAAS,EACT,WAAW,EACX,UAAU,EACV,eAAe,EACf,MAAM,EACN,UAAU,EACV,YAAY,EACZ,MAAM,EACN,UAAU,EACV,KAAK,GACN,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,YAAY,EAAE,aAAa,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC/E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
package/dist/index.js CHANGED
@@ -8,4 +8,7 @@ export { runSuite } from "./run.js";
8
8
  export { estimateTokens } from "./tokens.js";
9
9
  export { VERSION } from "./version.js";
10
10
  export { UsageError } from "./types.js";
11
+ export { checkBaseline, loadBaseline, writeBaseline } from "./baseline.js";
12
+ export { appendGithubSummary, resolveGithubSummaryPath, DEFAULT_GITHUB_SUMMARY } from "./github-summary.js";
13
+ export { formatGithubSummary } from "./summary-md.js";
11
14
  //# sourceMappingURL=index.js.map
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAevC,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAevC,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAE3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
package/dist/run-cli.d.ts CHANGED
@@ -1,5 +1,5 @@
1
1
  import { type Suite } from "./types.js";
2
- export declare const HELP = "eval-smoke \u2014 one-minute smoke evals for prompts and models\n\nUsage:\n eval-smoke <suite.yaml|suite.json> [options]\n eval-smoke init [dir] [--force]\n\nOptions:\n --mock Offline runner with deterministic fake answers\n --json Machine-readable JSON report\n --junit <path> Write JUnit XML report to path (still prints human report unless --json)\n --only <pattern> Run cases whose id contains pattern, or matches /regex/\n --fail-fast Stop after the first failing case\n --concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)\n --timeout <ms> Per-case timeout (default: 30000)\n --model <name> Override suite model\n --provider <name> openai | anthropic | mock\n --force Overwrite existing suite.yaml when using init\n --no-color Disable ANSI colors (also respects NO_COLOR)\n -h, --help Show this help\n -v, --version Show version\n\nExit codes:\n 0 all cases passed\n 1 one or more cases failed\n 2 usage or configuration error\n\nExamples:\n eval-smoke init\n eval-smoke init ./prompts --force\n eval-smoke examples/suite.yaml --mock\n eval-smoke examples/suite.yaml --mock --only greeting\n eval-smoke examples/suite.yaml --mock --only '/^capital/'\n eval-smoke examples/suite.yaml --mock --junit junit.xml\n eval-smoke examples/suite.yaml --json --fail-fast\n eval-smoke examples/suite.yaml --mock --concurrency 4\n OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml\n";
2
+ export declare const HELP = "eval-smoke \u2014 one-minute smoke evals for prompts and models\n\nUsage:\n eval-smoke <suite.yaml|suite.json> [options]\n eval-smoke init [dir] [--force]\n\nOptions:\n --mock Offline runner with deterministic fake answers\n --json Machine-readable JSON report\n --junit <path> Write JUnit XML report to path (still prints human report unless --json)\n --only <pattern> Run cases whose id contains pattern, or matches /regex/\n --fail-fast Stop after the first failing case\n --concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)\n --timeout <ms> Per-case timeout (default: 30000)\n --model <name> Override suite model\n --provider <name> openai | anthropic | mock\n --baseline <path> Fail if any baseline-passing case now fails or is missing\n --update-baseline <path> Write current report JSON to path (pretty)\n --github-summary [path] Append Markdown to GITHUB_STEP_SUMMARY or path\n --force Overwrite existing suite.yaml when using init\n --no-color Disable ANSI colors (also respects NO_COLOR)\n -h, --help Show this help\n -v, --version Show version\n\nExit codes:\n 0 all cases passed (and baseline clean when --baseline is set)\n 1 one or more cases failed, or baseline regression\n 2 usage or configuration error\n\nExamples:\n eval-smoke init\n eval-smoke init ./prompts --force\n eval-smoke examples/suite.yaml --mock\n eval-smoke examples/suite.yaml --mock --only greeting\n eval-smoke examples/suite.yaml --mock --only '/^capital/'\n eval-smoke examples/suite.yaml --mock --junit junit.xml\n eval-smoke examples/suite.yaml --json --fail-fast\n eval-smoke examples/suite.yaml --mock --concurrency 4\n eval-smoke examples/suite.yaml --mock --update-baseline baseline.json\n eval-smoke examples/suite.yaml --mock --baseline baseline.json\n eval-smoke examples/suite.yaml --mock --github-summary\n OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml\n";
3
3
  export interface CliIo {
4
4
  stdout: {
5
5
  write(text: string): void;
@@ -14,5 +14,11 @@ export interface CliIo {
14
14
  /** Match case ids by substring, or by /regex/ flags (same style as matchers). */
15
15
  export declare function matchOnly(id: string, pattern: string): boolean;
16
16
  export declare function filterSuiteByOnly(suite: Suite, pattern: string): Suite;
17
+ /**
18
+ * Rewrite bare `--github-summary` into `--github-summary=__auto__` so node:util
19
+ * parseArgs (string option) accepts it. `--github-summary=path` and
20
+ * `--github-summary path` stay as-is.
21
+ */
22
+ export declare function normalizeGithubSummaryArgv(argv: string[]): string[];
17
23
  export declare function runCli(argv: string[], io?: CliIo): Promise<number>;
18
24
  //# sourceMappingURL=run-cli.d.ts.map
package/dist/run-cli.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"run-cli.d.ts","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AASA,OAAO,EAAiC,KAAK,KAAK,EAAE,MAAM,YAAY,CAAC;AAGvE,eAAO,MAAM,IAAI,mgDAoChB,CAAC;AAEF,MAAM,WAAW,KAAK;IACpB,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,GAAG,EAAE,MAAM,CAAC,UAAU,CAAC;IACvB,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,OAAO,CAAC;CAChB;AAkCD,iFAAiF;AACjF,wBAAgB,SAAS,CAAC,EAAE,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAQ9D;AAED,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,GAAG,KAAK,CAMtE;AAED,wBAAsB,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,GAAE,KAAmB,GAAG,OAAO,CAAC,MAAM,CAAC,CA6FrF"}
1
+ {"version":3,"file":"run-cli.d.ts","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAYA,OAAO,EAAiC,KAAK,KAAK,EAAE,MAAM,YAAY,CAAC;AAMvE,eAAO,MAAM,IAAI,i/DA0ChB,CAAC;AAEF,MAAM,WAAW,KAAK;IACpB,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,GAAG,EAAE,MAAM,CAAC,UAAU,CAAC;IACvB,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,OAAO,CAAC;CAChB;AAkCD,iFAAiF;AACjF,wBAAgB,SAAS,CAAC,EAAE,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAQ9D;AAED,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,GAAG,KAAK,CAMtE;AAED;;;;GAIG;AACH,wBAAgB,0BAA0B,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,MAAM,EAAE,CAiBnE;AAQD,wBAAsB,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,GAAE,KAAmB,GAAG,OAAO,CAAC,MAAM,CAAC,CAgIrF"}
package/dist/run-cli.js CHANGED
@@ -1,14 +1,19 @@
1
1
  import { writeFileSync } from "node:fs";
2
2
  import { resolve } from "node:path";
3
3
  import { parseArgs } from "node:util";
4
+ import { checkBaseline, loadBaseline, writeBaseline } from "./baseline.js";
5
+ import { appendGithubSummary, resolveGithubSummaryPath } from "./github-summary.js";
4
6
  import { initSuite } from "./init.js";
5
7
  import { formatJUnit } from "./junit.js";
6
8
  import { loadSuite } from "./load-suite.js";
7
9
  import { compileRegex } from "./matchers.js";
8
10
  import { formatReport } from "./report.js";
9
11
  import { DEFAULT_TIMEOUT_MS, runSuite } from "./run.js";
12
+ import { formatGithubSummary } from "./summary-md.js";
10
13
  import { UsageError } from "./types.js";
11
14
  import { VERSION } from "./version.js";
15
+ /** Sentinel so node:util accepts a bare --github-summary flag. */
16
+ const GITHUB_SUMMARY_AUTO = "__auto__";
12
17
  export const HELP = `eval-smoke — one-minute smoke evals for prompts and models
13
18
 
14
19
  Usage:
@@ -25,14 +30,17 @@ Options:
25
30
  --timeout <ms> Per-case timeout (default: ${DEFAULT_TIMEOUT_MS})
26
31
  --model <name> Override suite model
27
32
  --provider <name> openai | anthropic | mock
33
+ --baseline <path> Fail if any baseline-passing case now fails or is missing
34
+ --update-baseline <path> Write current report JSON to path (pretty)
35
+ --github-summary [path] Append Markdown to GITHUB_STEP_SUMMARY or path
28
36
  --force Overwrite existing suite.yaml when using init
29
37
  --no-color Disable ANSI colors (also respects NO_COLOR)
30
38
  -h, --help Show this help
31
39
  -v, --version Show version
32
40
 
33
41
  Exit codes:
34
- 0 all cases passed
35
- 1 one or more cases failed
42
+ 0 all cases passed (and baseline clean when --baseline is set)
43
+ 1 one or more cases failed, or baseline regression
36
44
  2 usage or configuration error
37
45
 
38
46
  Examples:
@@ -44,6 +52,9 @@ Examples:
44
52
  eval-smoke examples/suite.yaml --mock --junit junit.xml
45
53
  eval-smoke examples/suite.yaml --json --fail-fast
46
54
  eval-smoke examples/suite.yaml --mock --concurrency 4
55
+ eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
56
+ eval-smoke examples/suite.yaml --mock --baseline baseline.json
57
+ eval-smoke examples/suite.yaml --mock --github-summary
47
58
  OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml
48
59
  `;
49
60
  const defaultIo = () => ({
@@ -95,11 +106,42 @@ export function filterSuiteByOnly(suite, pattern) {
95
106
  }
96
107
  return { ...suite, cases };
97
108
  }
109
+ /**
110
+ * Rewrite bare `--github-summary` into `--github-summary=__auto__` so node:util
111
+ * parseArgs (string option) accepts it. `--github-summary=path` and
112
+ * `--github-summary path` stay as-is.
113
+ */
114
+ export function normalizeGithubSummaryArgv(argv) {
115
+ const out = [];
116
+ for (let i = 0; i < argv.length; i += 1) {
117
+ const token = argv[i];
118
+ if (token === "--github-summary") {
119
+ const next = argv[i + 1];
120
+ if (!next || next.startsWith("-")) {
121
+ out.push(`--github-summary=${GITHUB_SUMMARY_AUTO}`);
122
+ }
123
+ else {
124
+ out.push("--github-summary", next);
125
+ i += 1;
126
+ }
127
+ continue;
128
+ }
129
+ out.push(token);
130
+ }
131
+ return out;
132
+ }
133
+ function githubSummaryFlag(raw) {
134
+ if (raw == null)
135
+ return undefined;
136
+ if (raw === GITHUB_SUMMARY_AUTO)
137
+ return true;
138
+ return raw;
139
+ }
98
140
  export async function runCli(argv, io = defaultIo()) {
99
141
  let parsed;
100
142
  try {
101
143
  parsed = parseArgs({
102
- args: argv,
144
+ args: normalizeGithubSummaryArgv(argv),
103
145
  allowPositionals: true,
104
146
  strict: true,
105
147
  options: {
@@ -112,6 +154,9 @@ export async function runCli(argv, io = defaultIo()) {
112
154
  timeout: { type: "string" },
113
155
  model: { type: "string" },
114
156
  provider: { type: "string" },
157
+ baseline: { type: "string" },
158
+ "update-baseline": { type: "string" },
159
+ "github-summary": { type: "string" },
115
160
  force: { type: "boolean", default: false },
116
161
  "no-color": { type: "boolean", default: false },
117
162
  help: { type: "boolean", short: "h", default: false },
@@ -158,6 +203,9 @@ export async function runCli(argv, io = defaultIo()) {
158
203
  const json = asBool(values.json);
159
204
  const only = asString(values.only);
160
205
  const junitPath = asString(values.junit);
206
+ const baselinePath = asString(values.baseline);
207
+ const updateBaselinePath = asString(values["update-baseline"]);
208
+ const githubSummary = githubSummaryFlag(asString(values["github-summary"]));
161
209
  let suite = loadSuite(suitePath, io.cwd);
162
210
  if (only != null) {
163
211
  suite = filterSuiteByOnly(suite, only);
@@ -175,9 +223,36 @@ export async function runCli(argv, io = defaultIo()) {
175
223
  const out = resolve(io.cwd, junitPath);
176
224
  writeFileSync(out, formatJUnit(report), "utf8");
177
225
  }
226
+ if (updateBaselinePath != null) {
227
+ const out = resolve(io.cwd, updateBaselinePath);
228
+ writeBaseline(out, report);
229
+ io.stderr.write(`Wrote baseline ${out}\n`);
230
+ }
231
+ let baselineFailed = false;
232
+ if (baselinePath != null) {
233
+ const baseline = loadBaseline(resolve(io.cwd, baselinePath));
234
+ const check = checkBaseline(report, baseline);
235
+ io.stderr.write(check.report);
236
+ if (!check.ok)
237
+ baselineFailed = true;
238
+ }
239
+ const outSummary = githubSummary === undefined
240
+ ? resolveGithubSummaryPath(undefined, io.env, io.cwd)
241
+ : resolveGithubSummaryPath(githubSummary, io.env, io.cwd);
242
+ if (outSummary) {
243
+ try {
244
+ appendGithubSummary(outSummary, formatGithubSummary(report));
245
+ }
246
+ catch (error) {
247
+ const message = error instanceof Error ? error.message : String(error);
248
+ io.stderr.write(`could not write github summary: ${message}\n`);
249
+ }
250
+ }
178
251
  const noColor = asBool(values["no-color"]) || Boolean(io.env.NO_COLOR);
179
252
  const color = io.isTTY && !json && !noColor;
180
253
  io.stdout.write(formatReport(report, { json, color }));
254
+ if (baselineFailed)
255
+ return 1;
181
256
  return report.ok ? 0 : 1;
182
257
  }
183
258
  catch (error) {
package/dist/run-cli.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"run-cli.js","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACxC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC5C,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,kBAAkB,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACxD,OAAO,EAAE,UAAU,EAAiC,MAAM,YAAY,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,MAAM,CAAC,MAAM,IAAI,GAAG;;;;;;;;;;;;;mDAa+B,kBAAkB;;;;;;;;;;;;;;;;;;;;;;;CAuBpE,CAAC;AAUF,MAAM,SAAS,GAAG,GAAU,EAAE,CAAC,CAAC;IAC9B,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,GAAG,EAAE,OAAO,CAAC,GAAG;IAChB,GAAG,EAAE,OAAO,CAAC,GAAG,EAAE;IAClB,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC;CACrC,CAAC,CAAC;AAEH,SAAS,YAAY,CAAC,GAAuB;IAC3C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,kBAAkB,CAAC;IAC3C,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,qDAAqD,CAAC,CAAC;IAC9E,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,gBAAgB,CAAC,GAAuB;IAC/C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,0CAA0C,CAAC,CAAC;IACnE,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,aAAa,CAAC,GAAuB;IAC5C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,GAAG,KAAK,MAAM,IAAI,GAAG,KAAK,QAAQ,IAAI,GAAG,KAAK,WAAW;QAAE,OAAO,GAAG,CAAC;IAC1E,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;AACxE,CAAC;AAED,iFAAiF;AACjF,MAAM,UAAU,SAAS,CAAC,EAAU,EAAE,OAAe;IACnD,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;IAC5D,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;QACzB,MAAM,EAAE,GAAG,YAAY,CAAC,OAAO,CAAC,CAAC;QACjC,EAAE,CAAC,SAAS,GAAG,CAAC,CAAC;QACjB,OAAO,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,EAAE,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;AAC9B,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KAAY,EAAE,OAAe;IAC7D,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,SAAS,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;IAChF,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACvB,MAAM,IAAI,UAAU,CAAC,UAAU,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,kBAAkB,CAAC,CAAC;IAC5E,CAAC;IACD,OAAO,EAAE,GAAG,KAAK,EAAE,KAAK,EAAE,CAAC;AAC7B,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,MAAM,CAAC,IAAc,EAAE,KAAY,SAAS,EAAE;IAClE,IAAI,MAAoC,CAAC;IACzC,IAAI,CAAC;QACH,MAAM,GAAG,SAAS,CAAC;YACjB,IAAI,EAAE,IAAI;YACV,gBAAgB,EAAE,IAAI;YACtB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE;gBACP,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACxB,WAAW,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAChD,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC/B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC3B,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC5B,KAAK,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC1C,UAAU,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC/C,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;gBACrD,OAAO,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;aACzD;SACF,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QAC/E,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,MAAM,CAAC;IACvC,MAAM,QAAQ,GAAG,CAAC,KAAc,EAAsB,EAAE,CACtD,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;IAChD,MAAM,MAAM,GAAG,CAAC,KAAc,EAAW,EAAE,CAAC,KAAK,KAAK,IAAI,CAAC;IAE3D,IAAI,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;QACxB,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;QAC3B,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,WAAW,CAAC,CAAC,CAAC,KAAK,MAAM,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;YAC9B,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,EAAE,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC;YAC/E,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,MAAM,CAAC,IAAI,IAAI,CAAC,CAAC;YAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,oBAAoB,MAAM,CAAC,IAAI,WAAW,CAAC,CAAC;YAC5D,OAAO,CAAC,CAAC;QACX,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;YAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;IAED,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IACjC,IAAI,CAAC,SAAS,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,yBAAyB,CAAC,CAAC;QAC3C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACjC,MAAM,IAAI,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,SAAS,GAAG,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACzC,IAAI,KAAK,GAAG,SAAS,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC;QACzC,IAAI,IAAI,IAAI,IAAI,EAAE,CAAC;YACjB,KAAK,GAAG,iBAAiB,CAAC,KAAK,EAAE,IAAI,CAAC,CAAC;QACzC,CAAC;QACD,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,KAAK,EAAE;YACnC,IAAI,EAAE,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC;YACzB,QAAQ,EAAE,aAAa,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;YAClD,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC;YAC7B,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YACrC,WAAW,EAAE,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YAC3D,SAAS,EAAE,YAAY,CAAC,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;YACjD,GAAG,EAAE,EAAE,CAAC,GAAG;SACZ,CAAC,CAAC;QACH,IAAI,SAAS,IAAI,IAAI,EAAE,CAAC;YACtB,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;YACvC,aAAa,CAAC,GAAG,EAAE,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAClD,CAAC;QACD,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,IAAI,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACvE,MAAM,KAAK,GAAG,EAAE,CAAC,KAAK,IAAI,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC;QAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QACvD,OAAO,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;IAC1D,CAAC;AACH,CAAC"}
1
+ {"version":3,"file":"run-cli.js","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACxC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpF,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC5C,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,kBAAkB,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,UAAU,EAAiC,MAAM,YAAY,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,kEAAkE;AAClE,MAAM,mBAAmB,GAAG,UAAU,CAAC;AAEvC,MAAM,CAAC,MAAM,IAAI,GAAG;;;;;;;;;;;;;mDAa+B,kBAAkB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CA6BpE,CAAC;AAUF,MAAM,SAAS,GAAG,GAAU,EAAE,CAAC,CAAC;IAC9B,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,GAAG,EAAE,OAAO,CAAC,GAAG;IAChB,GAAG,EAAE,OAAO,CAAC,GAAG,EAAE;IAClB,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC;CACrC,CAAC,CAAC;AAEH,SAAS,YAAY,CAAC,GAAuB;IAC3C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,kBAAkB,CAAC;IAC3C,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,qDAAqD,CAAC,CAAC;IAC9E,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,gBAAgB,CAAC,GAAuB;IAC/C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,0CAA0C,CAAC,CAAC;IACnE,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,aAAa,CAAC,GAAuB;IAC5C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,GAAG,KAAK,MAAM,IAAI,GAAG,KAAK,QAAQ,IAAI,GAAG,KAAK,WAAW;QAAE,OAAO,GAAG,CAAC;IAC1E,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;AACxE,CAAC;AAED,iFAAiF;AACjF,MAAM,UAAU,SAAS,CAAC,EAAU,EAAE,OAAe;IACnD,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;IAC5D,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;QACzB,MAAM,EAAE,GAAG,YAAY,CAAC,OAAO,CAAC,CAAC;QACjC,EAAE,CAAC,SAAS,GAAG,CAAC,CAAC;QACjB,OAAO,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,EAAE,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;AAC9B,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KAAY,EAAE,OAAe;IAC7D,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,SAAS,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;IAChF,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACvB,MAAM,IAAI,UAAU,CAAC,UAAU,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,kBAAkB,CAAC,CAAC;IAC5E,CAAC;IACD,OAAO,EAAE,GAAG,KAAK,EAAE,KAAK,EAAE,CAAC;AAC7B,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,0BAA0B,CAAC,IAAc;IACvD,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;QACxC,MAAM,KAAK,GAAG,IAAI,CAAC,CAAC,CAAE,CAAC;QACvB,IAAI,KAAK,KAAK,kBAAkB,EAAE,CAAC;YACjC,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;gBAClC,GAAG,CAAC,IAAI,CAAC,oBAAoB,mBAAmB,EAAE,CAAC,CAAC;YACtD,CAAC;iBAAM,CAAC;gBACN,GAAG,CAAC,IAAI,CAAC,kBAAkB,EAAE,IAAI,CAAC,CAAC;gBACnC,CAAC,IAAI,CAAC,CAAC;YACT,CAAC;YACD,SAAS;QACX,CAAC;QACD,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,iBAAiB,CAAC,GAAuB;IAChD,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,GAAG,KAAK,mBAAmB;QAAE,OAAO,IAAI,CAAC;IAC7C,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,MAAM,CAAC,IAAc,EAAE,KAAY,SAAS,EAAE;IAClE,IAAI,MAAoC,CAAC;IACzC,IAAI,CAAC;QACH,MAAM,GAAG,SAAS,CAAC;YACjB,IAAI,EAAE,0BAA0B,CAAC,IAAI,CAAC;YACtC,gBAAgB,EAAE,IAAI;YACtB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE;gBACP,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACxB,WAAW,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAChD,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC/B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC3B,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC5B,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC5B,iBAAiB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACrC,gBAAgB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACpC,KAAK,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC1C,UAAU,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC/C,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;gBACrD,OAAO,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;aACzD;SACF,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QAC/E,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,MAAM,CAAC;IACvC,MAAM,QAAQ,GAAG,CAAC,KAAc,EAAsB,EAAE,CACtD,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;IAChD,MAAM,MAAM,GAAG,CAAC,KAAc,EAAW,EAAE,CAAC,KAAK,KAAK,IAAI,CAAC;IAE3D,IAAI,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;QACxB,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;QAC3B,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,WAAW,CAAC,CAAC,CAAC,KAAK,MAAM,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;YAC9B,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,EAAE,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC;YAC/E,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,MAAM,CAAC,IAAI,IAAI,CAAC,CAAC;YAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,oBAAoB,MAAM,CAAC,IAAI,WAAW,CAAC,CAAC;YAC5D,OAAO,CAAC,CAAC;QACX,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;YAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;IAED,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IACjC,IAAI,CAAC,SAAS,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,yBAAyB,CAAC,CAAC;QAC3C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACjC,MAAM,IAAI,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,SAAS,GAAG,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACzC,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;QAC/C,MAAM,kBAAkB,GAAG,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,CAAC,CAAC;QAC/D,MAAM,aAAa,GAAG,iBAAiB,CAAC,QAAQ,CAAC,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC;QAE5E,IAAI,KAAK,GAAG,SAAS,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC;QACzC,IAAI,IAAI,IAAI,IAAI,EAAE,CAAC;YACjB,KAAK,GAAG,iBAAiB,CAAC,KAAK,EAAE,IAAI,CAAC,CAAC;QACzC,CAAC;QACD,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,KAAK,EAAE;YACnC,IAAI,EAAE,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC;YACzB,QAAQ,EAAE,aAAa,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;YAClD,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC;YAC7B,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YACrC,WAAW,EAAE,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YAC3D,SAAS,EAAE,YAAY,CAAC,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;YACjD,GAAG,EAAE,EAAE,CAAC,GAAG;SACZ,CAAC,CAAC;QACH,IAAI,SAAS,IAAI,IAAI,EAAE,CAAC;YACtB,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;YACvC,aAAa,CAAC,GAAG,EAAE,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAClD,CAAC;QACD,IAAI,kBAAkB,IAAI,IAAI,EAAE,CAAC;YAC/B,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,kBAAkB,CAAC,CAAC;YAChD,aAAa,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YAC3B,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,GAAG,IAAI,CAAC,CAAC;QAC7C,CAAC;QAED,IAAI,cAAc,GAAG,KAAK,CAAC;QAC3B,IAAI,YAAY,IAAI,IAAI,EAAE,CAAC;YACzB,MAAM,QAAQ,GAAG,YAAY,CAAC,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,YAAY,CAAC,CAAC,CAAC;YAC7D,MAAM,KAAK,GAAG,aAAa,CAAC,MAAM,EAAE,QAAQ,CAAC,CAAC;YAC9C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YAC9B,IAAI,CAAC,KAAK,CAAC,EAAE;gBAAE,cAAc,GAAG,IAAI,CAAC;QACvC,CAAC;QAED,MAAM,UAAU,GACd,aAAa,KAAK,SAAS;YACzB,CAAC,CAAC,wBAAwB,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC;YACrD,CAAC,CAAC,wBAAwB,CAAC,aAAa,EAAE,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC;QAC9D,IAAI,UAAU,EAAE,CAAC;YACf,IAAI,CAAC;gBACH,mBAAmB,CAAC,UAAU,EAAE,mBAAmB,CAAC,MAAM,CAAC,CAAC,CAAC;YAC/D,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,mCAAmC,OAAO,IAAI,CAAC,CAAC;YAClE,CAAC;QACH,CAAC;QAED,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,IAAI,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACvE,MAAM,KAAK,GAAG,EAAE,CAAC,KAAK,IAAI,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC;QAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QACvD,IAAI,cAAc;YAAE,OAAO,CAAC,CAAC;QAC7B,OAAO,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;IAC1D,CAAC;AACH,CAAC"}
package/dist/summary-md.d.ts ADDED
@@ -0,0 +1,4 @@
1
+ import type { Report } from "./types.js";
2
+ /** Concise pass/fail Markdown table for GitHub Actions step summary. */
3
+ export declare function formatGithubSummary(report: Report): string;
4
+ //# sourceMappingURL=summary-md.d.ts.map
package/dist/summary-md.d.ts.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"summary-md.d.ts","sourceRoot":"","sources":["../src/summary-md.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,YAAY,CAAC;AAEzC,wEAAwE;AACxE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAoB1D"}
package/dist/summary-md.js ADDED
@@ -0,0 +1,22 @@
1
+ /** Concise pass/fail Markdown table for GitHub Actions step summary. */
2
+ export function formatGithubSummary(report) {
3
+ const lines = [];
4
+ lines.push(`### eval-smoke · ${escape(report.suite ?? "suite")}`);
5
+ lines.push("");
6
+ lines.push(`**${report.passed} passed · ${report.failed} failed${report.skipped ? ` · ${report.skipped} skipped` : ""} · ${report.duration_ms}ms** · \`${report.runner}\` / \`${escape(report.model)}\``);
7
+ lines.push("");
8
+ lines.push("| Case | Result | ms |");
9
+ lines.push("| --- | --- | ---: |");
10
+ for (const c of report.cases) {
11
+ const status = c.skipped ? "skip" : c.ok ? "✅ pass" : "❌ fail";
12
+ lines.push(`| \`${escape(c.id)}\` | ${status} | ${c.duration_ms} |`);
13
+ }
14
+ lines.push("");
15
+ lines.push(report.ok ? "**Suite: PASS**" : "**Suite: FAIL**");
16
+ lines.push("");
17
+ return lines.join("\n");
18
+ }
19
+ function escape(text) {
20
+ return text.replace(/\|/g, "\\|").replace(/\r?\n/g, " ");
21
+ }
22
+ //# sourceMappingURL=summary-md.js.map
package/dist/summary-md.js.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"summary-md.js","sourceRoot":"","sources":["../src/summary-md.ts"],"names":[],"mappings":"AAEA,wEAAwE;AACxE,MAAM,UAAU,mBAAmB,CAAC,MAAc;IAChD,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,KAAK,CAAC,IAAI,CAAC,oBAAoB,MAAM,CAAC,MAAM,CAAC,KAAK,IAAI,OAAO,CAAC,EAAE,CAAC,CAAC;IAClE,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACf,KAAK,CAAC,IAAI,CACR,KAAK,MAAM,CAAC,MAAM,aAAa,MAAM,CAAC,MAAM,UAC1C,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,MAAM,CAAC,OAAO,UAAU,CAAC,CAAC,CAAC,EACpD,MAAM,MAAM,CAAC,WAAW,YAAY,MAAM,CAAC,MAAM,UAAU,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CACpF,CAAC;IACF,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACf,KAAK,CAAC,IAAI,CAAC,wBAAwB,CAAC,CAAC;IACrC,KAAK,CAAC,IAAI,CAAC,sBAAsB,CAAC,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC;QAC7B,MAAM,MAAM,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC;QAC/D,KAAK,CAAC,IAAI,CAAC,OAAO,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,QAAQ,MAAM,MAAM,CAAC,CAAC,WAAW,IAAI,CAAC,CAAC;IACvE,CAAC;IACD,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACf,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,iBAAiB,CAAC,CAAC,CAAC,iBAAiB,CAAC,CAAC;IAC9D,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACf,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC1B,CAAC;AAED,SAAS,MAAM,CAAC,IAAY;IAC1B,OAAO,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;AAC3D,CAAC"}
package/examples/github-action.yml ADDED
@@ -0,0 +1,41 @@
1
+ # Copy-paste: run eval-smoke in GitHub Actions (mock offline + JUnit + step summary).
2
+ # No clone of this repo required — npx pulls @webywz/eval-smoke from npm.
3
+ #
4
+ # Human report → stdout (job log). JUnit → file. Markdown → $GITHUB_STEP_SUMMARY.
5
+
6
+ name: Smoke eval with eval-smoke
7
+
8
+ on:
9
+ workflow_dispatch:
10
+ # Or: push / pull_request when you have a suite.yaml in the repo.
11
+
12
+ jobs:
13
+ eval:
14
+ runs-on: ubuntu-latest
15
+ steps:
16
+ - uses: actions/checkout@v4
17
+
18
+ - uses: actions/setup-node@v4
19
+ with:
20
+ node-version: 22
21
+
22
+ # Offline smoke against the published example suite (no API key):
23
+ - name: Mock smoke + JUnit + step summary
24
+ run: |
25
+ npx --yes @webywz/eval-smoke --help >/dev/null
26
+ # Prefer a suite checked into YOUR repo; this uses the package example via init:
27
+ npx --yes @webywz/eval-smoke init ./eval-smoke-ci --force
28
+ npx --yes @webywz/eval-smoke ./eval-smoke-ci/suite.yaml \
29
+ --mock \
30
+ --junit junit-eval-smoke.xml \
31
+ --github-summary \
32
+ --update-baseline eval-smoke-baseline.json
33
+
34
+ - name: Upload JUnit
35
+ if: always()
36
+ uses: actions/upload-artifact@v4
37
+ with:
38
+ name: eval-smoke-junit
39
+ path: |
40
+ junit-eval-smoke.xml
41
+ eval-smoke-baseline.json
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@webywz/eval-smoke",
3
- "version": "0.3.1",
4
- "description": "One-minute smoke evals for prompts and models — equals/json matchers, diffs, concurrency, JUnit, init.",
3
+ "version": "0.4.0",
4
+ "description": "One-minute smoke evals for prompts and models \u2014 equals/json matchers, diffs, concurrency, JUnit, init.",
5
5
  "type": "module",
6
6
  "bin": {
7
7
  "eval-smoke": "dist/cli.js"