@webywz/eval-smoke 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -6
- package/dist/html-report.d.ts +8 -0
- package/dist/html-report.d.ts.map +1 -0
- package/dist/html-report.js +196 -0
- package/dist/html-report.js.map +1 -0
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/load-suite.d.ts.map +1 -1
- package/dist/load-suite.js +5 -1
- package/dist/load-suite.js.map +1 -1
- package/dist/matchers.d.ts.map +1 -1
- package/dist/matchers.js +64 -0
- package/dist/matchers.js.map +1 -1
- package/dist/run-cli.d.ts +1 -1
- package/dist/run-cli.d.ts.map +1 -1
- package/dist/run-cli.js +9 -0
- package/dist/run-cli.js.map +1 -1
- package/dist/types.d.ts +9 -1
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/examples/suite.yaml +16 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -25,7 +25,7 @@ eval-smoke suite.yaml --mock
|
|
|
25
25
|
```
|
|
26
26
|
|
|
27
27
|
```text
|
|
28
|
-
eval-smoke 0.
|
|
28
|
+
eval-smoke 0.5.0 · mock · gpt-4o-mini · smoke · 4 cases
|
|
29
29
|
|
|
30
30
|
pass greeting 0ms
|
|
31
31
|
pass capital-france 0ms
|
|
@@ -42,10 +42,10 @@ eval-smoke 0.4.0 · mock · gpt-4o-mini · smoke · 4 cases
|
|
|
42
42
|
| | |
|
|
43
43
|
| --- | --- |
|
|
44
44
|
| **Tiny suite** | A handful of cases — not a full harness |
|
|
45
|
-
| **Matchers** | `equals`, `contains`, `not_contains`, `regex`, `json`, `min_tokens` / `max_tokens` |
|
|
45
|
+
| **Matchers** | `equals`, `contains`, `contains_any` / `contains_all`, `starts_with` / `ends_with`, `not_contains`, `regex`, `json`, `min_tokens` / `max_tokens` |
|
|
46
46
|
| **init** | `eval-smoke init` scaffolds a mock-friendly `suite.yaml` |
|
|
47
47
|
| **Filter** | `--only` by id substring or `/regex/` |
|
|
48
|
-
| **CI** | `--junit`, `--github-summary`, exit codes, `--fail-fast`, `--concurrency` |
|
|
48
|
+
| **CI** | `--junit`, `--html`, `--github-summary`, exit codes, `--fail-fast`, `--concurrency` |
|
|
49
49
|
| **Baseline** | `--baseline` / `--update-baseline` catch pass→fail regressions |
|
|
50
50
|
| **Providers** | Mock offline; live OpenAI / Anthropic when keyed |
|
|
51
51
|
|
|
@@ -66,6 +66,7 @@ eval-smoke init
|
|
|
66
66
|
eval-smoke suite.yaml --mock
|
|
67
67
|
eval-smoke suite.yaml --mock --only greeting
|
|
68
68
|
eval-smoke suite.yaml --mock --junit junit.xml
|
|
69
|
+
eval-smoke suite.yaml --mock --html report.html
|
|
69
70
|
eval-smoke examples/suite.yaml --mock --json
|
|
70
71
|
eval-smoke examples/suite.yaml --mock --concurrency 4
|
|
71
72
|
eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
|
|
@@ -116,6 +117,9 @@ cases:
|
|
|
116
117
|
| --- | --- |
|
|
117
118
|
| `id` / `prompt` | Required |
|
|
118
119
|
| `expect.contains` | Substring, case-insensitive; string or list |
|
|
120
|
+
| `expect.contains_any` | Pass if **any** needle is found (OR) |
|
|
121
|
+
| `expect.contains_all` | Pass if **all** needles are found (AND) |
|
|
122
|
+
| `expect.starts_with` / `ends_with` | Prefix / suffix; string or list (OR) |
|
|
119
123
|
| `expect.not_contains` | Must be absent |
|
|
120
124
|
| `expect.regex` | `/pattern/i`, `(?i)pattern`, or raw |
|
|
121
125
|
| `expect.equals` | Exact after NFC + LF normalize |
|
|
@@ -138,6 +142,7 @@ eval-smoke init [dir] [--force]
|
|
|
138
142
|
| `--mock` | Offline deterministic runner |
|
|
139
143
|
| `--json` | Machine-readable JSON report |
|
|
140
144
|
| `--junit <path>` | Write JUnit XML (still prints human report unless `--json`) |
|
|
145
|
+
| `--html <path>` | Write self-contained HTML report (offline, no CDN) |
|
|
141
146
|
| `--baseline <path>` | Exit `1` if a baseline-passing case now fails or is missing |
|
|
142
147
|
| `--update-baseline <path>` | Write current report JSON (pretty) to path |
|
|
143
148
|
| `--github-summary [path]` | Append pass/fail Markdown to `$GITHUB_STEP_SUMMARY` or path |
|
|
@@ -184,7 +189,7 @@ if (!report.ok) process.exit(1);
|
|
|
184
189
|
### CI snippet
|
|
185
190
|
|
|
186
191
|
```yaml
|
|
187
|
-
- run: npx @webywz/eval-smoke@0.
|
|
192
|
+
- run: npx @webywz/eval-smoke@0.5.0 prompts/smoke.yaml --mock --junit junit.xml
|
|
188
193
|
- uses: actions/upload-artifact@v4
|
|
189
194
|
if: always()
|
|
190
195
|
with:
|
|
@@ -219,10 +224,10 @@ eval-smoke suite.yaml --mock
|
|
|
219
224
|
| | |
|
|
220
225
|
| --- | --- |
|
|
221
226
|
| **轻量套件** | 少量用例,不是完整评测框架 |
|
|
222
|
-
| **匹配器** | `equals`、`contains`、`not_contains`、`regex`、`json`、token 上下限 |
|
|
227
|
+
| **匹配器** | `equals`、`contains`、`contains_any` / `contains_all`、`starts_with` / `ends_with`、`not_contains`、`regex`、`json`、token 上下限 |
|
|
223
228
|
| **init** | `eval-smoke init` 生成可 mock 的 `suite.yaml` |
|
|
224
229
|
| **过滤** | `--only`:id 子串或 `/regex/` |
|
|
225
|
-
| **CI** | `--junit`、`--github-summary`、退出码、`--fail-fast`、`--concurrency` |
|
|
230
|
+
| **CI** | `--junit`、`--html`、`--github-summary`、退出码、`--fail-fast`、`--concurrency` |
|
|
226
231
|
| **基线** | `--baseline` / `--update-baseline` 拦住 pass→fail 回归 |
|
|
227
232
|
| **提供商** | Mock 离线;有 key 时走 OpenAI / Anthropic |
|
|
228
233
|
|
|
@@ -243,6 +248,7 @@ eval-smoke init
|
|
|
243
248
|
eval-smoke suite.yaml --mock
|
|
244
249
|
eval-smoke suite.yaml --mock --only greeting
|
|
245
250
|
eval-smoke suite.yaml --mock --junit junit.xml
|
|
251
|
+
eval-smoke suite.yaml --mock --html report.html
|
|
246
252
|
eval-smoke examples/suite.yaml --mock --json
|
|
247
253
|
eval-smoke examples/suite.yaml --mock --concurrency 4
|
|
248
254
|
eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
|
|
@@ -285,6 +291,9 @@ cases:
|
|
|
285
291
|
| --- | --- |
|
|
286
292
|
| `id` / `prompt` | 必填 |
|
|
287
293
|
| `expect.contains` | 子串,忽略大小写;字符串或列表 |
|
|
294
|
+
| `expect.contains_any` | **任一**针命中即过(OR) |
|
|
295
|
+
| `expect.contains_all` | **全部**针都要命中(AND) |
|
|
296
|
+
| `expect.starts_with` / `ends_with` | 前缀 / 后缀;字符串或列表(OR) |
|
|
288
297
|
| `expect.not_contains` | 不得出现 |
|
|
289
298
|
| `expect.regex` | `/pattern/i`、`(?i)pattern` 或裸模式 |
|
|
290
299
|
| `expect.equals` | NFC + LF 规范化后精确匹配 |
|
|
@@ -302,6 +311,7 @@ cases:
|
|
|
302
311
|
| `--mock` | 离线确定性运行 |
|
|
303
312
|
| `--json` | JSON 报告 |
|
|
304
313
|
| `--junit <path>` | 写出 JUnit XML(仍打印人类可读报告,除非同时 `--json`) |
|
|
314
|
+
| `--html <path>` | 写入自包含 HTML 报告(离线、无 CDN) |
|
|
305
315
|
| `--baseline <path>` | 基线里曾通过的用例若失败或缺失则退出 `1` |
|
|
306
316
|
| `--update-baseline <path>` | 把当前报告 JSON(pretty)写入路径 |
|
|
307
317
|
| `--github-summary [path]` | 追加通过/失败 Markdown 到 `$GITHUB_STEP_SUMMARY` 或指定路径 |
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { Report } from "./types.js";
|
|
2
|
+
/** Escape text for safe insertion into HTML text/attr contexts. */
|
|
3
|
+
export declare function escapeHtml(text: string): string;
|
|
4
|
+
/**
|
|
5
|
+
* Self-contained dark HTML report for an eval-smoke run (inline CSS+JS, no CDN).
|
|
6
|
+
*/
|
|
7
|
+
export declare function formatHtmlReport(report: Report): string;
|
|
8
|
+
//# sourceMappingURL=html-report.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"html-report.d.ts","sourceRoot":"","sources":["../src/html-report.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAA2B,MAAM,EAAE,MAAM,YAAY,CAAC;AAElE,mEAAmE;AACnE,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAO/C;AA8CD;;GAEG;AACH,wBAAgB,gBAAgB,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAgJvD"}
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
/** Escape text for safe insertion into HTML text/attr contexts. */
|
|
2
|
+
export function escapeHtml(text) {
|
|
3
|
+
return text
|
|
4
|
+
.replace(/&/g, "&")
|
|
5
|
+
.replace(/</g, "<")
|
|
6
|
+
.replace(/>/g, ">")
|
|
7
|
+
.replace(/"/g, """)
|
|
8
|
+
.replace(/'/g, "'");
|
|
9
|
+
}
|
|
10
|
+
function truncate(text, max = 400) {
|
|
11
|
+
const compact = text.replace(/\s+/g, " ").trim();
|
|
12
|
+
if (compact.length <= max)
|
|
13
|
+
return compact;
|
|
14
|
+
return `${compact.slice(0, max - 1)}…`;
|
|
15
|
+
}
|
|
16
|
+
function failedChecks(result) {
|
|
17
|
+
return (result.checks ?? []).filter((check) => !check.ok);
|
|
18
|
+
}
|
|
19
|
+
function statusLabel(result) {
|
|
20
|
+
if (result.skipped)
|
|
21
|
+
return "skip";
|
|
22
|
+
return result.ok ? "pass" : "fail";
|
|
23
|
+
}
|
|
24
|
+
function statusClass(result) {
|
|
25
|
+
if (result.skipped)
|
|
26
|
+
return "skip";
|
|
27
|
+
return result.ok ? "pass" : "fail";
|
|
28
|
+
}
|
|
29
|
+
function failureDetails(result) {
|
|
30
|
+
const parts = [];
|
|
31
|
+
if (result.error)
|
|
32
|
+
parts.push(result.error);
|
|
33
|
+
for (const check of failedChecks(result)) {
|
|
34
|
+
const detail = check.detail ?? "failed";
|
|
35
|
+
parts.push(`${check.name}: ${detail}`);
|
|
36
|
+
if (check.expected != null && !detail.includes("expected:")) {
|
|
37
|
+
parts.push(`expected: ${truncate(check.expected, 200)}`);
|
|
38
|
+
}
|
|
39
|
+
if (check.actual != null && !detail.includes("actual:")) {
|
|
40
|
+
parts.push(`actual: ${truncate(check.actual, 200)}`);
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
if (result.output &&
|
|
44
|
+
!result.ok &&
|
|
45
|
+
failedChecks(result).every((c) => c.name !== "equals")) {
|
|
46
|
+
parts.push(`output: ${truncate(result.output)}`);
|
|
47
|
+
}
|
|
48
|
+
if (!parts.length)
|
|
49
|
+
return "";
|
|
50
|
+
return `<details><summary>details</summary><pre>${escapeHtml(parts.join("\n"))}</pre></details>`;
|
|
51
|
+
}
|
|
52
|
+
/**
|
|
53
|
+
* Self-contained dark HTML report for an eval-smoke run (inline CSS+JS, no CDN).
|
|
54
|
+
*/
|
|
55
|
+
export function formatHtmlReport(report) {
|
|
56
|
+
const suite = escapeHtml(report.suite ?? "suite");
|
|
57
|
+
const version = escapeHtml(report.version);
|
|
58
|
+
const runner = escapeHtml(report.runner);
|
|
59
|
+
const model = escapeHtml(report.model);
|
|
60
|
+
const okClass = report.ok ? "ok" : "bad";
|
|
61
|
+
const rows = report.cases
|
|
62
|
+
.map((result) => {
|
|
63
|
+
const st = statusClass(result);
|
|
64
|
+
const details = failureDetails(result);
|
|
65
|
+
return `<tr class="${st}">
|
|
66
|
+
<td><span class="badge ${st}">${statusLabel(result)}</span></td>
|
|
67
|
+
<td><code>${escapeHtml(result.id)}</code></td>
|
|
68
|
+
<td class="num">${result.duration_ms}ms</td>
|
|
69
|
+
<td class="detail">${details}</td>
|
|
70
|
+
</tr>`;
|
|
71
|
+
})
|
|
72
|
+
.join("\n");
|
|
73
|
+
return `<!DOCTYPE html>
|
|
74
|
+
<html lang="en">
|
|
75
|
+
<head>
|
|
76
|
+
<meta charset="utf-8"/>
|
|
77
|
+
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
|
78
|
+
<title>eval-smoke · ${suite}</title>
|
|
79
|
+
<style>
|
|
80
|
+
:root {
|
|
81
|
+
--bg: #0a0a0a;
|
|
82
|
+
--panel: #111111;
|
|
83
|
+
--border: #222222;
|
|
84
|
+
--text: #eeeeee;
|
|
85
|
+
--muted: #888888;
|
|
86
|
+
--pass: #7ee787;
|
|
87
|
+
--fail: #ff7b72;
|
|
88
|
+
--skip: #d2a8ff;
|
|
89
|
+
--font: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
|
90
|
+
}
|
|
91
|
+
* { box-sizing: border-box; }
|
|
92
|
+
html, body {
|
|
93
|
+
margin: 0; padding: 0;
|
|
94
|
+
background: var(--bg); color: var(--text);
|
|
95
|
+
font: 13px/1.45 var(--font);
|
|
96
|
+
}
|
|
97
|
+
body { padding: 24px 20px 48px; max-width: 960px; margin: 0 auto; }
|
|
98
|
+
header.top {
|
|
99
|
+
border: 1px solid var(--border); background: var(--panel);
|
|
100
|
+
padding: 16px 18px; margin-bottom: 16px;
|
|
101
|
+
}
|
|
102
|
+
header.top h1 { margin: 0 0 6px; font-size: 15px; font-weight: 600; }
|
|
103
|
+
header.top .meta { color: var(--muted); font-size: 12px; }
|
|
104
|
+
header.top .summary { margin-top: 10px; font-size: 13px; }
|
|
105
|
+
header.top.ok .summary { color: var(--pass); }
|
|
106
|
+
header.top.bad .summary { color: var(--fail); }
|
|
107
|
+
.stats {
|
|
108
|
+
display: grid; grid-template-columns: repeat(4, minmax(0, 1fr));
|
|
109
|
+
gap: 8px; margin: 0 0 18px;
|
|
110
|
+
}
|
|
111
|
+
.stat {
|
|
112
|
+
border: 1px solid var(--border); background: var(--panel);
|
|
113
|
+
padding: 10px 12px;
|
|
114
|
+
}
|
|
115
|
+
.stat .label { color: var(--muted); font-size: 11px; text-transform: uppercase; letter-spacing: 0.06em; }
|
|
116
|
+
.stat .value { margin-top: 4px; font-size: 16px; }
|
|
117
|
+
table {
|
|
118
|
+
width: 100%; border-collapse: collapse;
|
|
119
|
+
border: 1px solid var(--border); background: var(--panel);
|
|
120
|
+
}
|
|
121
|
+
th, td {
|
|
122
|
+
text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--border);
|
|
123
|
+
vertical-align: top;
|
|
124
|
+
}
|
|
125
|
+
th { color: var(--muted); font-weight: 500; font-size: 11px; text-transform: uppercase; }
|
|
126
|
+
td.num { text-align: right; color: var(--muted); white-space: nowrap; }
|
|
127
|
+
.badge {
|
|
128
|
+
display: inline-block; min-width: 3.2em; text-align: center;
|
|
129
|
+
padding: 1px 6px; border: 1px solid var(--border);
|
|
130
|
+
}
|
|
131
|
+
.badge.pass { color: var(--pass); border-color: #1e3d28; }
|
|
132
|
+
.badge.fail { color: var(--fail); border-color: #5a1d1d; }
|
|
133
|
+
.badge.skip { color: var(--skip); border-color: #3a2a55; }
|
|
134
|
+
tr.fail { background: #140a0a; }
|
|
135
|
+
details { margin-top: 4px; }
|
|
136
|
+
summary { cursor: pointer; color: var(--muted); user-select: none; }
|
|
137
|
+
pre {
|
|
138
|
+
margin: 6px 0 0; padding: 10px; overflow: auto;
|
|
139
|
+
background: #050505; border: 1px solid var(--border);
|
|
140
|
+
white-space: pre-wrap; word-break: break-word; color: #d8d8d8;
|
|
141
|
+
}
|
|
142
|
+
.toolbar { margin: 0 0 12px; display: flex; gap: 8px; }
|
|
143
|
+
.toolbar button {
|
|
144
|
+
background: var(--panel); color: var(--text); border: 1px solid var(--border);
|
|
145
|
+
padding: 6px 10px; font: inherit; cursor: pointer;
|
|
146
|
+
}
|
|
147
|
+
.toolbar button:hover { border-color: #444; }
|
|
148
|
+
footer {
|
|
149
|
+
margin-top: 28px; color: var(--muted); font-size: 11px;
|
|
150
|
+
border-top: 1px solid var(--border); padding-top: 12px;
|
|
151
|
+
}
|
|
152
|
+
code { font: inherit; }
|
|
153
|
+
@media (max-width: 640px) {
|
|
154
|
+
.stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
|
155
|
+
}
|
|
156
|
+
</style>
|
|
157
|
+
</head>
|
|
158
|
+
<body>
|
|
159
|
+
<header class="top ${okClass}">
|
|
160
|
+
<h1>eval-smoke ${version}</h1>
|
|
161
|
+
<div class="meta">${runner} · ${model} · ${suite} · ${report.cases.length} cases</div>
|
|
162
|
+
<div class="summary">${report.passed} passed · ${report.failed} failed${report.skipped ? ` · ${report.skipped} skipped` : ""} · ${report.duration_ms}ms</div>
|
|
163
|
+
</header>
|
|
164
|
+
<section class="stats" aria-label="stats">
|
|
165
|
+
<div class="stat"><div class="label">passed</div><div class="value">${report.passed}</div></div>
|
|
166
|
+
<div class="stat"><div class="label">failed</div><div class="value">${report.failed}</div></div>
|
|
167
|
+
<div class="stat"><div class="label">skipped</div><div class="value">${report.skipped}</div></div>
|
|
168
|
+
<div class="stat"><div class="label">duration</div><div class="value">${report.duration_ms}ms</div></div>
|
|
169
|
+
</section>
|
|
170
|
+
<div class="toolbar">
|
|
171
|
+
<button type="button" id="expand-all">Expand failures</button>
|
|
172
|
+
<button type="button" id="collapse-all">Collapse</button>
|
|
173
|
+
</div>
|
|
174
|
+
<table>
|
|
175
|
+
<thead><tr><th>Status</th><th>Case</th><th>Duration</th><th>Details</th></tr></thead>
|
|
176
|
+
<tbody>
|
|
177
|
+
${rows || `<tr><td colspan="4" style="color:var(--muted)">no cases</td></tr>`}
|
|
178
|
+
</tbody>
|
|
179
|
+
</table>
|
|
180
|
+
<footer>Generated by eval-smoke · self-contained · works offline (file://)</footer>
|
|
181
|
+
<script>
|
|
182
|
+
(function () {
|
|
183
|
+
var expand = document.getElementById("expand-all");
|
|
184
|
+
var collapse = document.getElementById("collapse-all");
|
|
185
|
+
function setAll(open) {
|
|
186
|
+
document.querySelectorAll("details").forEach(function (d) { d.open = open; });
|
|
187
|
+
}
|
|
188
|
+
if (expand) expand.addEventListener("click", function () { setAll(true); });
|
|
189
|
+
if (collapse) collapse.addEventListener("click", function () { setAll(false); });
|
|
190
|
+
})();
|
|
191
|
+
</script>
|
|
192
|
+
</body>
|
|
193
|
+
</html>
|
|
194
|
+
`;
|
|
195
|
+
}
|
|
196
|
+
//# sourceMappingURL=html-report.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"html-report.js","sourceRoot":"","sources":["../src/html-report.ts"],"names":[],"mappings":"AAEA,mEAAmE;AACnE,MAAM,UAAU,UAAU,CAAC,IAAY;IACrC,OAAO,IAAI;SACR,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC;SACtB,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC;SACrB,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC;SACrB,OAAO,CAAC,IAAI,EAAE,QAAQ,CAAC;SACvB,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;AAC5B,CAAC;AAED,SAAS,QAAQ,CAAC,IAAY,EAAE,GAAG,GAAG,GAAG;IACvC,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IACjD,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG;QAAE,OAAO,OAAO,CAAC;IAC1C,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,CAAC;AACzC,CAAC;AAED,SAAS,YAAY,CAAC,MAAkB;IACtC,OAAO,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;AAC5D,CAAC;AAED,SAAS,WAAW,CAAC,MAAkB;IACrC,IAAI,MAAM,CAAC,OAAO;QAAE,OAAO,MAAM,CAAC;IAClC,OAAO,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;AACrC,CAAC;AAED,SAAS,WAAW,CAAC,MAAkB;IACrC,IAAI,MAAM,CAAC,OAAO;QAAE,OAAO,MAAM,CAAC;IAClC,OAAO,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;AACrC,CAAC;AAED,SAAS,cAAc,CAAC,MAAkB;IACxC,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,IAAI,MAAM,CAAC,KAAK;QAAE,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;IAC3C,KAAK,MAAM,KAAK,IAAI,YAAY,CAAC,MAAM,CAAC,EAAE,CAAC;QACzC,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,IAAI,QAAQ,CAAC;QACxC,KAAK,CAAC,IAAI,CAAC,GAAG,KAAK,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC,CAAC;QACvC,IAAI,KAAK,CAAC,QAAQ,IAAI,IAAI,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,WAAW,CAAC,EAAE,CAAC;YAC5D,KAAK,CAAC,IAAI,CAAC,aAAa,QAAQ,CAAC,KAAK,CAAC,QAAQ,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;QAC3D,CAAC;QACD,IAAI,KAAK,CAAC,MAAM,IAAI,IAAI,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,CAAC;YACxD,KAAK,CAAC,IAAI,CAAC,WAAW,QAAQ,CAAC,KAAK,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;QACvD,CAAC;IACH,CAAC;IACD,IACE,MAAM,CAAC,MAAM;QACb,CAAC,MAAM,CAAC,EAAE;QACV,YAAY,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,QAAQ,CAAC,EACtD,CAAC;QACD,KAAK,CAAC,IAAI,CAAC,WAAW,QAAQ,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IACnD,CAAC;IACD,IAAI,CAAC,KAAK,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAC7B,OAAO,2CAA2C,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,kBAAkB,CAAC;AACnG,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,gBAAgB,CAAC,MAAc;IAC7C,MAAM,KAAK,GAAG,UAAU,CAAC,MAAM,CAAC,KAAK,IAAI,OAAO,CAAC,CAAC;IAClD,MAAM,OAAO,GAAG,UAAU,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IAC3C,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;IACzC,MAAM,KAAK,GAAG,UAAU,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;IACvC,MAAM,OAAO,GAAG,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC;IAEzC,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK;SACtB,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE;QACd,MAAM,EAAE,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC;QAC/B,MAAM,OAAO,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACvC,OAAO,cAAc,EAAE;2BACF,EAAE,KAAK,WAAW,CAAC,MAAM,CAAC;cACvC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;oBACf,MAAM,CAAC,WAAW;uBACf,OAAO;MACxB,CAAC;IACH,CAAC,CAAC;SACD,IAAI,CAAC,IAAI,CAAC,CAAC;IAEd,OAAO;;;;;sBAKa,KAAK;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;qBAiFN,OAAO;mBACT,OAAO;sBACJ,MAAM,MAAM,KAAK,MAAM,KAAK,MAAM,MAAM,CAAC,KAAK,CAAC,MAAM;yBAClD,MAAM,CAAC,MAAM,aAAa,MAAM,CAAC,MAAM,UAC5D,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,MAAM,CAAC,OAAO,UAAU,CAAC,CAAC,CAAC,EACpD,MAAM,MAAM,CAAC,WAAW;;;wEAG8C,MAAM,CAAC,MAAM;wEACb,MAAM,CAAC,MAAM;yEACZ,MAAM,CAAC,OAAO;0EACb,MAAM,CAAC,WAAW;;;;;;;;;EAS1F,IAAI,IAAI,mEAAmE;;;;;;;;;;;;;;;;;CAiB5E,CAAC;AACF,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -4,6 +4,7 @@ export { formatJUnit } from "./junit.js";
|
|
|
4
4
|
export { evaluateCase, evaluateExpect, equalsDiff, normalizeEquals } from "./matchers.js";
|
|
5
5
|
export { createComplete, mockComplete, resolveProvider } from "./providers.js";
|
|
6
6
|
export { formatReport } from "./report.js";
|
|
7
|
+
export { formatHtmlReport, escapeHtml } from "./html-report.js";
|
|
7
8
|
export { runSuite } from "./run.js";
|
|
8
9
|
export { estimateTokens } from "./tokens.js";
|
|
9
10
|
export { VERSION } from "./version.js";
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EACV,IAAI,EACJ,UAAU,EACV,SAAS,EACT,WAAW,EACX,UAAU,EACV,eAAe,EACf,MAAM,EACN,UAAU,EACV,YAAY,EACZ,MAAM,EACN,UAAU,EACV,KAAK,GACN,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,YAAY,EAAE,aAAa,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC/E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,gBAAgB,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAChE,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EACV,IAAI,EACJ,UAAU,EACV,SAAS,EACT,WAAW,EACX,UAAU,EACV,eAAe,EACf,MAAM,EACN,UAAU,EACV,YAAY,EACZ,MAAM,EACN,UAAU,EACV,KAAK,GACN,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,YAAY,EAAE,aAAa,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC/E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -4,6 +4,7 @@ export { formatJUnit } from "./junit.js";
|
|
|
4
4
|
export { evaluateCase, evaluateExpect, equalsDiff, normalizeEquals } from "./matchers.js";
|
|
5
5
|
export { createComplete, mockComplete, resolveProvider } from "./providers.js";
|
|
6
6
|
export { formatReport } from "./report.js";
|
|
7
|
+
export { formatHtmlReport, escapeHtml } from "./html-report.js";
|
|
7
8
|
export { runSuite } from "./run.js";
|
|
8
9
|
export { estimateTokens } from "./tokens.js";
|
|
9
10
|
export { VERSION } from "./version.js";
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAevC,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAE3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AACxD,OAAO,EAAE,SAAS,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,cAAc,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAC1F,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC/E,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,gBAAgB,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAChE,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAevC,OAAO,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAExC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAE3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,sBAAsB,EAAE,MAAM,qBAAqB,CAAC;AAC5G,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC"}
|
package/dist/load-suite.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"load-suite.d.ts","sourceRoot":"","sources":["../src/load-suite.ts"],"names":[],"mappings":"AAIA,OAAO,EAML,KAAK,KAAK,EACX,MAAM,YAAY,CAAC;
|
|
1
|
+
{"version":3,"file":"load-suite.d.ts","sourceRoot":"","sources":["../src/load-suite.ts"],"names":[],"mappings":"AAIA,OAAO,EAML,KAAK,KAAK,EACX,MAAM,YAAY,CAAC;AA+IpB,wBAAgB,UAAU,CAAC,GAAG,EAAE,OAAO,GAAG,KAAK,CAuC9C;AAyBD,wBAAgB,SAAS,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,SAAgB,GAAG,KAAK,CASlE"}
|
package/dist/load-suite.js
CHANGED
|
@@ -73,6 +73,10 @@ function parseExpect(raw, caseId) {
|
|
|
73
73
|
const expect = {
|
|
74
74
|
contains: parseStringList(raw.contains, `Case "${caseId}" expect.contains`),
|
|
75
75
|
not_contains: parseStringList(raw.not_contains, `Case "${caseId}" expect.not_contains`),
|
|
76
|
+
contains_any: parseStringList(raw.contains_any, `Case "${caseId}" expect.contains_any`),
|
|
77
|
+
contains_all: parseStringList(raw.contains_all, `Case "${caseId}" expect.contains_all`),
|
|
78
|
+
starts_with: parseStringList(raw.starts_with, `Case "${caseId}" expect.starts_with`),
|
|
79
|
+
ends_with: parseStringList(raw.ends_with, `Case "${caseId}" expect.ends_with`),
|
|
76
80
|
regex: parseStringList(raw.regex, `Case "${caseId}" expect.regex`),
|
|
77
81
|
};
|
|
78
82
|
if (raw.equals != null) {
|
|
@@ -93,7 +97,7 @@ function parseExpect(raw, caseId) {
|
|
|
93
97
|
throw new UsageError(`Case "${caseId}" expect.min_tokens (${expect.min_tokens}) cannot exceed max_tokens (${expect.max_tokens})`);
|
|
94
98
|
}
|
|
95
99
|
if (!hasMatchers(expect)) {
|
|
96
|
-
throw new UsageError(`Case "${caseId}" expect needs at least one of contains, not_contains, regex, equals, json, min_tokens, max_tokens`);
|
|
100
|
+
throw new UsageError(`Case "${caseId}" expect needs at least one of contains, not_contains, contains_any, contains_all, starts_with, ends_with, regex, equals, json, min_tokens, max_tokens`);
|
|
97
101
|
}
|
|
98
102
|
for (const rawRegex of asList(expect.regex)) {
|
|
99
103
|
try {
|
package/dist/load-suite.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"load-suite.js","sourceRoot":"","sources":["../src/load-suite.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAC7C,OAAO,EAAE,KAAK,IAAI,SAAS,EAAE,MAAM,MAAM,CAAC;AAC1C,OAAO,EAAE,MAAM,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,eAAe,CAAC;AAClE,OAAO,EACL,UAAU,GAMX,MAAM,YAAY,CAAC;AAEpB,MAAM,SAAS,GAAG,IAAI,GAAG,CAAe,CAAC,MAAM,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC;AAEzE,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC9E,CAAC;AAED,SAAS,QAAQ,CAAC,KAAc,EAAE,KAAa;IAC7C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC;QACrD,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,6BAA6B,CAAC,CAAC;IAC9D,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,cAAc,CAAC,KAAc,EAAE,KAAa;IACnD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,OAAO,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;AAChC,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,KAAK,KAAK,EAAE;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;QACrE,OAAO,KAAK,CAAC;IACf,CAAC;IACD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,4BAA4B,CAAC,CAAC;QACnF,OAAO,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,EAAE,GAAG,KAAK,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC;IAC1E,CAAC;IACD,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,sCAAsC,CAAC,CAAC;AACvE,CAAC;AAED,SAAS,gBAAgB,CAAC,KAAc,EAAE,KAAa;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QACxE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,6BAA6B,CAAC,CAAC;IAC9D,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,MAAc;IACrD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,IAAI,CAAC;IAChC,IAAI,KAAK,KAAK,KAAK,EAAE,CAAC;QACpB,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,0DAA0D,CAAC,CAAC;IAClG,CAAC;IACD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,OAAO,KAAmB,CAAC;IAC7B,CAAC;IACD,qDAAqD;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAY,CAAC;YAC5C,IAAI,MAAM,KAAK,IAAI;gBAAE,OAAO,IAAI,CAAC;YACjC,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,QAAQ,CAAC,MAAM,CAAC;gBAAE,OAAO,MAAoB,CAAC;QAC7E,CAAC;QAAC,MAAM,CAAC;YACP,eAAe;QACjB,CAAC;QACD,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,4EAA4E,CAC5F,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,oDAAoD,CAAC,CAAC;AAC5F,CAAC;AAED,SAAS,WAAW,CAAC,GAAY,EAAE,MAAc;IAC/C,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;QACnB,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,oCAAoC,CAAC,CAAC;IAC5E,CAAC;IAED,MAAM,MAAM,GAAW;QACrB,QAAQ,EAAE,eAAe,CAAC,GAAG,CAAC,QAAQ,EAAE,SAAS,MAAM,mBAAmB,CAAC;QAC3E,YAAY,EAAE,eAAe,CAAC,GAAG,CAAC,YAAY,EAAE,SAAS,MAAM,uBAAuB,CAAC;QACvF,KAAK,EAAE,eAAe,CAAC,GAAG,CAAC,KAAK,EAAE,SAAS,MAAM,gBAAgB,CAAC;KACnE,CAAC;IAEF,IAAI,GAAG,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,CAAC,MAAM,GAAG,QAAQ,CAAC,GAAG,CAAC,MAAM,EAAE,SAAS,MAAM,iBAAiB,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,IAAI,GAAG,eAAe,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAC/C,IAAI,IAAI,KAAK,SAAS;QAAE,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC;IAE3C,IAAI,GAAG,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC3B,MAAM,CAAC,UAAU,GAAG,gBAAgB,CAAC,GAAG,CAAC,UAAU,EAAE,SAAS,MAAM,qBAAqB,CAAC,CAAC;IAC7F,CAAC;IACD,IAAI,GAAG,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC3B,MAAM,CAAC,UAAU,GAAG,gBAAgB,CAAC,GAAG,CAAC,UAAU,EAAE,SAAS,MAAM,qBAAqB,CAAC,CAAC;IAC7F,CAAC;IAED,IACE,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,GAAG,MAAM,CAAC,UAAU,EACrC,CAAC;QACD,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,wBAAwB,MAAM,CAAC,UAAU,+BAA+B,MAAM,CAAC,UAAU,GAAG,CAC5G,CAAC;IACJ,CAAC;IAED,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC,EAAE,CAAC;QACzB,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,
|
|
1
|
+
{"version":3,"file":"load-suite.js","sourceRoot":"","sources":["../src/load-suite.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAC7C,OAAO,EAAE,KAAK,IAAI,SAAS,EAAE,MAAM,MAAM,CAAC;AAC1C,OAAO,EAAE,MAAM,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,eAAe,CAAC;AAClE,OAAO,EACL,UAAU,GAMX,MAAM,YAAY,CAAC;AAEpB,MAAM,SAAS,GAAG,IAAI,GAAG,CAAe,CAAC,MAAM,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC;AAEzE,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC9E,CAAC;AAED,SAAS,QAAQ,CAAC,KAAc,EAAE,KAAa;IAC7C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC;QACrD,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,6BAA6B,CAAC,CAAC;IAC9D,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,cAAc,CAAC,KAAc,EAAE,KAAa;IACnD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,OAAO,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;AAChC,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,KAAK,KAAK,EAAE;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;QACrE,OAAO,KAAK,CAAC;IACf,CAAC;IACD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,4BAA4B,CAAC,CAAC;QACnF,OAAO,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,EAAE,GAAG,KAAK,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC;IAC1E,CAAC;IACD,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,sCAAsC,CAAC,CAAC;AACvE,CAAC;AAED,SAAS,gBAAgB,CAAC,KAAc,EAAE,KAAa;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QACxE,MAAM,IAAI,UAAU,CAAC,GAAG,KAAK,6BAA6B,CAAC,CAAC;IAC9D,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,MAAc;IACrD,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,IAAI,CAAC;IAChC,IAAI,KAAK,KAAK,KAAK,EAAE,CAAC;QACpB,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,0DAA0D,CAAC,CAAC;IAClG,CAAC;IACD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,OAAO,KAAmB,CAAC;IAC7B,CAAC;IACD,qDAAqD;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAY,CAAC;YAC5C,IAAI,MAAM,KAAK,IAAI;gBAAE,OAAO,IAAI,CAAC;YACjC,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,QAAQ,CAAC,MAAM,CAAC;gBAAE,OAAO,MAAoB,CAAC;QAC7E,CAAC;QAAC,MAAM,CAAC;YACP,eAAe;QACjB,CAAC;QACD,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,4EAA4E,CAC5F,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,oDAAoD,CAAC,CAAC;AAC5F,CAAC;AAED,SAAS,WAAW,CAAC,GAAY,EAAE,MAAc;IAC/C,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;QACnB,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,oCAAoC,CAAC,CAAC;IAC5E,CAAC;IAED,MAAM,MAAM,GAAW;QACrB,QAAQ,EAAE,eAAe,CAAC,GAAG,CAAC,QAAQ,EAAE,SAAS,MAAM,mBAAmB,CAAC;QAC3E,YAAY,EAAE,eAAe,CAAC,GAAG,CAAC,YAAY,EAAE,SAAS,MAAM,uBAAuB,CAAC;QACvF,YAAY,EAAE,eAAe,CAAC,GAAG,CAAC,YAAY,EAAE,SAAS,MAAM,uBAAuB,CAAC;QACvF,YAAY,EAAE,eAAe,CAAC,GAAG,CAAC,YAAY,EAAE,SAAS,MAAM,uBAAuB,CAAC;QACvF,WAAW,EAAE,eAAe,CAAC,GAAG,CAAC,WAAW,EAAE,SAAS,MAAM,sBAAsB,CAAC;QACpF,SAAS,EAAE,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,SAAS,MAAM,oBAAoB,CAAC;QAC9E,KAAK,EAAE,eAAe,CAAC,GAAG,CAAC,KAAK,EAAE,SAAS,MAAM,gBAAgB,CAAC;KACnE,CAAC;IAEF,IAAI,GAAG,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,CAAC,MAAM,GAAG,QAAQ,CAAC,GAAG,CAAC,MAAM,EAAE,SAAS,MAAM,iBAAiB,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,IAAI,GAAG,eAAe,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAC/C,IAAI,IAAI,KAAK,SAAS;QAAE,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC;IAE3C,IAAI,GAAG,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC3B,MAAM,CAAC,UAAU,GAAG,gBAAgB,CAAC,GAAG,CAAC,UAAU,EAAE,SAAS,MAAM,qBAAqB,CAAC,CAAC;IAC7F,CAAC;IACD,IAAI,GAAG,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC3B,MAAM,CAAC,UAAU,GAAG,gBAAgB,CAAC,GAAG,CAAC,UAAU,EAAE,SAAS,MAAM,qBAAqB,CAAC,CAAC;IAC7F,CAAC;IAED,IACE,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,GAAG,MAAM,CAAC,UAAU,EACrC,CAAC;QACD,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,wBAAwB,MAAM,CAAC,UAAU,+BAA+B,MAAM,CAAC,UAAU,GAAG,CAC5G,CAAC;IACJ,CAAC;IAED,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC,EAAE,CAAC;QACzB,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,wJAAwJ,CACxK,CAAC;IACJ,CAAC;IAED,KAAK,MAAM,QAAQ,IAAI,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;QAC5C,IAAI,CAAC;YACH,YAAY,CAAC,QAAQ,CAAC,CAAC;QACzB,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,IAAI,UAAU,CAAC,SAAS,MAAM,2BAA2B,QAAQ,EAAE,CAAC,CAAC;QAC7E,CAAC;IACH,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,SAAS,CAAC,GAAY,EAAE,KAAa;IAC5C,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;QACnB,MAAM,IAAI,UAAU,CAAC,iBAAiB,KAAK,oBAAoB,CAAC,CAAC;IACnE,CAAC;IACD,MAAM,EAAE,GAAG,QAAQ,CAAC,GAAG,CAAC,EAAE,EAAE,iBAAiB,KAAK,KAAK,CAAC,CAAC;IACzD,OAAO;QACL,EAAE;QACF,MAAM,EAAE,QAAQ,CAAC,GAAG,CAAC,MAAM,EAAE,SAAS,EAAE,UAAU,CAAC;QACnD,MAAM,EAAE,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,SAAS,EAAE,UAAU,CAAC;QACzD,KAAK,EAAE,cAAc,CAAC,GAAG,CAAC,KAAK,EAAE,SAAS,EAAE,SAAS,CAAC;QACtD,MAAM,EAAE,WAAW,CAAC,GAAG,CAAC,MAAM,EAAE,EAAE,CAAC;KACpC,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,KAAqB,CAAC,EAAE,CAAC;QACvE,MAAM,IAAI,UAAU,CAAC,2BAA2B,CAAC,GAAG,SAAS,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IAC/E,CAAC;IACD,OAAO,KAAqB,CAAC;AAC/B,CAAC;AAED,MAAM,UAAU,UAAU,CAAC,GAAY;IACrC,IAAI,IAAwB,CAAC;IAC7B,IAAI,KAAyB,CAAC;IAC9B,IAAI,QAAkC,CAAC;IACvC,IAAI,MAA0B,CAAC;IAC/B,IAAI,QAAmB,CAAC;IAExB,IAAI,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;QACvB,QAAQ,GAAG,GAAG,CAAC;IACjB,CAAC;SAAM,IAAI,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;QACzB,IAAI,GAAG,cAAc,CAAC,GAAG,CAAC,IAAI,EAAE,YAAY,CAAC,CAAC;QAC9C,KAAK,GAAG,cAAc,CAAC,GAAG,CAAC,KAAK,EAAE,aAAa,CAAC,CAAC;QACjD,QAAQ,GAAG,aAAa,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACvC,MAAM,GAAG,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC;QACpD,IAAI,GAAG,CAAC,KAAK,IAAI,IAAI,IAAI,IAAI,IAAI,GAAG,IAAI,QAAQ,IAAI,GAAG,EAAE,CAAC;YACxD,QAAQ,GAAG,CAAC,GAAG,CAAC,CAAC;QACnB,CAAC;aAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,UAAU,CAAC,+DAA+D,CAAC,CAAC;QACxF,CAAC;aAAM,CAAC;YACN,QAAQ,GAAG,GAAG,CAAC,KAAK,CAAC;QACvB,CAAC;IACH,CAAC;SAAM,CAAC;QACN,MAAM,IAAI,UAAU,CAAC,0CAA0C,CAAC,CAAC;IACnE,CAAC;IAED,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC1B,MAAM,IAAI,UAAU,CAAC,oBAAoB,CAAC,CAAC;IAC7C,CAAC;IAED,MAAM,KAAK,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAC;IACpE,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;IAC/B,KAAK,MAAM,QAAQ,IAAI,KAAK,EAAE,CAAC;QAC7B,IAAI,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,CAAC;YAC1B,MAAM,IAAI,UAAU,CAAC,sBAAsB,QAAQ,CAAC,EAAE,GAAG,CAAC,CAAC;QAC7D,CAAC;QACD,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;IACxB,CAAC;IAED,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AAClD,CAAC;AAED,SAAS,aAAa,CAAC,IAAY,EAAE,MAAc;IACjD,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACzC,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,WAAW,EAAE,CAAC;IAC1C,IAAI,CAAC;QACH,IAAI,GAAG,KAAK,OAAO;YAAE,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAY,CAAC;QACxD,IAAI,GAAG,KAAK,OAAO,IAAI,GAAG,KAAK,MAAM;YAAE,OAAO,SAAS,CAAC,IAAI,CAAC,CAAC;IAChE,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACtE,MAAM,IAAI,UAAU,CAAC,mBAAmB,MAAM,KAAK,MAAM,EAAE,CAAC,CAAC;IAC/D,CAAC;IAED,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAY,CAAC;IACrC,CAAC;IAAC,MAAM,CAAC;QACP,IAAI,CAAC;YACH,OAAO,SAAS,CAAC,IAAI,CAAC,CAAC;QACzB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACtE,MAAM,IAAI,UAAU,CAAC,mBAAmB,MAAM,KAAK,MAAM,EAAE,CAAC,CAAC;QAC/D,CAAC;IACH,CAAC;AACH,CAAC;AAED,MAAM,UAAU,SAAS,CAAC,IAAY,EAAE,GAAG,GAAG,OAAO,CAAC,GAAG,EAAE;IACzD,MAAM,IAAI,GAAG,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IAChC,IAAI,IAAY,CAAC;IACjB,IAAI,CAAC;QACH,IAAI,GAAG,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IACpC,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,UAAU,CAAC,yBAAyB,IAAI,EAAE,CAAC,CAAC;IACxD,CAAC;IACD,OAAO,UAAU,CAAC,aAAa,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;AAC/C,CAAC"}
|
package/dist/matchers.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"matchers.d.ts","sourceRoot":"","sources":["../src/matchers.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,IAAI,EAAE,WAAW,EAAE,MAAM,EAAc,MAAM,YAAY,CAAC;AAExE,wBAAgB,MAAM,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,MAAM,EAAE,CAG1D;AAID,wBAAgB,YAAY,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,CA8BhD;AAED,uEAAuE;AACvE,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAIpD;AAED,wBAAgB,WAAW,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,
|
|
1
|
+
{"version":3,"file":"matchers.d.ts","sourceRoot":"","sources":["../src/matchers.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,IAAI,EAAE,WAAW,EAAE,MAAM,EAAc,MAAM,YAAY,CAAC;AAExE,wBAAgB,MAAM,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,MAAM,EAAE,CAG1D;AAID,wBAAgB,YAAY,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,CA8BhD;AAED,uEAAuE;AACvE,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAIpD;AAED,wBAAgB,WAAW,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAcnD;AAQD,wEAAwE;AACxE,wBAAgB,UAAU,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,MAAM,CAWnE;AAwDD,wBAAgB,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,WAAW,EAAE,CAgJ5E;AAED,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,GAAG,WAAW,EAAE,CAE1E"}
|
package/dist/matchers.js
CHANGED
|
@@ -47,6 +47,10 @@ export function normalizeEquals(text) {
|
|
|
47
47
|
export function hasMatchers(expect) {
|
|
48
48
|
return (asList(expect.contains).length > 0 ||
|
|
49
49
|
asList(expect.not_contains).length > 0 ||
|
|
50
|
+
asList(expect.contains_any).length > 0 ||
|
|
51
|
+
asList(expect.contains_all).length > 0 ||
|
|
52
|
+
asList(expect.starts_with).length > 0 ||
|
|
53
|
+
asList(expect.ends_with).length > 0 ||
|
|
50
54
|
asList(expect.regex).length > 0 ||
|
|
51
55
|
expect.equals != null ||
|
|
52
56
|
expect.json != null ||
|
|
@@ -151,6 +155,66 @@ export function evaluateExpect(output, expect) {
|
|
|
151
155
|
detail: ok ? undefined : `found ${JSON.stringify(needle)}`,
|
|
152
156
|
});
|
|
153
157
|
}
|
|
158
|
+
{
|
|
159
|
+
const needles = asList(expect.contains_any);
|
|
160
|
+
if (needles.length) {
|
|
161
|
+
const hay = output.toLowerCase();
|
|
162
|
+
const hit = needles.find((n) => hay.includes(n.toLowerCase()));
|
|
163
|
+
const ok = hit != null;
|
|
164
|
+
checks.push({
|
|
165
|
+
name: "contains_any",
|
|
166
|
+
ok,
|
|
167
|
+
expected: needles.join(" | "),
|
|
168
|
+
actual: ok ? undefined : clip(output),
|
|
169
|
+
detail: ok ? undefined : `none of ${JSON.stringify(needles)} found`,
|
|
170
|
+
});
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
{
|
|
174
|
+
const needles = asList(expect.contains_all);
|
|
175
|
+
if (needles.length) {
|
|
176
|
+
const hay = output.toLowerCase();
|
|
177
|
+
const missing = needles.filter((n) => !hay.includes(n.toLowerCase()));
|
|
178
|
+
const ok = missing.length === 0;
|
|
179
|
+
checks.push({
|
|
180
|
+
name: "contains_all",
|
|
181
|
+
ok,
|
|
182
|
+
expected: needles.join(" & "),
|
|
183
|
+
actual: ok ? undefined : clip(output),
|
|
184
|
+
detail: ok ? undefined : `missing ${JSON.stringify(missing)}`,
|
|
185
|
+
});
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
{
|
|
189
|
+
const needles = asList(expect.starts_with);
|
|
190
|
+
if (needles.length) {
|
|
191
|
+
const hay = output.toLowerCase();
|
|
192
|
+
const hit = needles.find((n) => hay.startsWith(n.toLowerCase()));
|
|
193
|
+
const ok = hit != null;
|
|
194
|
+
checks.push({
|
|
195
|
+
name: "starts_with",
|
|
196
|
+
ok,
|
|
197
|
+
expected: needles.join(" | "),
|
|
198
|
+
actual: ok ? undefined : clip(output),
|
|
199
|
+
detail: ok ? undefined : `does not start with any of ${JSON.stringify(needles)}`,
|
|
200
|
+
});
|
|
201
|
+
}
|
|
202
|
+
}
|
|
203
|
+
{
|
|
204
|
+
const needles = asList(expect.ends_with);
|
|
205
|
+
if (needles.length) {
|
|
206
|
+
const hay = output.toLowerCase();
|
|
207
|
+
const hit = needles.find((n) => hay.endsWith(n.toLowerCase()));
|
|
208
|
+
const ok = hit != null;
|
|
209
|
+
checks.push({
|
|
210
|
+
name: "ends_with",
|
|
211
|
+
ok,
|
|
212
|
+
expected: needles.join(" | "),
|
|
213
|
+
actual: ok ? undefined : clip(output),
|
|
214
|
+
detail: ok ? undefined : `does not end with any of ${JSON.stringify(needles)}`,
|
|
215
|
+
});
|
|
216
|
+
}
|
|
217
|
+
}
|
|
154
218
|
for (const raw of asList(expect.regex)) {
|
|
155
219
|
const re = compileRegex(raw);
|
|
156
220
|
re.lastIndex = 0;
|
package/dist/matchers.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"matchers.js","sourceRoot":"","sources":["../src/matchers.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAG7C,MAAM,UAAU,MAAM,CAAC,KAAyB;IAC9C,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,EAAE,CAAC;IAC7B,OAAO,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;AAChD,CAAC;AAED,MAAM,gBAAgB,GAAG,GAAG,CAAC;AAE7B,MAAM,UAAU,YAAY,CAAC,GAAW;IACtC,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAChD,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;IAC9D,CAAC;IACD,IAAI,GAAG,CAAC,MAAM,GAAG,gBAAgB,EAAE,CAAC;QAClC,MAAM,IAAI,KAAK,CAAC,+BAA+B,gBAAgB,SAAS,CAAC,CAAC;IAC5E,CAAC;IACD,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,GAAG,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;QACxD,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;YACzB,IAAI,OAAO,CAAC,CAAC,CAAC,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC5B,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;YAC9D,CAAC;YACD,OAAO,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;QAC5C,CAAC;QACD,IAAI,GAAG,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;YAC3B,MAAM,IAAI,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YAC1B,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBACtB,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;YAC9D,CAAC;YACD,OAAO,IAAI,MAAM,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;QAC/B,CAAC;QACD,OAAO,IAAI,MAAM,CAAC,GAAG,CAAC,CAAC;IACzB,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,KAAK,IAAI,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,CAAC;YACvE,MAAM,KAAK,CAAC;QACd,CAAC;QACD,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACtE,MAAM,IAAI,KAAK,CAAC,kBAAkB,MAAM,EAAE,CAAC,CAAC;IAC9C,CAAC;AACH,CAAC;AAED,uEAAuE;AACvE,MAAM,UAAU,eAAe,CAAC,IAAY;IAC1C,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC;IACvD,IAAI,GAAG,CAAC,QAAQ,CAAC,IAAI,CAAC;QAAE,GAAG,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAC/C,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,UAAU,WAAW,CAAC,MAAc;IACxC,OAAO,CACL,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;QAClC,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,MAAM,GAAG,CAAC;QACtC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,GAAG,CAAC;QAC/B,MAAM,CAAC,MAAM,IAAI,IAAI;QACrB,MAAM,CAAC,IAAI,IAAI,IAAI;QACnB,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,IAAI,IAAI,CAC1B,CAAC;AACJ,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,GAAG,GAAG,GAAG;IACnC,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IACjD,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG;QAAE,OAAO,OAAO,CAAC;IAC1C,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,CAAC;AACzC,CAAC;AAED,wEAAwE;AACxE,MAAM,UAAU,UAAU,CAAC,QAAgB,EAAE,MAAc;IACzD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC;IACrD,IAAI,CAAC,GAAG,CAAC,CAAC;IACV,OAAO,CAAC,GAAG,GAAG,IAAI,QAAQ,CAAC,CAAC,CAAC,KAAK,MAAM,CAAC,CAAC,CAAC;QAAE,CAAC,IAAI,CAAC,CAAC;IACpD,IAAI,CAAC,IAAI,GAAG;QAAE,OAAO,sCAAsC,CAAC;IAC5D,MAAM,MAAM,GAAG,EAAE,CAAC;IAClB,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;IACjC,MAAM,QAAQ,GAAG,QAAQ,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,MAAM,CAAC,CAAC;IACvD,MAAM,QAAQ,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,MAAM,CAAC,CAAC;IACrD,MAAM,OAAO,GAAG,GAAG,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC;IAC5C,OAAO,uBAAuB,CAAC,yBAAyB,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,yBAAyB,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,yBAAyB,OAAO,EAAE,CAAC;AACtK,CAAC;AAED,SAAS,SAAS,CAAC,CAAU,EAAE,CAAU;IACvC,IAAI,MAAM,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;QAAE,OAAO,IAAI,CAAC;IACjC,IAAI,OAAO,CAAC,KAAK,OAAO,CAAC;QAAE,OAAO,KAAK,CAAC;IACxC,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,IAAI;QAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IAC7C,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QACzC,IAAI,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,MAAM;YAAE,OAAO,KAAK,CAAC;QACxC,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IAC7D,CAAC;IACD,IAAI,OAAO,CAAC,KAAK,QAAQ,IAAI,OAAO,CAAC,KAAK,QAAQ,EAAE,CAAC;QACnD,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,CAAW,CAAC,CAAC,IAAI,EAAE,CAAC;QAC9C,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,CAAW,CAAC,CAAC,IAAI,EAAE,CAAC;QAC9C,IAAI,KAAK,CAAC,MAAM,KAAK,KAAK,CAAC,MAAM;YAAE,OAAO,KAAK,CAAC;QAChD,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,KAAK,KAAK,CAAC,KAAK,CAAC,CAAC;YAAE,OAAO,KAAK,CAAC;QACrE,OAAO,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE,CACzB,SAAS,CAAE,CAA6B,CAAC,GAAG,CAAC,EAAG,CAA6B,CAAC,GAAG,CAAC,CAAC,CACpF,CAAC;IACJ,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,YAAY,CAAC,MAAc,EAAE,MAAkB;IACtD,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAY,CAAC;IACzC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACtE,OAAO;YACL,IAAI,EAAE,MAAM;YACZ,EAAE,EAAE,KAAK;YACT,QAAQ,EAAE,MAAM,KAAK,IAAI,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;YACjE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC;YACpB,MAAM,EAAE,iBAAiB,MAAM,EAAE;SAClC,CAAC;IACJ,CAAC;IAED,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;QACpB,OAAO;YACL,IAAI,EAAE,MAAM;YACZ,EAAE,EAAE,IAAI;YACR,QAAQ,EAAE,YAAY;YACtB,MAAM,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC;SACrC,CAAC;IACJ,CAAC;IAED,MAAM,EAAE,GAAG,SAAS,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACrC,OAAO;QACL,IAAI,EAAE,MAAM;QACZ,EAAE;QACF,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;QAChC,MAAM,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;QAC9B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,4CAA4C;KACtE,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,MAAc,EAAE,MAAc;IAC3D,MAAM,MAAM,GAAkB,EAAE,CAAC;IAEjC,KAAK,MAAM,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;QAC7C,MAAM,EAAE,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;QAC/D,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,UAAU;YAChB,EAAE;YACF,QAAQ,EAAE,MAAM;YAChB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE;SAC7D,CAAC,CAAC;IACL,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,CAAC;QACjD,MAAM,EAAE,GAAG,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;QAChE,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,cAAc;YACpB,EAAE;YACF,QAAQ,EAAE,MAAM;YAChB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE;SAC3D,CAAC,CAAC;IACL,CAAC;IAED,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;QACvC,MAAM,EAAE,GAAG,YAAY,CAAC,GAAG,CAAC,CAAC;QAC7B,EAAE,CAAC,SAAS,GAAG,CAAC,CAAC;QACjB,MAAM,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAC3B,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,OAAO;YACb,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,EAAE,CAAC;YACpB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,gBAAgB,EAAE,EAAE;SAC9C,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;QAC1B,MAAM,QAAQ,GAAG,eAAe,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAChD,MAAM,MAAM,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,EAAE,GAAG,QAAQ,KAAK,MAAM,CAAC;QAC/B,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,QAAQ;YACd,EAAE;YACF,QAAQ;YACR,MAAM;YACN,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,QAAQ,EAAE,MAAM,CAAC;SACtD,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,MAAM,CAAC,IAAI,CAAC,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACjD,CAAC;IAED,IAAI,MAAM,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACtC,MAAM,EAAE,GAAG,MAAM,IAAI,MAAM,CAAC,UAAU,CAAC;QACvC,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,YAAY;YAClB,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC;YACnC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC;YACtB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,MAAM,aAAa,MAAM,CAAC,UAAU,EAAE;SAC7E,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACtC,MAAM,EAAE,GAAG,MAAM,IAAI,MAAM,CAAC,UAAU,CAAC;QACvC,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,YAAY;YAClB,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC;YACnC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC;YACtB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,MAAM,aAAa,MAAM,CAAC,UAAU,EAAE;SAC7E,CAAC,CAAC;IACL,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,MAAM,UAAU,YAAY,CAAC,MAAc,EAAE,QAAc;IACzD,OAAO,cAAc,CAAC,MAAM,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC;AACjD,CAAC"}
|
|
1
|
+
{"version":3,"file":"matchers.js","sourceRoot":"","sources":["../src/matchers.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAG7C,MAAM,UAAU,MAAM,CAAC,KAAyB;IAC9C,IAAI,KAAK,IAAI,IAAI;QAAE,OAAO,EAAE,CAAC;IAC7B,OAAO,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;AAChD,CAAC;AAED,MAAM,gBAAgB,GAAG,GAAG,CAAC;AAE7B,MAAM,UAAU,YAAY,CAAC,GAAW;IACtC,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAChD,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;IAC9D,CAAC;IACD,IAAI,GAAG,CAAC,MAAM,GAAG,gBAAgB,EAAE,CAAC;QAClC,MAAM,IAAI,KAAK,CAAC,+BAA+B,gBAAgB,SAAS,CAAC,CAAC;IAC5E,CAAC;IACD,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,GAAG,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;QACxD,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;YACzB,IAAI,OAAO,CAAC,CAAC,CAAC,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC5B,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;YAC9D,CAAC;YACD,OAAO,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;QAC5C,CAAC;QACD,IAAI,GAAG,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;YAC3B,MAAM,IAAI,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YAC1B,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBACtB,MAAM,IAAI,KAAK,CAAC,0CAA0C,CAAC,CAAC;YAC9D,CAAC;YACD,OAAO,IAAI,MAAM,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;QAC/B,CAAC;QACD,OAAO,IAAI,MAAM,CAAC,GAAG,CAAC,CAAC;IACzB,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,KAAK,IAAI,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,CAAC;YACvE,MAAM,KAAK,CAAC;QACd,CAAC;QACD,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACtE,MAAM,IAAI,KAAK,CAAC,kBAAkB,MAAM,EAAE,CAAC,CAAC;IAC9C,CAAC;AACH,CAAC;AAED,uEAAuE;AACvE,MAAM,UAAU,eAAe,CAAC,IAAY;IAC1C,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC;IACvD,IAAI,GAAG,CAAC,QAAQ,CAAC,IAAI,CAAC;QAAE,GAAG,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAC/C,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,UAAU,WAAW,CAAC,MAAc;IACxC,OAAO,CACL,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;QAClC,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,MAAM,GAAG,CAAC;QACtC,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,MAAM,GAAG,CAAC;QACtC,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,MAAM,GAAG,CAAC;QACtC,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC,MAAM,GAAG,CAAC;QACrC,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,GAAG,CAAC;QACnC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,GAAG,CAAC;QAC/B,MAAM,CAAC,MAAM,IAAI,IAAI;QACrB,MAAM,CAAC,IAAI,IAAI,IAAI;QACnB,MAAM,CAAC,UAAU,IAAI,IAAI;QACzB,MAAM,CAAC,UAAU,IAAI,IAAI,CAC1B,CAAC;AACJ,CAAC;AAED,SAAS,IAAI,CAAC,IAAY,EAAE,GAAG,GAAG,GAAG;IACnC,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IACjD,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG;QAAE,OAAO,OAAO,CAAC;IAC1C,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,CAAC;AACzC,CAAC;AAED,wEAAwE;AACxE,MAAM,UAAU,UAAU,CAAC,QAAgB,EAAE,MAAc;IACzD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC;IACrD,IAAI,CAAC,GAAG,CAAC,CAAC;IACV,OAAO,CAAC,GAAG,GAAG,IAAI,QAAQ,CAAC,CAAC,CAAC,KAAK,MAAM,CAAC,CAAC,CAAC;QAAE,CAAC,IAAI,CAAC,CAAC;IACpD,IAAI,CAAC,IAAI,GAAG;QAAE,OAAO,sCAAsC,CAAC;IAC5D,MAAM,MAAM,GAAG,EAAE,CAAC;IAClB,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;IACjC,MAAM,QAAQ,GAAG,QAAQ,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,MAAM,CAAC,CAAC;IACvD,MAAM,QAAQ,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,MAAM,CAAC,CAAC;IACrD,MAAM,OAAO,GAAG,GAAG,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC;IAC5C,OAAO,uBAAuB,CAAC,yBAAyB,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,yBAAyB,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,yBAAyB,OAAO,EAAE,CAAC;AACtK,CAAC;AAED,SAAS,SAAS,CAAC,CAAU,EAAE,CAAU;IACvC,IAAI,MAAM,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;QAAE,OAAO,IAAI,CAAC;IACjC,IAAI,OAAO,CAAC,KAAK,OAAO,CAAC;QAAE,OAAO,KAAK,CAAC;IACxC,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,IAAI;QAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IAC7C,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QACzC,IAAI,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,MAAM;YAAE,OAAO,KAAK,CAAC;QACxC,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IAC7D,CAAC;IACD,IAAI,OAAO,CAAC,KAAK,QAAQ,IAAI,OAAO,CAAC,KAAK,QAAQ,EAAE,CAAC;QACnD,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,CAAW,CAAC,CAAC,IAAI,EAAE,CAAC;QAC9C,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,CAAW,CAAC,CAAC,IAAI,EAAE,CAAC;QAC9C,IAAI,KAAK,CAAC,MAAM,KAAK,KAAK,CAAC,MAAM;YAAE,OAAO,KAAK,CAAC;QAChD,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,KAAK,KAAK,CAAC,KAAK,CAAC,CAAC;YAAE,OAAO,KAAK,CAAC;QACrE,OAAO,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE,CACzB,SAAS,CAAE,CAA6B,CAAC,GAAG,CAAC,EAAG,CAA6B,CAAC,GAAG,CAAC,CAAC,CACpF,CAAC;IACJ,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,YAAY,CAAC,MAAc,EAAE,MAAkB;IACtD,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAY,CAAC;IACzC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,MAAM,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACtE,OAAO;YACL,IAAI,EAAE,MAAM;YACZ,EAAE,EAAE,KAAK;YACT,QAAQ,EAAE,MAAM,KAAK,IAAI,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;YACjE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC;YACpB,MAAM,EAAE,iBAAiB,MAAM,EAAE;SAClC,CAAC;IACJ,CAAC;IAED,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;QACpB,OAAO;YACL,IAAI,EAAE,MAAM;YACZ,EAAE,EAAE,IAAI;YACR,QAAQ,EAAE,YAAY;YACtB,MAAM,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC;SACrC,CAAC;IACJ,CAAC;IAED,MAAM,EAAE,GAAG,SAAS,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACrC,OAAO;QACL,IAAI,EAAE,MAAM;QACZ,EAAE;QACF,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;QAChC,MAAM,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;QAC9B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,4CAA4C;KACtE,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,MAAc,EAAE,MAAc;IAC3D,MAAM,MAAM,GAAkB,EAAE,CAAC;IAEjC,KAAK,MAAM,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;QAC7C,MAAM,EAAE,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;QAC/D,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,UAAU;YAChB,EAAE;YACF,QAAQ,EAAE,MAAM;YAChB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE;SAC7D,CAAC,CAAC;IACL,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,CAAC;QACjD,MAAM,EAAE,GAAG,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;QAChE,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,cAAc;YACpB,EAAE;YACF,QAAQ,EAAE,MAAM;YAChB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE;SAC3D,CAAC,CAAC;IACL,CAAC;IAED,CAAC;QACC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QAC5C,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,GAAG,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;YACjC,MAAM,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;YAC/D,MAAM,EAAE,GAAG,GAAG,IAAI,IAAI,CAAC;YACvB,MAAM,CAAC,IAAI,CAAC;gBACV,IAAI,EAAE,cAAc;gBACpB,EAAE;gBACF,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC;gBAC7B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,QAAQ;aACpE,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,CAAC;QACC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QAC5C,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,GAAG,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;YACjC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;YACtE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC;YAChC,MAAM,CAAC,IAAI,CAAC;gBACV,IAAI,EAAE,cAAc;gBACpB,EAAE;gBACF,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC;gBAC7B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,EAAE;aAC9D,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,CAAC;QACC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;QAC3C,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,GAAG,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;YACjC,MAAM,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;YACjE,MAAM,EAAE,GAAG,GAAG,IAAI,IAAI,CAAC;YACvB,MAAM,CAAC,IAAI,CAAC;gBACV,IAAI,EAAE,aAAa;gBACnB,EAAE;gBACF,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC;gBAC7B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,8BAA8B,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,EAAE;aACjF,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,CAAC;QACC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACzC,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,GAAG,GAAG,MAAM,CAAC,WAAW,EAAE,CAAC;YACjC,MAAM,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;YAC/D,MAAM,EAAE,GAAG,GAAG,IAAI,IAAI,CAAC;YACvB,MAAM,CAAC,IAAI,CAAC;gBACV,IAAI,EAAE,WAAW;gBACjB,EAAE;gBACF,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC;gBAC7B,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,4BAA4B,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,EAAE;aAC/E,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;QACvC,MAAM,EAAE,GAAG,YAAY,CAAC,GAAG,CAAC,CAAC;QAC7B,EAAE,CAAC,SAAS,GAAG,CAAC,CAAC;QACjB,MAAM,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAC3B,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,OAAO;YACb,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,EAAE,CAAC;YACpB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YACrC,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,gBAAgB,EAAE,EAAE;SAC9C,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;QAC1B,MAAM,QAAQ,GAAG,eAAe,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAChD,MAAM,MAAM,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,EAAE,GAAG,QAAQ,KAAK,MAAM,CAAC;QAC/B,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,QAAQ;YACd,EAAE;YACF,QAAQ;YACR,MAAM;YACN,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,QAAQ,EAAE,MAAM,CAAC;SACtD,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,MAAM,CAAC,IAAI,CAAC,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACjD,CAAC;IAED,IAAI,MAAM,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACtC,MAAM,EAAE,GAAG,MAAM,IAAI,MAAM,CAAC,UAAU,CAAC;QACvC,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,YAAY;YAClB,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC;YACnC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC;YACtB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,MAAM,aAAa,MAAM,CAAC,UAAU,EAAE;SAC7E,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,UAAU,IAAI,IAAI,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,cAAc,CAAC,MAAM,CAAC,CAAC;QACtC,MAAM,EAAE,GAAG,MAAM,IAAI,MAAM,CAAC,UAAU,CAAC;QACvC,MAAM,CAAC,IAAI,CAAC;YACV,IAAI,EAAE,YAAY;YAClB,EAAE;YACF,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC;YACnC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC;YACtB,MAAM,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,MAAM,aAAa,MAAM,CAAC,UAAU,EAAE;SAC7E,CAAC,CAAC;IACL,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,MAAM,UAAU,YAAY,CAAC,MAAc,EAAE,QAAc;IACzD,OAAO,cAAc,CAAC,MAAM,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC;AACjD,CAAC"}
|
package/dist/run-cli.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { type Suite } from "./types.js";
|
|
2
|
-
export declare const HELP = "eval-smoke \u2014 one-minute smoke evals for prompts and models\n\nUsage:\n eval-smoke <suite.yaml|suite.json> [options]\n eval-smoke init [dir] [--force]\n\nOptions:\n --mock Offline runner with deterministic fake answers\n --json Machine-readable JSON report\n --junit <path> Write JUnit XML report to path (still prints human report unless --json)\n --only <pattern> Run cases whose id contains pattern, or matches /regex/\n --fail-fast Stop after the first failing case\n --concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)\n --timeout <ms> Per-case timeout (default: 30000)\n --model <name> Override suite model\n --provider <name> openai | anthropic | mock\n --baseline <path> Fail if any baseline-passing case now fails or is missing\n --update-baseline <path> Write current report JSON to path (pretty)\n --github-summary [path] Append Markdown to GITHUB_STEP_SUMMARY or path\n --force Overwrite existing suite.yaml when using init\n --no-color Disable ANSI colors (also respects NO_COLOR)\n -h, --help Show this help\n -v, --version Show version\n\nExit codes:\n 0 all cases passed (and baseline clean when --baseline is set)\n 1 one or more cases failed, or baseline regression\n 2 usage or configuration error\n\nExamples:\n eval-smoke init\n eval-smoke init ./prompts --force\n eval-smoke examples/suite.yaml --mock\n eval-smoke examples/suite.yaml --mock --only greeting\n eval-smoke examples/suite.yaml --mock --only '/^capital/'\n eval-smoke examples/suite.yaml --mock --junit junit.xml\n eval-smoke examples/suite.yaml --json --fail-fast\n eval-smoke examples/suite.yaml --mock --concurrency 4\n eval-smoke examples/suite.yaml --mock --update-baseline baseline.json\n eval-smoke examples/suite.yaml --mock --baseline baseline.json\n eval-smoke examples/suite.yaml --mock --github-summary\n OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml\n";
|
|
2
|
+
export declare const HELP = "eval-smoke \u2014 one-minute smoke evals for prompts and models\n\nUsage:\n eval-smoke <suite.yaml|suite.json> [options]\n eval-smoke init [dir] [--force]\n\nOptions:\n --mock Offline runner with deterministic fake answers\n --json Machine-readable JSON report\n --junit <path> Write JUnit XML report to path (still prints human report unless --json)\n --html <path> Write self-contained HTML report to path (offline, no CDN)\n --only <pattern> Run cases whose id contains pattern, or matches /regex/\n --fail-fast Stop after the first failing case\n --concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)\n --timeout <ms> Per-case timeout (default: 30000)\n --model <name> Override suite model\n --provider <name> openai | anthropic | mock\n --baseline <path> Fail if any baseline-passing case now fails or is missing\n --update-baseline <path> Write current report JSON to path (pretty)\n --github-summary [path] Append Markdown to GITHUB_STEP_SUMMARY or path\n --force Overwrite existing suite.yaml when using init\n --no-color Disable ANSI colors (also respects NO_COLOR)\n -h, --help Show this help\n -v, --version Show version\n\nExit codes:\n 0 all cases passed (and baseline clean when --baseline is set)\n 1 one or more cases failed, or baseline regression\n 2 usage or configuration error\n\nExamples:\n eval-smoke init\n eval-smoke init ./prompts --force\n eval-smoke examples/suite.yaml --mock\n eval-smoke examples/suite.yaml --mock --only greeting\n eval-smoke examples/suite.yaml --mock --only '/^capital/'\n eval-smoke examples/suite.yaml --mock --junit junit.xml\n eval-smoke examples/suite.yaml --mock --html report.html\n eval-smoke examples/suite.yaml --json --fail-fast\n eval-smoke examples/suite.yaml --mock --concurrency 4\n eval-smoke examples/suite.yaml --mock --update-baseline baseline.json\n eval-smoke examples/suite.yaml --mock --baseline baseline.json\n eval-smoke examples/suite.yaml --mock --github-summary\n OPENAI_API_KEY=sk-... eval-smoke examples/suite.yaml\n";
|
|
3
3
|
export interface CliIo {
|
|
4
4
|
stdout: {
|
|
5
5
|
write(text: string): void;
|
package/dist/run-cli.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run-cli.d.ts","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"run-cli.d.ts","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAaA,OAAO,EAAiC,KAAK,KAAK,EAAE,MAAM,YAAY,CAAC;AAMvE,eAAO,MAAM,IAAI,+nEA4ChB,CAAC;AAEF,MAAM,WAAW,KAAK;IACpB,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,MAAM,EAAE;QAAE,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;KAAE,CAAC;IACtC,GAAG,EAAE,MAAM,CAAC,UAAU,CAAC;IACvB,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,OAAO,CAAC;CAChB;AAkCD,iFAAiF;AACjF,wBAAgB,SAAS,CAAC,EAAE,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAQ9D;AAED,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,GAAG,KAAK,CAMtE;AAED;;;;GAIG;AACH,wBAAgB,0BAA0B,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,MAAM,EAAE,CAiBnE;AAQD,wBAAsB,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,GAAE,KAAmB,GAAG,OAAO,CAAC,MAAM,CAAC,CAsIrF"}
|
package/dist/run-cli.js
CHANGED
|
@@ -7,6 +7,7 @@ import { initSuite } from "./init.js";
|
|
|
7
7
|
import { formatJUnit } from "./junit.js";
|
|
8
8
|
import { loadSuite } from "./load-suite.js";
|
|
9
9
|
import { compileRegex } from "./matchers.js";
|
|
10
|
+
import { formatHtmlReport } from "./html-report.js";
|
|
10
11
|
import { formatReport } from "./report.js";
|
|
11
12
|
import { DEFAULT_TIMEOUT_MS, runSuite } from "./run.js";
|
|
12
13
|
import { formatGithubSummary } from "./summary-md.js";
|
|
@@ -24,6 +25,7 @@ Options:
|
|
|
24
25
|
--mock Offline runner with deterministic fake answers
|
|
25
26
|
--json Machine-readable JSON report
|
|
26
27
|
--junit <path> Write JUnit XML report to path (still prints human report unless --json)
|
|
28
|
+
--html <path> Write self-contained HTML report to path (offline, no CDN)
|
|
27
29
|
--only <pattern> Run cases whose id contains pattern, or matches /regex/
|
|
28
30
|
--fail-fast Stop after the first failing case
|
|
29
31
|
--concurrency <n> Run up to N cases in parallel (default: 1; ignored with --fail-fast)
|
|
@@ -50,6 +52,7 @@ Examples:
|
|
|
50
52
|
eval-smoke examples/suite.yaml --mock --only greeting
|
|
51
53
|
eval-smoke examples/suite.yaml --mock --only '/^capital/'
|
|
52
54
|
eval-smoke examples/suite.yaml --mock --junit junit.xml
|
|
55
|
+
eval-smoke examples/suite.yaml --mock --html report.html
|
|
53
56
|
eval-smoke examples/suite.yaml --json --fail-fast
|
|
54
57
|
eval-smoke examples/suite.yaml --mock --concurrency 4
|
|
55
58
|
eval-smoke examples/suite.yaml --mock --update-baseline baseline.json
|
|
@@ -148,6 +151,7 @@ export async function runCli(argv, io = defaultIo()) {
|
|
|
148
151
|
mock: { type: "boolean", default: false },
|
|
149
152
|
json: { type: "boolean", default: false },
|
|
150
153
|
junit: { type: "string" },
|
|
154
|
+
html: { type: "string" },
|
|
151
155
|
only: { type: "string" },
|
|
152
156
|
"fail-fast": { type: "boolean", default: false },
|
|
153
157
|
concurrency: { type: "string" },
|
|
@@ -203,6 +207,7 @@ export async function runCli(argv, io = defaultIo()) {
|
|
|
203
207
|
const json = asBool(values.json);
|
|
204
208
|
const only = asString(values.only);
|
|
205
209
|
const junitPath = asString(values.junit);
|
|
210
|
+
const htmlPath = asString(values.html);
|
|
206
211
|
const baselinePath = asString(values.baseline);
|
|
207
212
|
const updateBaselinePath = asString(values["update-baseline"]);
|
|
208
213
|
const githubSummary = githubSummaryFlag(asString(values["github-summary"]));
|
|
@@ -223,6 +228,10 @@ export async function runCli(argv, io = defaultIo()) {
|
|
|
223
228
|
const out = resolve(io.cwd, junitPath);
|
|
224
229
|
writeFileSync(out, formatJUnit(report), "utf8");
|
|
225
230
|
}
|
|
231
|
+
if (htmlPath != null) {
|
|
232
|
+
const out = resolve(io.cwd, htmlPath);
|
|
233
|
+
writeFileSync(out, formatHtmlReport(report), "utf8");
|
|
234
|
+
}
|
|
226
235
|
if (updateBaselinePath != null) {
|
|
227
236
|
const out = resolve(io.cwd, updateBaselinePath);
|
|
228
237
|
writeBaseline(out, report);
|
package/dist/run-cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run-cli.js","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACxC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpF,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC5C,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,kBAAkB,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,UAAU,EAAiC,MAAM,YAAY,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,kEAAkE;AAClE,MAAM,mBAAmB,GAAG,UAAU,CAAC;AAEvC,MAAM,CAAC,MAAM,IAAI,GAAG
|
|
1
|
+
{"version":3,"file":"run-cli.js","sourceRoot":"","sources":["../src/run-cli.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACxC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAC3E,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpF,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC5C,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,kBAAkB,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,UAAU,EAAiC,MAAM,YAAY,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,kEAAkE;AAClE,MAAM,mBAAmB,GAAG,UAAU,CAAC;AAEvC,MAAM,CAAC,MAAM,IAAI,GAAG;;;;;;;;;;;;;;mDAc+B,kBAAkB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CA8BpE,CAAC;AAUF,MAAM,SAAS,GAAG,GAAU,EAAE,CAAC,CAAC;IAC9B,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,MAAM,EAAE,OAAO,CAAC,MAAM;IACtB,GAAG,EAAE,OAAO,CAAC,GAAG;IAChB,GAAG,EAAE,OAAO,CAAC,GAAG,EAAE;IAClB,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC;CACrC,CAAC,CAAC;AAEH,SAAS,YAAY,CAAC,GAAuB;IAC3C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,kBAAkB,CAAC;IAC3C,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,qDAAqD,CAAC,CAAC;IAC9E,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,gBAAgB,CAAC,GAAuB;IAC/C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC3C,MAAM,IAAI,UAAU,CAAC,0CAA0C,CAAC,CAAC;IACnE,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,aAAa,CAAC,GAAuB;IAC5C,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,GAAG,KAAK,MAAM,IAAI,GAAG,KAAK,QAAQ,IAAI,GAAG,KAAK,WAAW;QAAE,OAAO,GAAG,CAAC;IAC1E,MAAM,IAAI,UAAU,CAAC,+CAA+C,CAAC,CAAC;AACxE,CAAC;AAED,iFAAiF;AACjF,MAAM,UAAU,SAAS,CAAC,EAAU,EAAE,OAAe;IACnD,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;IAC5D,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;QACzB,MAAM,EAAE,GAAG,YAAY,CAAC,OAAO,CAAC,CAAC;QACjC,EAAE,CAAC,SAAS,GAAG,CAAC,CAAC;QACjB,OAAO,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,EAAE,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;AAC9B,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KAAY,EAAE,OAAe;IAC7D,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,SAAS,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;IAChF,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACvB,MAAM,IAAI,UAAU,CAAC,UAAU,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,kBAAkB,CAAC,CAAC;IAC5E,CAAC;IACD,OAAO,EAAE,GAAG,KAAK,EAAE,KAAK,EAAE,CAAC;AAC7B,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,0BAA0B,CAAC,IAAc;IACvD,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;QACxC,MAAM,KAAK,GAAG,IAAI,CAAC,CAAC,CAAE,CAAC;QACvB,IAAI,KAAK,KAAK,kBAAkB,EAAE,CAAC;YACjC,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;gBAClC,GAAG,CAAC,IAAI,CAAC,oBAAoB,mBAAmB,EAAE,CAAC,CAAC;YACtD,CAAC;iBAAM,CAAC;gBACN,GAAG,CAAC,IAAI,CAAC,kBAAkB,EAAE,IAAI,CAAC,CAAC;gBACnC,CAAC,IAAI,CAAC,CAAC;YACT,CAAC;YACD,SAAS;QACX,CAAC;QACD,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,iBAAiB,CAAC,GAAuB;IAChD,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,GAAG,KAAK,mBAAmB;QAAE,OAAO,IAAI,CAAC;IAC7C,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,MAAM,CAAC,IAAc,EAAE,KAAY,SAAS,EAAE;IAClE,IAAI,MAAoC,CAAC;IACzC,IAAI,CAAC;QACH,MAAM,GAAG,SAAS,CAAC;YACjB,IAAI,EAAE,0BAA0B,CAAC,IAAI,CAAC;YACtC,gBAAgB,EAAE,IAAI;YACtB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE;gBACP,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBACzC,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACxB,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACxB,WAAW,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAChD,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC/B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC3B,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACzB,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC5B,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBAC5B,iBAAiB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACrC,gBAAgB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;gBACpC,KAAK,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC1C,UAAU,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE;gBAC/C,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;gBACrD,OAAO,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,KAAK,EAAE;aACzD;SACF,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QAC/E,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,MAAM,CAAC;IACvC,MAAM,QAAQ,GAAG,CAAC,KAAc,EAAsB,EAAE,CACtD,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;IAChD,MAAM,MAAM,GAAG,CAAC,KAAc,EAAW,EAAE,CAAC,KAAK,KAAK,IAAI,CAAC;IAE3D,IAAI,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;QACxB,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;QAC3B,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,WAAW,CAAC,CAAC,CAAC,KAAK,MAAM,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;YAC9B,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,EAAE,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC;YAC/E,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,MAAM,CAAC,IAAI,IAAI,CAAC,CAAC;YAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,oBAAoB,MAAM,CAAC,IAAI,WAAW,CAAC,CAAC;YAC5D,OAAO,CAAC,CAAC;QACX,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;YAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;IAED,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IACjC,IAAI,CAAC,SAAS,EAAE,CAAC;QACf,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,yBAAyB,CAAC,CAAC;QAC3C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtB,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACjC,MAAM,IAAI,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,SAAS,GAAG,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACzC,MAAM,QAAQ,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QACvC,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;QAC/C,MAAM,kBAAkB,GAAG,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,CAAC,CAAC;QAC/D,MAAM,aAAa,GAAG,iBAAiB,CAAC,QAAQ,CAAC,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC;QAE5E,IAAI,KAAK,GAAG,SAAS,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC;QACzC,IAAI,IAAI,IAAI,IAAI,EAAE,CAAC;YACjB,KAAK,GAAG,iBAAiB,CAAC,KAAK,EAAE,IAAI,CAAC,CAAC;QACzC,CAAC;QACD,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,KAAK,EAAE;YACnC,IAAI,EAAE,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC;YACzB,QAAQ,EAAE,aAAa,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;YAClD,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC;YAC7B,QAAQ,EAAE,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YACrC,WAAW,EAAE,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;YAC3D,SAAS,EAAE,YAAY,CAAC,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;YACjD,GAAG,EAAE,EAAE,CAAC,GAAG;SACZ,CAAC,CAAC;QACH,IAAI,SAAS,IAAI,IAAI,EAAE,CAAC;YACtB,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;YACvC,aAAa,CAAC,GAAG,EAAE,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAClD,CAAC;QACD,IAAI,QAAQ,IAAI,IAAI,EAAE,CAAC;YACrB,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;YACtC,aAAa,CAAC,GAAG,EAAE,gBAAgB,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QACvD,CAAC;QACD,IAAI,kBAAkB,IAAI,IAAI,EAAE,CAAC;YAC/B,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,kBAAkB,CAAC,CAAC;YAChD,aAAa,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YAC3B,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,GAAG,IAAI,CAAC,CAAC;QAC7C,CAAC;QAED,IAAI,cAAc,GAAG,KAAK,CAAC;QAC3B,IAAI,YAAY,IAAI,IAAI,EAAE,CAAC;YACzB,MAAM,QAAQ,GAAG,YAAY,CAAC,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,YAAY,CAAC,CAAC,CAAC;YAC7D,MAAM,KAAK,GAAG,aAAa,CAAC,MAAM,EAAE,QAAQ,CAAC,CAAC;YAC9C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YAC9B,IAAI,CAAC,KAAK,CAAC,EAAE;gBAAE,cAAc,GAAG,IAAI,CAAC;QACvC,CAAC;QAED,MAAM,UAAU,GACd,aAAa,KAAK,SAAS;YACzB,CAAC,CAAC,wBAAwB,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC;YACrD,CAAC,CAAC,wBAAwB,CAAC,aAAa,EAAE,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC;QAC9D,IAAI,UAAU,EAAE,CAAC;YACf,IAAI,CAAC;gBACH,mBAAmB,CAAC,UAAU,EAAE,mBAAmB,CAAC,MAAM,CAAC,CAAC,CAAC;YAC/D,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,mCAAmC,OAAO,IAAI,CAAC,CAAC;YAClE,CAAC;QACH,CAAC;QAED,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,IAAI,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACvE,MAAM,KAAK,GAAG,EAAE,CAAC,KAAK,IAAI,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC;QAC5C,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QACvD,IAAI,cAAc;YAAE,OAAO,CAAC,CAAC;QAC7B,OAAO,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvE,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;QAChC,OAAO,KAAK,YAAY,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;IAC1D,CAAC;AACH,CAAC"}
|
package/dist/types.d.ts
CHANGED
|
@@ -5,6 +5,14 @@ export type JsonExpect = true | Record<string, unknown> | unknown[];
|
|
|
5
5
|
export interface Expect {
|
|
6
6
|
contains?: StringList;
|
|
7
7
|
not_contains?: StringList;
|
|
8
|
+
/** Pass if ANY needle is found (case-insensitive). */
|
|
9
|
+
contains_any?: StringList;
|
|
10
|
+
/** Pass if ALL needles are found (case-insensitive). */
|
|
11
|
+
contains_all?: StringList;
|
|
12
|
+
/** Pass if output starts with any needle (case-insensitive). */
|
|
13
|
+
starts_with?: StringList;
|
|
14
|
+
/** Pass if output ends with any needle (case-insensitive). */
|
|
15
|
+
ends_with?: StringList;
|
|
8
16
|
regex?: StringList;
|
|
9
17
|
/** Exact string match (Unicode NFC, LF newlines). Trailing newline stripped on both sides. */
|
|
10
18
|
equals?: string;
|
|
@@ -35,7 +43,7 @@ export interface CompleteRequest {
|
|
|
35
43
|
signal: AbortSignal;
|
|
36
44
|
}
|
|
37
45
|
export type CompleteFn = (request: CompleteRequest) => Promise<string>;
|
|
38
|
-
export type CheckName = "contains" | "not_contains" | "regex" | "equals" | "json" | "max_tokens" | "min_tokens";
|
|
46
|
+
export type CheckName = "contains" | "not_contains" | "contains_any" | "contains_all" | "starts_with" | "ends_with" | "regex" | "equals" | "json" | "max_tokens" | "min_tokens";
|
|
39
47
|
export interface CheckResult {
|
|
40
48
|
name: CheckName;
|
|
41
49
|
ok: boolean;
|
package/dist/types.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,MAAM,MAAM,YAAY,GAAG,MAAM,GAAG,QAAQ,GAAG,WAAW,CAAC;AAE3D,MAAM,MAAM,UAAU,GAAG,MAAM,GAAG,MAAM,EAAE,CAAC;AAE3C,oFAAoF;AACpF,MAAM,MAAM,UAAU,GAAG,IAAI,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,OAAO,EAAE,CAAC;AAEpE,MAAM,WAAW,MAAM;IACrB,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,YAAY,CAAC,EAAE,UAAU,CAAC;IAC1B,KAAK,CAAC,EAAE,UAAU,CAAC;IACnB,8FAA8F;IAC9F,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,oEAAoE;IACpE,IAAI,CAAC,EAAE,UAAU,CAAC;IAClB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,IAAI;IACnB,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,KAAK;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,IAAI,EAAE,CAAC;CACf;AAED,MAAM,WAAW,eAAe;IAC9B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,WAAW,CAAC;CACrB;AAED,MAAM,MAAM,UAAU,GAAG,CAAC,OAAO,EAAE,eAAe,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC;AAEvE,MAAM,MAAM,SAAS,GACjB,UAAU,GACV,cAAc,GACd,OAAO,GACP,QAAQ,GACR,MAAM,GACN,YAAY,GACZ,YAAY,CAAC;AAEjB,MAAM,WAAW,WAAW;IAC1B,IAAI,EAAE,SAAS,CAAC;IAChB,EAAE,EAAE,OAAO,CAAC;IACZ,QAAQ,EAAE,MAAM,CAAC;IACjB,gDAAgD;IAChD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,UAAU;IACzB,EAAE,EAAE,MAAM,CAAC;IACX,EAAE,EAAE,OAAO,CAAC;IACZ,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,WAAW,EAAE,CAAC;CACxB;AAED,MAAM,WAAW,MAAM;IACrB,EAAE,EAAE,OAAO,CAAC;IACZ,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,YAAY,CAAC;IACrB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,UAAU,EAAE,CAAC;CACrB;AAED,MAAM,WAAW,UAAU;IACzB,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,2EAA2E;IAC3E,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,GAAG,CAAC,EAAE,MAAM,CAAC,UAAU,CAAC;IACxB,KAAK,CAAC,EAAE,OAAO,KAAK,CAAC;CACtB;AAED,qBAAa,UAAW,SAAQ,KAAK;IACnC,QAAQ,CAAC,QAAQ,KAAK;gBACV,OAAO,EAAE,MAAM;CAI5B"}
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,MAAM,MAAM,YAAY,GAAG,MAAM,GAAG,QAAQ,GAAG,WAAW,CAAC;AAE3D,MAAM,MAAM,UAAU,GAAG,MAAM,GAAG,MAAM,EAAE,CAAC;AAE3C,oFAAoF;AACpF,MAAM,MAAM,UAAU,GAAG,IAAI,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,OAAO,EAAE,CAAC;AAEpE,MAAM,WAAW,MAAM;IACrB,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,YAAY,CAAC,EAAE,UAAU,CAAC;IAC1B,sDAAsD;IACtD,YAAY,CAAC,EAAE,UAAU,CAAC;IAC1B,wDAAwD;IACxD,YAAY,CAAC,EAAE,UAAU,CAAC;IAC1B,gEAAgE;IAChE,WAAW,CAAC,EAAE,UAAU,CAAC;IACzB,8DAA8D;IAC9D,SAAS,CAAC,EAAE,UAAU,CAAC;IACvB,KAAK,CAAC,EAAE,UAAU,CAAC;IACnB,8FAA8F;IAC9F,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,oEAAoE;IACpE,IAAI,CAAC,EAAE,UAAU,CAAC;IAClB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,IAAI;IACnB,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,KAAK;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,IAAI,EAAE,CAAC;CACf;AAED,MAAM,WAAW,eAAe;IAC9B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,WAAW,CAAC;CACrB;AAED,MAAM,MAAM,UAAU,GAAG,CAAC,OAAO,EAAE,eAAe,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC;AAEvE,MAAM,MAAM,SAAS,GACjB,UAAU,GACV,cAAc,GACd,cAAc,GACd,cAAc,GACd,aAAa,GACb,WAAW,GACX,OAAO,GACP,QAAQ,GACR,MAAM,GACN,YAAY,GACZ,YAAY,CAAC;AAEjB,MAAM,WAAW,WAAW;IAC1B,IAAI,EAAE,SAAS,CAAC;IAChB,EAAE,EAAE,OAAO,CAAC;IACZ,QAAQ,EAAE,MAAM,CAAC;IACjB,gDAAgD;IAChD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,UAAU;IACzB,EAAE,EAAE,MAAM,CAAC;IACX,EAAE,EAAE,OAAO,CAAC;IACZ,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,WAAW,EAAE,CAAC;CACxB;AAED,MAAM,WAAW,MAAM;IACrB,EAAE,EAAE,OAAO,CAAC;IACZ,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,YAAY,CAAC;IACrB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,UAAU,EAAE,CAAC;CACrB;AAED,MAAM,WAAW,UAAU;IACzB,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,2EAA2E;IAC3E,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,GAAG,CAAC,EAAE,MAAM,CAAC,UAAU,CAAC;IACxB,KAAK,CAAC,EAAE,OAAO,KAAK,CAAC;CACtB;AAED,qBAAa,UAAW,SAAQ,KAAK;IACnC,QAAQ,CAAC,QAAQ,KAAK;gBACV,OAAO,EAAE,MAAM;CAI5B"}
|
package/dist/types.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.js","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"types.js","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AA+GA,MAAM,OAAO,UAAW,SAAQ,KAAK;IAC1B,QAAQ,GAAG,CAAC,CAAC;IACtB,YAAY,OAAe;QACzB,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,YAAY,CAAC;IAC3B,CAAC;CACF"}
|
package/examples/suite.yaml
CHANGED
|
@@ -34,3 +34,19 @@ cases:
|
|
|
34
34
|
prompt: Please say hello to the user.
|
|
35
35
|
expect:
|
|
36
36
|
equals: Hello
|
|
37
|
+
|
|
38
|
+
- id: soft-any-city
|
|
39
|
+
prompt: What is the capital of France? Answer with the city name only.
|
|
40
|
+
expect:
|
|
41
|
+
contains_any:
|
|
42
|
+
- Paris
|
|
43
|
+
- Lyon
|
|
44
|
+
starts_with: Par
|
|
45
|
+
|
|
46
|
+
- id: soft-all-hello
|
|
47
|
+
prompt: Reply with exactly the word Hello.
|
|
48
|
+
expect:
|
|
49
|
+
contains_all:
|
|
50
|
+
- Hel
|
|
51
|
+
- lo
|
|
52
|
+
ends_with: lo
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@webywz/eval-smoke",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "One-minute smoke evals for prompts and models
|
|
3
|
+
"version": "0.6.0",
|
|
4
|
+
"description": "One-minute smoke evals for prompts and models — equals/json matchers, diffs, concurrency, JUnit, init.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"eval-smoke": "dist/cli.js"
|