@mappies/miniwat-eval 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +91 -10
- package/dist/index.cjs +577 -120
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +33 -6
- package/dist/index.d.ts +34 -6
- package/dist/index.js +577 -120
- package/dist/index.js.map +1 -1
- package/dist/main.js +588 -125
- package/dist/main.js.map +1 -1
- package/package.json +1 -1
- package/schemas/conversation.schema.json +35 -1
- package/schemas/eval-config.schema.json +6 -1
- package/schemas/report.schema.json +1 -0
package/README.md
CHANGED
|
@@ -127,10 +127,11 @@ Unknown keys are errors. The `evaluator` section is required, but its provider p
|
|
|
127
127
|
|
|
128
128
|
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
|
|
129
129
|
|
|
130
|
-
1. the
|
|
131
|
-
2. the
|
|
132
|
-
3. the
|
|
133
|
-
4. the
|
|
130
|
+
1. the run's variables, for a scenario with `runWith:` (see [Running a scenario several times](#running-a-scenario-several-times));
|
|
131
|
+
2. the scenario's `variables:` (see [Variables](#variables));
|
|
132
|
+
3. the configuration's `variables:`;
|
|
133
|
+
4. the `--parameters` file, when one is given;
|
|
134
|
+
5. the environment variable of that name.
|
|
134
135
|
|
|
135
136
|
A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
|
|
136
137
|
|
|
@@ -156,7 +157,7 @@ Keeping one file per environment, such as `parameters.json` and `parameters_cicd
|
|
|
156
157
|
|
|
157
158
|
### Variables
|
|
158
159
|
|
|
159
|
-
`variables:` is a map of names to strings, at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
|
|
160
|
+
`variables:` is a map of names to strings, or to lists of strings (see [Running a scenario several times](#running-a-scenario-several-times)), at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
|
|
160
161
|
|
|
161
162
|
```yaml
|
|
162
163
|
# eval-config.yml
|
|
@@ -244,6 +245,48 @@ An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations
|
|
|
244
245
|
|
|
245
246
|
The JSON Schemas in `schemas/` describe the configuration, scenarios and the JSON report, and can be used for editor completion.
|
|
246
247
|
|
|
248
|
+
### Running a scenario several times
|
|
249
|
+
|
|
250
|
+
`runWith:` runs one scenario once per entry, each with its own variables. Each run is a scenario of its own in every count, in the exit code and in the reports, named `Greeting [student]`. It has its own connection, so a run can change the token.
|
|
251
|
+
|
|
252
|
+
Rows set several variables per run. `name` is required and labels the run; every other key is a variable:
|
|
253
|
+
|
|
254
|
+
```yaml
|
|
255
|
+
version: 1
|
|
256
|
+
name: Greeting
|
|
257
|
+
runWith:
|
|
258
|
+
- name: instructor
|
|
259
|
+
token: ${instructor-token}
|
|
260
|
+
org: Acme University
|
|
261
|
+
- name: visitor # sets no token, so it connects without one
|
|
262
|
+
org: Acme University
|
|
263
|
+
turns:
|
|
264
|
+
- input: Hello, I am from ${org}.
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`{ variable, from }` sets one variable from a list of variable or parameter names. Each run is named after its entry, never after the value, so a token never appears in a run's name:
|
|
268
|
+
|
|
269
|
+
```yaml
|
|
270
|
+
runWith:
|
|
271
|
+
variable: token
|
|
272
|
+
from: [instructor-token, student-token] # or a list variable: ${authenticated-tokens}
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
A list defined once in the configuration can be shared by many scenarios:
|
|
276
|
+
|
|
277
|
+
```yaml
|
|
278
|
+
# eval-config.yml
|
|
279
|
+
variables:
|
|
280
|
+
authenticated-tokens: [instructor-token, student-token, user-token]
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
- **Values:** a run's variables come first in [references](#references). A row value may use the scenario's variables, but a scenario variable never sees a run's value.
|
|
284
|
+
- **Defaults:** a scenario variable is the default for runs that do not set it, so one that every run overrides is allowed.
|
|
285
|
+
- **Lists:** a list variable can only be used as the whole value of `from`. Its items are names, and are not resolved themselves. Lists come only from `variables:` sections, because parameters and environment variables are strings.
|
|
286
|
+
- **Checks:** run names are unique within a scenario and match `[A-Za-z0-9][A-Za-z0-9_.-]*`. A run variable that neither the scenario nor the `websocket` section references is an error, as for scenario variables.
|
|
287
|
+
- **Errors:** a problem only some runs have names the run, for example `scenarios/order.yml:9:14 turns[0].input: run 'lost': ...`. One that every run has is reported once.
|
|
288
|
+
- **Reports:** the JSON report keeps each scenario's `name` and gives the run's name as `runWith`.
|
|
289
|
+
|
|
247
290
|
## Results
|
|
248
291
|
|
|
249
292
|
| Level | Statuses |
|
|
@@ -256,13 +299,51 @@ A turn stops its attempt when the host reports an error (`statusCode >= 400` or
|
|
|
256
299
|
|
|
257
300
|
### Reports
|
|
258
301
|
|
|
259
|
-
The console prints one line per scenario as it finishes, in file order
|
|
302
|
+
The console prints one line per scenario as it finishes, in file order: a status symbol, the name, the attempts that passed out of `repeat`, and the time it took.
|
|
303
|
+
|
|
304
|
+
```text
|
|
305
|
+
✔ Greeting [user-token] 1/1 2.6s
|
|
306
|
+
✖ Password reset 0/1 7.9s
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
| Symbol | Status |
|
|
310
|
+
|---|---|
|
|
311
|
+
| `✔` | `PASS` |
|
|
312
|
+
| `✖` | `FAIL` |
|
|
313
|
+
| `⚠` | `FLAKY` |
|
|
314
|
+
| `●` | `ERROR` |
|
|
315
|
+
| `○` | `SKIPPED` (a test that did not run) |
|
|
316
|
+
| `◌` | `INTERRUPTED`, or a scenario that never started |
|
|
317
|
+
|
|
318
|
+
A `FAILURES` section follows, with one block per test that did not pass and per attempt that stopped:
|
|
319
|
+
|
|
320
|
+
```text
|
|
321
|
+
test at scenarios/authenticated/knowledge-base/password-reset.yml:11:9
|
|
322
|
+
✖ Password reset › turn 1: response
|
|
323
|
+
Score 0.00, threshold 0.85
|
|
324
|
+
|
|
325
|
+
Expected
|
|
326
|
+
Explains how to reset the password using the steps from the knowledge
|
|
327
|
+
base article it found, without adding steps the article does not support.
|
|
328
|
+
|
|
329
|
+
Explanation
|
|
330
|
+
The response does not explain any password-reset steps from a
|
|
331
|
+
knowledge-base article.
|
|
332
|
+
|
|
333
|
+
Evidence
|
|
334
|
+
• Actual response: "I'm afraid the available documentation does not..."
|
|
335
|
+
|
|
336
|
+
Attempts 1: 0 passed, 1 failed, 0 errors (1 required)
|
|
337
|
+
```
|
|
260
338
|
|
|
261
|
-
-
|
|
262
|
-
-
|
|
263
|
-
-
|
|
339
|
+
- **Semantic checks** show the score and threshold, the `should` text, the evaluator's explanation and its evidence.
|
|
340
|
+
- **Other checks** list their details, such as the closest tool use and how its arguments differ.
|
|
341
|
+
- **Repeated runs:** with `repeat` above 1, each failing attempt has its own entry.
|
|
342
|
+
- **Flaky tests** are listed in a `FLAKY` section after it.
|
|
343
|
+
- **Summary:** the report ends with a `SUMMARY` of scenario and test counts, agent and evaluator token usage with coverage (such as `coverage 8/9, partial`, when not every call reported usage), the duration and the exit code.
|
|
344
|
+
- **Wrapping:** text wraps at the terminal's width, or at 100 columns when output is redirected.
|
|
264
345
|
|
|
265
|
-
`--verbose` adds every attempt's transcript.
|
|
346
|
+
`--verbose` adds every attempt's transcript under its scenario line. Symbols are coloured on a terminal unless `NO_COLOR` is set; the symbol is shown either way. The console format changed in 1.4.0; scripts should read the JSON report, not the console.
|
|
266
347
|
|
|
267
348
|
`json` writes `miniwat-eval-report.json` to `report.directory`, replacing the previous one, and prints `Report: <path>` above the final line. It follows `schemas/report.schema.json` and holds:
|
|
268
349
|
|