@mappies/miniwat-eval 1.2.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -127,10 +127,11 @@ Unknown keys are errors. The `evaluator` section is required, but its provider p
127
127
 
128
128
  `${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
129
129
 
130
- 1. the scenario's `variables:` (see [Variables](#variables));
131
- 2. the configuration's `variables:`;
132
- 3. the `--parameters` file, when one is given;
133
- 4. the environment variable of that name.
130
+ 1. the run's variables, for a scenario with `runWith:` (see [Running a scenario several times](#running-a-scenario-several-times));
131
+ 2. the scenario's `variables:` (see [Variables](#variables));
132
+ 3. the configuration's `variables:`;
133
+ 4. the `--parameters` file, when one is given;
134
+ 5. the environment variable of that name.
134
135
 
135
136
  A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
136
137
 
@@ -156,7 +157,7 @@ Keeping one file per environment, such as `parameters.json` and `parameters_cicd
156
157
 
157
158
  ### Variables
158
159
 
159
- `variables:` is a map of names to strings, at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
160
+ `variables:` is a map of names to strings, or to lists of strings (see [Running a scenario several times](#running-a-scenario-several-times)), at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
160
161
 
161
162
  ```yaml
162
163
  # eval-config.yml
@@ -244,6 +245,48 @@ An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations
244
245
 
245
246
  The JSON Schemas in `schemas/` describe the configuration, scenarios and the JSON report, and can be used for editor completion.
246
247
 
248
+ ### Running a scenario several times
249
+
250
+ `runWith:` runs one scenario once per entry, each with its own variables. Each run is a scenario of its own in every count, in the exit code and in the reports, named `Greeting [student]`. It has its own connection, so a run can change the token.
251
+
252
+ Rows set several variables per run. `name` is required and labels the run; every other key is a variable:
253
+
254
+ ```yaml
255
+ version: 1
256
+ name: Greeting
257
+ runWith:
258
+ - name: instructor
259
+ token: ${instructor-token}
260
+ org: Acme University
261
+ - name: visitor # sets no token, so it connects without one
262
+ org: Acme University
263
+ turns:
264
+ - input: Hello, I am from ${org}.
265
+ ```
266
+
267
+ `{ variable, from }` sets one variable from a list of variable or parameter names. Each run is named after its entry, never after the value, so a token never appears in a run's name:
268
+
269
+ ```yaml
270
+ runWith:
271
+ variable: token
272
+ from: [instructor-token, student-token] # or a list variable: ${authenticated-tokens}
273
+ ```
274
+
275
+ A list defined once in the configuration can be shared by many scenarios:
276
+
277
+ ```yaml
278
+ # eval-config.yml
279
+ variables:
280
+ authenticated-tokens: [instructor-token, student-token, user-token]
281
+ ```
282
+
283
+ - **Values:** a run's variables come first in [references](#references). A row value may use the scenario's variables, but a scenario variable never sees a run's value.
284
+ - **Defaults:** a scenario variable is the default for runs that do not set it, so one that every run overrides is allowed.
285
+ - **Lists:** a list variable can only be used as the whole value of `from`. Its items are names, and are not resolved themselves. Lists come only from `variables:` sections, because parameters and environment variables are strings.
286
+ - **Checks:** run names are unique within a scenario and match `[A-Za-z0-9][A-Za-z0-9_.-]*`. A run variable that neither the scenario nor the `websocket` section references is an error, as for scenario variables.
287
+ - **Errors:** a problem only some runs have names the run, for example `scenarios/order.yml:9:14 turns[0].input: run 'lost': ...`. One that every run has is reported once.
288
+ - **Reports:** the JSON report keeps each scenario's `name` and gives the run's name as `runWith`.
289
+
247
290
  ## Results
248
291
 
249
292
  | Level | Statuses |
@@ -256,13 +299,51 @@ A turn stops its attempt when the host reports an error (`statusCode >= 400` or
256
299
 
257
300
  ### Reports
258
301
 
259
- The console prints one line per scenario as it finishes, in file order. A Problems section follows for each scenario that did not pass. It lists attempts that stopped and why, then each test that did not pass with its location, its counts across attempts and the details of each failing attempt, including evaluator scores and explanations. The report ends with:
302
+ The console prints one line per scenario as it finishes, in file order: a status symbol, the name, the attempts that passed out of `repeat`, and the time it took.
303
+
304
+ ```text
305
+ ✔ Greeting [user-token] 1/1 2.6s
306
+ ✖ Password reset 0/1 7.9s
307
+ ```
308
+
309
+ | Symbol | Status |
310
+ |---|---|
311
+ | `✔` | `PASS` |
312
+ | `✖` | `FAIL` |
313
+ | `⚠` | `FLAKY` |
314
+ | `●` | `ERROR` |
315
+ | `○` | `SKIPPED` (a test that did not run) |
316
+ | `◌` | `INTERRUPTED`, or a scenario that never started |
317
+
318
+ A `FAILURES` section follows, with one block per test that did not pass and per attempt that stopped:
319
+
320
+ ```text
321
+ test at scenarios/authenticated/knowledge-base/password-reset.yml:11:9
322
+ ✖ Password reset › turn 1: response
323
+ Score 0.00, threshold 0.85
324
+
325
+ Expected
326
+ Explains how to reset the password using the steps from the knowledge
327
+ base article it found, without adding steps the article does not support.
328
+
329
+ Explanation
330
+ The response does not explain any password-reset steps from a
331
+ knowledge-base article.
332
+
333
+ Evidence
334
+ • Actual response: "I'm afraid the available documentation does not..."
335
+
336
+ Attempts 1: 0 passed, 1 failed, 0 errors (1 required)
337
+ ```
260
338
 
261
- - scenario and test counts;
262
- - agent and evaluator token usage with coverage, such as `coverage 8/9, partial`, when not every call reported usage;
263
- - the duration and the exit code.
339
+ - **Semantic checks** show the score and threshold, the `should` text, the evaluator's explanation and its evidence.
340
+ - **Other checks** list their details, such as the closest tool use and how its arguments differ.
341
+ - **Repeated runs:** with `repeat` above 1, each failing attempt has its own entry.
342
+ - **Flaky tests** are listed in a `FLAKY` section after it.
343
+ - **Summary:** the report ends with a `SUMMARY` of scenario and test counts, agent and evaluator token usage with coverage (such as `coverage 8/9, partial`, when not every call reported usage), the duration and the exit code.
344
+ - **Wrapping:** text wraps at the terminal's width, or at 100 columns when output is redirected.
264
345
 
265
- `--verbose` adds every attempt's transcript. Status words are coloured on a terminal unless `NO_COLOR` is set.
346
+ `--verbose` adds every attempt's transcript under its scenario line. Symbols are coloured on a terminal unless `NO_COLOR` is set; the symbol is shown either way. The console format changed in 1.4.0; scripts should read the JSON report, not the console.
266
347
 
267
348
  `json` writes `miniwat-eval-report.json` to `report.directory`, replacing the previous one, and prints `Report: <path>` above the final line. It follows `schemas/report.schema.json` and holds:
268
349