@mappies/miniwat-eval 1.0.0 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -12,7 +12,7 @@ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
12
12
  npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
13
13
  ```
14
14
 
15
- The provider is only needed when a scenario has semantic criteria. Node.js 22 or later is required.
15
+ The provider is only needed when a scenario has a `should` check. Node.js 22 or later is required.
16
16
 
17
17
  ## Quick start
18
18
 
@@ -30,8 +30,8 @@ npx miniwat-eval run examples/scenarios --quick
30
30
  ## Commands
31
31
 
32
32
  ```bash
33
- miniwat-eval validate <target> [--config <path>]
34
- miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
33
+ miniwat-eval validate <target> [--config <path>] [--parameters <path>]
34
+ miniwat-eval run <target> [--config <path>] [--parameters <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
35
35
  ```
36
36
 
37
37
  `<target>` is a scenario file, or a directory searched recursively for `*.yml` and `*.yaml`. The configuration file and the report directory are skipped.
@@ -39,6 +39,7 @@ miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--q
39
39
  | Option | Effect |
40
40
  |---|---|
41
41
  | `--config <path>` | The configuration file. Without it, `eval-config.yml` is searched for upward from the target, stopping at the nearest directory with `.git` or `package.json`, then in the working directory. |
42
+ | `--parameters <path>` | A JSON file of values for `${name}` references, looked up before environment variables. See [References](#references). |
42
43
  | `--repeat <n>` | Overrides `execution.repeat`. |
43
44
  | `--min-pass <n>` | Overrides `execution.minPass`. |
44
45
  | `--quick` | One attempt that must pass: `repeat 1`, `minPass 1`. Wins over `--repeat` and `--min-pass`. |
@@ -109,7 +110,7 @@ evaluator:
109
110
  | `execution.timeouts.conversation` | 180000 | Milliseconds for one attempt's connection, from connect to close. Semantic checks after close are not counted. |
110
111
  | `evaluator.provider` | required | `openai` or `bedrock`. |
111
112
  | `evaluator.model` | required | The provider's model ID. |
112
- | `evaluator.apiKey` | OpenAI only | Must be an environment reference. |
113
+ | `evaluator.apiKey` | OpenAI only | Must be a reference, such as `${OPENAI_API_KEY}`. |
113
114
  | `evaluator.region` | Bedrock only | Credentials come from the AWS credential chain, for example `AWS_PROFILE`. |
114
115
  | `evaluator.concurrency` | 3 | Evaluator calls at once, across all scenarios. |
115
116
  | `evaluator.timeout` | 60000 | Milliseconds per call. |
@@ -120,18 +121,78 @@ evaluator:
120
121
  | `report.directory` | `./reports` | Where report files are written. |
121
122
  | `report.formats` | `[console]` | `console` and `json`. |
122
123
 
123
- Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has semantic criteria.
124
+ Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has a `should` check.
124
125
 
125
- ### Environment references
126
+ ### References
126
127
 
127
- `${NAME}` in any string value, in the configuration or a scenario, is replaced with the environment variable `NAME` after the YAML is parsed, so a value containing YAML syntax stays one string. A missing variable is an error at its location; its value never appears in messages. `.env` files are not loaded.
128
+ `${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
129
+
130
+ 1. the scenario's `variables:` (see [Variables](#variables));
131
+ 2. the configuration's `variables:`;
132
+ 3. the `--parameters` file, when one is given;
133
+ 4. the environment variable of that name.
134
+
135
+ A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
136
+
137
+ The parameters file is a JSON object. Each top-level string value is a parameter, and anything else is ignored, so an apiunit `parameters.json`, with its `settings` object, can be used as it is. Names may contain hyphens, as in apiunit:
138
+
139
+ ```json
140
+ { "knowledge-ws": "ws://localhost:8122", "user-token": "eyJ..." }
141
+ ```
142
+
143
+ ```yaml
144
+ websocket:
145
+ endpoint: ${knowledge-ws}
146
+ connection:
147
+ subprotocols:
148
+ - ${user-token}
149
+ ```
150
+
151
+ ```bash
152
+ miniwat-eval run eval/scenarios --parameters apiunit/parameters.json
153
+ ```
154
+
155
+ Keeping one file per environment, such as `parameters.json` and `parameters_cicd.json`, switches endpoints and tokens without changing the configuration.
156
+
157
+ ### Variables
158
+
159
+ `variables:` is a map of names to strings, at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
160
+
161
+ ```yaml
162
+ # eval-config.yml
163
+ websocket:
164
+ endpoint: ${knowledge-ws}
165
+ connection:
166
+ subprotocols:
167
+ - ${token}
168
+ ```
169
+
170
+ ```yaml
171
+ # scenarios/superadmin/greeting.yml
172
+ version: 1
173
+ name: Superadmin greeting
174
+ variables:
175
+ token: ${superadmin-token}
176
+ turns:
177
+ - input: Hello
178
+ ```
179
+
180
+ - **Scope.** A scenario's variables apply to that scenario and to the configuration's `websocket` section, which is resolved again for each scenario. `execution`, `evaluator` and `report` apply to the whole run, so they resolve once, from the configuration's variables, parameters and the environment; a reference in them that only a scenario sets is an error.
181
+ - **Defaults.** The configuration's `variables:` is optional and only needed for a default value.
182
+ - **Values.** A variable's value may contain references, which resolve from the levels below its own: `token: ${superadmin-token}` in a scenario reads the configuration's variables, parameters and the environment, never another scenario variable.
183
+ - **Subprotocols.** In `websocket.connection.subprotocols` only, an entry that resolves to an empty string, or whose reference is set nowhere, is left out. A scenario that does not set `token` above connects with no subprotocol. Anywhere else an unset reference is an error.
184
+ - **Typos.** A scenario variable that neither the scenario nor the `websocket` section references is an error at its key. Configuration variables are not checked, since some scenarios may not use them.
185
+ - **Errors.** A `websocket` problem that only some scenarios have is reported for each of them, for example `scenarios/a.yml:3:1: eval-config.yml:3:13 websocket.endpoint: ...`. One that every scenario has is a configuration error, reported once.
186
+
187
+ `--verbose` starts each scenario's transcript with `Connection: <endpoint>, <n> subprotocol(s)`, and the JSON report gives each scenario a `connection` with its endpoint and subprotocol count, so a scenario that connected without a token is visible.
128
188
 
129
189
  ### Redaction
130
190
 
131
191
  Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
132
192
 
133
193
  - the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
134
- - the value of every `${NAME}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY`, in any case.
194
+ - the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from a variable, `--parameters` or the environment. `${user-token}` is one;
195
+ - the value of every variable with such a name, such as `token:`.
135
196
 
136
197
  Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
137
198
 
@@ -152,7 +213,7 @@ turns:
152
213
  response:
153
214
  contains: [ORD-123]
154
215
  notContains: [error]
155
- criteria: Explains the delivery status.
216
+ should: Explains the delivery status.
156
217
  toolUsed:
157
218
  - name: getOrder
158
219
  status: true
@@ -163,21 +224,21 @@ turns:
163
224
  - getShipment
164
225
  expect:
165
226
  conversation:
166
- criteria: Keeps the order ID across turns.
227
+ should: Keeps the order ID across turns.
167
228
  ```
168
229
 
169
230
  Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
170
231
 
171
232
  | Expectation | Passes when |
172
233
  |---|---|
173
- | `response: <text>` | Shorthand for `response.criteria`. |
234
+ | `response: <text>` | Shorthand for `response.should`. |
174
235
  | `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
175
236
  | `response.notContains` | No entry appears in the response text. |
176
- | `response.criteria` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `criteria` together are one test. |
237
+ | `response.should` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `should` together are one test. |
177
238
  | `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
178
239
  | `toolNotUsed` | No tool use in the turn has the name. Names only. |
179
240
  | `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
180
- | `expect.conversation.criteria` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
241
+ | `expect.conversation.should` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
181
242
 
182
243
  An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
183
244