@mappies/miniwat-eval 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +89 -11
- package/dist/index.cjs +743 -178
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +65 -48
- package/dist/index.d.ts +66 -48
- package/dist/index.js +743 -178
- package/dist/index.js.map +1 -1
- package/dist/main.js +699 -134
- package/dist/main.js.map +1 -1
- package/package.json +1 -1
- package/schemas/conversation.schema.json +43 -4
- package/schemas/eval-config.schema.json +10 -0
- package/schemas/report.schema.json +12 -1
package/README.md
CHANGED
|
@@ -12,7 +12,7 @@ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
|
|
|
12
12
|
npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
The provider is only needed when a scenario has
|
|
15
|
+
The provider is only needed when a scenario has a `should` check. Node.js 22 or later is required.
|
|
16
16
|
|
|
17
17
|
## Quick start
|
|
18
18
|
|
|
@@ -121,14 +121,17 @@ evaluator:
|
|
|
121
121
|
| `report.directory` | `./reports` | Where report files are written. |
|
|
122
122
|
| `report.formats` | `[console]` | `console` and `json`. |
|
|
123
123
|
|
|
124
|
-
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has
|
|
124
|
+
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has a `should` check.
|
|
125
125
|
|
|
126
126
|
### References
|
|
127
127
|
|
|
128
|
-
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from:
|
|
128
|
+
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
|
|
129
129
|
|
|
130
|
-
1. the
|
|
131
|
-
2.
|
|
130
|
+
1. the run's variables, for a scenario with `runWith:` (see [Running a scenario several times](#running-a-scenario-several-times));
|
|
131
|
+
2. the scenario's `variables:` (see [Variables](#variables));
|
|
132
|
+
3. the configuration's `variables:`;
|
|
133
|
+
4. the `--parameters` file, when one is given;
|
|
134
|
+
5. the environment variable of that name.
|
|
132
135
|
|
|
133
136
|
A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
|
|
134
137
|
|
|
@@ -152,12 +155,45 @@ miniwat-eval run eval/scenarios --parameters apiunit/parameters.json
|
|
|
152
155
|
|
|
153
156
|
Keeping one file per environment, such as `parameters.json` and `parameters_cicd.json`, switches endpoints and tokens without changing the configuration.
|
|
154
157
|
|
|
158
|
+
### Variables
|
|
159
|
+
|
|
160
|
+
`variables:` is a map of names to strings, or to lists of strings (see [Running a scenario several times](#running-a-scenario-several-times)), at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
|
|
161
|
+
|
|
162
|
+
```yaml
|
|
163
|
+
# eval-config.yml
|
|
164
|
+
websocket:
|
|
165
|
+
endpoint: ${knowledge-ws}
|
|
166
|
+
connection:
|
|
167
|
+
subprotocols:
|
|
168
|
+
- ${token}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
```yaml
|
|
172
|
+
# scenarios/superadmin/greeting.yml
|
|
173
|
+
version: 1
|
|
174
|
+
name: Superadmin greeting
|
|
175
|
+
variables:
|
|
176
|
+
token: ${superadmin-token}
|
|
177
|
+
turns:
|
|
178
|
+
- input: Hello
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
- **Scope.** A scenario's variables apply to that scenario and to the configuration's `websocket` section, which is resolved again for each scenario. `execution`, `evaluator` and `report` apply to the whole run, so they resolve once, from the configuration's variables, parameters and the environment; a reference in them that only a scenario sets is an error.
|
|
182
|
+
- **Defaults.** The configuration's `variables:` is optional and only needed for a default value.
|
|
183
|
+
- **Values.** A variable's value may contain references, which resolve from the levels below its own: `token: ${superadmin-token}` in a scenario reads the configuration's variables, parameters and the environment, never another scenario variable.
|
|
184
|
+
- **Subprotocols.** In `websocket.connection.subprotocols` only, an entry that resolves to an empty string, or whose reference is set nowhere, is left out. A scenario that does not set `token` above connects with no subprotocol. Anywhere else an unset reference is an error.
|
|
185
|
+
- **Typos.** A scenario variable that neither the scenario nor the `websocket` section references is an error at its key. Configuration variables are not checked, since some scenarios may not use them.
|
|
186
|
+
- **Errors.** A `websocket` problem that only some scenarios have is reported for each of them, for example `scenarios/a.yml:3:1: eval-config.yml:3:13 websocket.endpoint: ...`. One that every scenario has is a configuration error, reported once.
|
|
187
|
+
|
|
188
|
+
`--verbose` starts each scenario's transcript with `Connection: <endpoint>, <n> subprotocol(s)`, and the JSON report gives each scenario a `connection` with its endpoint and subprotocol count, so a scenario that connected without a token is visible.
|
|
189
|
+
|
|
155
190
|
### Redaction
|
|
156
191
|
|
|
157
192
|
Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
|
|
158
193
|
|
|
159
194
|
- the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
|
|
160
|
-
- the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from `--parameters` or the environment. `${user-token}` is one
|
|
195
|
+
- the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from a variable, `--parameters` or the environment. `${user-token}` is one;
|
|
196
|
+
- the value of every variable with such a name, such as `token:`.
|
|
161
197
|
|
|
162
198
|
Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
|
|
163
199
|
|
|
@@ -178,7 +214,7 @@ turns:
|
|
|
178
214
|
response:
|
|
179
215
|
contains: [ORD-123]
|
|
180
216
|
notContains: [error]
|
|
181
|
-
|
|
217
|
+
should: Explains the delivery status.
|
|
182
218
|
toolUsed:
|
|
183
219
|
- name: getOrder
|
|
184
220
|
status: true
|
|
@@ -189,26 +225,68 @@ turns:
|
|
|
189
225
|
- getShipment
|
|
190
226
|
expect:
|
|
191
227
|
conversation:
|
|
192
|
-
|
|
228
|
+
should: Keeps the order ID across turns.
|
|
193
229
|
```
|
|
194
230
|
|
|
195
231
|
Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
|
|
196
232
|
|
|
197
233
|
| Expectation | Passes when |
|
|
198
234
|
|---|---|
|
|
199
|
-
| `response: <text>` | Shorthand for `response.
|
|
235
|
+
| `response: <text>` | Shorthand for `response.should`. |
|
|
200
236
|
| `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
|
|
201
237
|
| `response.notContains` | No entry appears in the response text. |
|
|
202
|
-
| `response.
|
|
238
|
+
| `response.should` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `should` together are one test. |
|
|
203
239
|
| `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
|
|
204
240
|
| `toolNotUsed` | No tool use in the turn has the name. Names only. |
|
|
205
241
|
| `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
|
|
206
|
-
| `expect.conversation.
|
|
242
|
+
| `expect.conversation.should` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
|
|
207
243
|
|
|
208
244
|
An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
|
|
209
245
|
|
|
210
246
|
The JSON Schemas in `schemas/` describe the configuration, scenarios and the JSON report, and can be used for editor completion.
|
|
211
247
|
|
|
248
|
+
### Running a scenario several times
|
|
249
|
+
|
|
250
|
+
`runWith:` runs one scenario once per entry, each with its own variables. Each run is a scenario of its own in every count, in the exit code and in the reports, named `Greeting [student]`. It has its own connection, so a run can change the token.
|
|
251
|
+
|
|
252
|
+
Rows set several variables per run. `name` is required and labels the run; every other key is a variable:
|
|
253
|
+
|
|
254
|
+
```yaml
|
|
255
|
+
version: 1
|
|
256
|
+
name: Greeting
|
|
257
|
+
runWith:
|
|
258
|
+
- name: instructor
|
|
259
|
+
token: ${instructor-token}
|
|
260
|
+
org: Acme University
|
|
261
|
+
- name: visitor # sets no token, so it connects without one
|
|
262
|
+
org: Acme University
|
|
263
|
+
turns:
|
|
264
|
+
- input: Hello, I am from ${org}.
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`{ variable, from }` sets one variable from a list of variable or parameter names. Each run is named after its entry, never after the value, so a token never appears in a run's name:
|
|
268
|
+
|
|
269
|
+
```yaml
|
|
270
|
+
runWith:
|
|
271
|
+
variable: token
|
|
272
|
+
from: [instructor-token, student-token] # or a list variable: ${authenticated-tokens}
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
A list defined once in the configuration can be shared by many scenarios:
|
|
276
|
+
|
|
277
|
+
```yaml
|
|
278
|
+
# eval-config.yml
|
|
279
|
+
variables:
|
|
280
|
+
authenticated-tokens: [instructor-token, student-token, user-token]
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
- **Values:** a run's variables come first in [references](#references). A row value may use the scenario's variables, but a scenario variable never sees a run's value.
|
|
284
|
+
- **Defaults:** a scenario variable is the default for runs that do not set it, so one that every run overrides is allowed.
|
|
285
|
+
- **Lists:** a list variable can only be used as the whole value of `from`. Its items are names, and are not resolved themselves. Lists come only from `variables:` sections, because parameters and environment variables are strings.
|
|
286
|
+
- **Checks:** run names are unique within a scenario and match `[A-Za-z0-9][A-Za-z0-9_.-]*`. A run variable that neither the scenario nor the `websocket` section references is an error, as for scenario variables.
|
|
287
|
+
- **Errors:** a problem only some runs have names the run, for example `scenarios/order.yml:9:14 turns[0].input: run 'lost': ...`. One that every run has is reported once.
|
|
288
|
+
- **Reports:** the JSON report keeps each scenario's `name` and gives the run's name as `runWith`.
|
|
289
|
+
|
|
212
290
|
## Results
|
|
213
291
|
|
|
214
292
|
| Level | Statuses |
|