@mappies/miniwat-eval 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -11
- package/dist/index.cjs +389 -173
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +58 -48
- package/dist/index.d.ts +58 -48
- package/dist/index.js +389 -173
- package/dist/index.js.map +1 -1
- package/dist/main.js +345 -129
- package/dist/main.js.map +1 -1
- package/package.json +1 -1
- package/schemas/conversation.schema.json +9 -4
- package/schemas/eval-config.schema.json +5 -0
- package/schemas/report.schema.json +11 -1
package/README.md
CHANGED
|
@@ -12,7 +12,7 @@ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
|
|
|
12
12
|
npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
The provider is only needed when a scenario has
|
|
15
|
+
The provider is only needed when a scenario has a `should` check. Node.js 22 or later is required.
|
|
16
16
|
|
|
17
17
|
## Quick start
|
|
18
18
|
|
|
@@ -121,14 +121,16 @@ evaluator:
|
|
|
121
121
|
| `report.directory` | `./reports` | Where report files are written. |
|
|
122
122
|
| `report.formats` | `[console]` | `console` and `json`. |
|
|
123
123
|
|
|
124
|
-
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has
|
|
124
|
+
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has a `should` check.
|
|
125
125
|
|
|
126
126
|
### References
|
|
127
127
|
|
|
128
|
-
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from:
|
|
128
|
+
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
|
|
129
129
|
|
|
130
|
-
1. the
|
|
131
|
-
2.
|
|
130
|
+
1. the scenario's `variables:` (see [Variables](#variables));
|
|
131
|
+
2. the configuration's `variables:`;
|
|
132
|
+
3. the `--parameters` file, when one is given;
|
|
133
|
+
4. the environment variable of that name.
|
|
132
134
|
|
|
133
135
|
A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
|
|
134
136
|
|
|
@@ -152,12 +154,45 @@ miniwat-eval run eval/scenarios --parameters apiunit/parameters.json
|
|
|
152
154
|
|
|
153
155
|
Keeping one file per environment, such as `parameters.json` and `parameters_cicd.json`, switches endpoints and tokens without changing the configuration.
|
|
154
156
|
|
|
157
|
+
### Variables
|
|
158
|
+
|
|
159
|
+
`variables:` is a map of names to strings, at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
|
|
160
|
+
|
|
161
|
+
```yaml
|
|
162
|
+
# eval-config.yml
|
|
163
|
+
websocket:
|
|
164
|
+
endpoint: ${knowledge-ws}
|
|
165
|
+
connection:
|
|
166
|
+
subprotocols:
|
|
167
|
+
- ${token}
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
```yaml
|
|
171
|
+
# scenarios/superadmin/greeting.yml
|
|
172
|
+
version: 1
|
|
173
|
+
name: Superadmin greeting
|
|
174
|
+
variables:
|
|
175
|
+
token: ${superadmin-token}
|
|
176
|
+
turns:
|
|
177
|
+
- input: Hello
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
- **Scope.** A scenario's variables apply to that scenario and to the configuration's `websocket` section, which is resolved again for each scenario. `execution`, `evaluator` and `report` apply to the whole run, so they resolve once, from the configuration's variables, parameters and the environment; a reference in them that only a scenario sets is an error.
|
|
181
|
+
- **Defaults.** The configuration's `variables:` is optional and only needed for a default value.
|
|
182
|
+
- **Values.** A variable's value may contain references, which resolve from the levels below its own: `token: ${superadmin-token}` in a scenario reads the configuration's variables, parameters and the environment, never another scenario variable.
|
|
183
|
+
- **Subprotocols.** In `websocket.connection.subprotocols` only, an entry that resolves to an empty string, or whose reference is set nowhere, is left out. A scenario that does not set `token` above connects with no subprotocol. Anywhere else an unset reference is an error.
|
|
184
|
+
- **Typos.** A scenario variable that neither the scenario nor the `websocket` section references is an error at its key. Configuration variables are not checked, since some scenarios may not use them.
|
|
185
|
+
- **Errors.** A `websocket` problem that only some scenarios have is reported for each of them, for example `scenarios/a.yml:3:1: eval-config.yml:3:13 websocket.endpoint: ...`. One that every scenario has is a configuration error, reported once.
|
|
186
|
+
|
|
187
|
+
`--verbose` starts each scenario's transcript with `Connection: <endpoint>, <n> subprotocol(s)`, and the JSON report gives each scenario a `connection` with its endpoint and subprotocol count, so a scenario that connected without a token is visible.
|
|
188
|
+
|
|
155
189
|
### Redaction
|
|
156
190
|
|
|
157
191
|
Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
|
|
158
192
|
|
|
159
193
|
- the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
|
|
160
|
-
- the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from `--parameters` or the environment. `${user-token}` is one
|
|
194
|
+
- the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from a variable, `--parameters` or the environment. `${user-token}` is one;
|
|
195
|
+
- the value of every variable with such a name, such as `token:`.
|
|
161
196
|
|
|
162
197
|
Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
|
|
163
198
|
|
|
@@ -178,7 +213,7 @@ turns:
|
|
|
178
213
|
response:
|
|
179
214
|
contains: [ORD-123]
|
|
180
215
|
notContains: [error]
|
|
181
|
-
|
|
216
|
+
should: Explains the delivery status.
|
|
182
217
|
toolUsed:
|
|
183
218
|
- name: getOrder
|
|
184
219
|
status: true
|
|
@@ -189,21 +224,21 @@ turns:
|
|
|
189
224
|
- getShipment
|
|
190
225
|
expect:
|
|
191
226
|
conversation:
|
|
192
|
-
|
|
227
|
+
should: Keeps the order ID across turns.
|
|
193
228
|
```
|
|
194
229
|
|
|
195
230
|
Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
|
|
196
231
|
|
|
197
232
|
| Expectation | Passes when |
|
|
198
233
|
|---|---|
|
|
199
|
-
| `response: <text>` | Shorthand for `response.
|
|
234
|
+
| `response: <text>` | Shorthand for `response.should`. |
|
|
200
235
|
| `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
|
|
201
236
|
| `response.notContains` | No entry appears in the response text. |
|
|
202
|
-
| `response.
|
|
237
|
+
| `response.should` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `should` together are one test. |
|
|
203
238
|
| `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
|
|
204
239
|
| `toolNotUsed` | No tool use in the turn has the name. Names only. |
|
|
205
240
|
| `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
|
|
206
|
-
| `expect.conversation.
|
|
241
|
+
| `expect.conversation.should` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
|
|
207
242
|
|
|
208
243
|
An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
|
|
209
244
|
|