@mappies/miniwat-eval 1.0.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +74 -13
- package/dist/index.cjs +435 -178
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +72 -51
- package/dist/index.d.ts +72 -51
- package/dist/index.js +434 -178
- package/dist/index.js.map +1 -1
- package/dist/main.js +410 -148
- package/dist/main.js.map +1 -1
- package/package.json +1 -1
- package/schemas/conversation.schema.json +9 -4
- package/schemas/eval-config.schema.json +5 -0
- package/schemas/report.schema.json +12 -1
package/README.md
CHANGED
|
@@ -12,7 +12,7 @@ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
|
|
|
12
12
|
npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
The provider is only needed when a scenario has
|
|
15
|
+
The provider is only needed when a scenario has a `should` check. Node.js 22 or later is required.
|
|
16
16
|
|
|
17
17
|
## Quick start
|
|
18
18
|
|
|
@@ -30,8 +30,8 @@ npx miniwat-eval run examples/scenarios --quick
|
|
|
30
30
|
## Commands
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
|
-
miniwat-eval validate <target> [--config <path>]
|
|
34
|
-
miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
|
|
33
|
+
miniwat-eval validate <target> [--config <path>] [--parameters <path>]
|
|
34
|
+
miniwat-eval run <target> [--config <path>] [--parameters <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
|
|
35
35
|
```
|
|
36
36
|
|
|
37
37
|
`<target>` is a scenario file, or a directory searched recursively for `*.yml` and `*.yaml`. The configuration file and the report directory are skipped.
|
|
@@ -39,6 +39,7 @@ miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--q
|
|
|
39
39
|
| Option | Effect |
|
|
40
40
|
|---|---|
|
|
41
41
|
| `--config <path>` | The configuration file. Without it, `eval-config.yml` is searched for upward from the target, stopping at the nearest directory with `.git` or `package.json`, then in the working directory. |
|
|
42
|
+
| `--parameters <path>` | A JSON file of values for `${name}` references, looked up before environment variables. See [References](#references). |
|
|
42
43
|
| `--repeat <n>` | Overrides `execution.repeat`. |
|
|
43
44
|
| `--min-pass <n>` | Overrides `execution.minPass`. |
|
|
44
45
|
| `--quick` | One attempt that must pass: `repeat 1`, `minPass 1`. Wins over `--repeat` and `--min-pass`. |
|
|
@@ -109,7 +110,7 @@ evaluator:
|
|
|
109
110
|
| `execution.timeouts.conversation` | 180000 | Milliseconds for one attempt's connection, from connect to close. Semantic checks after close are not counted. |
|
|
110
111
|
| `evaluator.provider` | required | `openai` or `bedrock`. |
|
|
111
112
|
| `evaluator.model` | required | The provider's model ID. |
|
|
112
|
-
| `evaluator.apiKey` | OpenAI only | Must be
|
|
113
|
+
| `evaluator.apiKey` | OpenAI only | Must be a reference, such as `${OPENAI_API_KEY}`. |
|
|
113
114
|
| `evaluator.region` | Bedrock only | Credentials come from the AWS credential chain, for example `AWS_PROFILE`. |
|
|
114
115
|
| `evaluator.concurrency` | 3 | Evaluator calls at once, across all scenarios. |
|
|
115
116
|
| `evaluator.timeout` | 60000 | Milliseconds per call. |
|
|
@@ -120,18 +121,78 @@ evaluator:
|
|
|
120
121
|
| `report.directory` | `./reports` | Where report files are written. |
|
|
121
122
|
| `report.formats` | `[console]` | `console` and `json`. |
|
|
122
123
|
|
|
123
|
-
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has
|
|
124
|
+
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has a `should` check.
|
|
124
125
|
|
|
125
|
-
###
|
|
126
|
+
### References
|
|
126
127
|
|
|
127
|
-
`${
|
|
128
|
+
`${name}` in any string value, in the configuration or a scenario, is replaced after the YAML is parsed, so a value containing YAML syntax stays one string. The value comes from the first of these that has it:
|
|
129
|
+
|
|
130
|
+
1. the scenario's `variables:` (see [Variables](#variables));
|
|
131
|
+
2. the configuration's `variables:`;
|
|
132
|
+
3. the `--parameters` file, when one is given;
|
|
133
|
+
4. the environment variable of that name.
|
|
134
|
+
|
|
135
|
+
A missing value is an error at its location; values never appear in messages. `.env` files are not loaded.
|
|
136
|
+
|
|
137
|
+
The parameters file is a JSON object. Each top-level string value is a parameter, and anything else is ignored, so an apiunit `parameters.json`, with its `settings` object, can be used as it is. Names may contain hyphens, as in apiunit:
|
|
138
|
+
|
|
139
|
+
```json
|
|
140
|
+
{ "knowledge-ws": "ws://localhost:8122", "user-token": "eyJ..." }
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
```yaml
|
|
144
|
+
websocket:
|
|
145
|
+
endpoint: ${knowledge-ws}
|
|
146
|
+
connection:
|
|
147
|
+
subprotocols:
|
|
148
|
+
- ${user-token}
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
miniwat-eval run eval/scenarios --parameters apiunit/parameters.json
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Keeping one file per environment, such as `parameters.json` and `parameters_cicd.json`, switches endpoints and tokens without changing the configuration.
|
|
156
|
+
|
|
157
|
+
### Variables
|
|
158
|
+
|
|
159
|
+
`variables:` is a map of names to strings, at the top of a scenario file or of the configuration. Names follow the same rule as references. One configuration can then serve scenarios that connect differently, for example as different users:
|
|
160
|
+
|
|
161
|
+
```yaml
|
|
162
|
+
# eval-config.yml
|
|
163
|
+
websocket:
|
|
164
|
+
endpoint: ${knowledge-ws}
|
|
165
|
+
connection:
|
|
166
|
+
subprotocols:
|
|
167
|
+
- ${token}
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
```yaml
|
|
171
|
+
# scenarios/superadmin/greeting.yml
|
|
172
|
+
version: 1
|
|
173
|
+
name: Superadmin greeting
|
|
174
|
+
variables:
|
|
175
|
+
token: ${superadmin-token}
|
|
176
|
+
turns:
|
|
177
|
+
- input: Hello
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
- **Scope.** A scenario's variables apply to that scenario and to the configuration's `websocket` section, which is resolved again for each scenario. `execution`, `evaluator` and `report` apply to the whole run, so they resolve once, from the configuration's variables, parameters and the environment; a reference in them that only a scenario sets is an error.
|
|
181
|
+
- **Defaults.** The configuration's `variables:` is optional and only needed for a default value.
|
|
182
|
+
- **Values.** A variable's value may contain references, which resolve from the levels below its own: `token: ${superadmin-token}` in a scenario reads the configuration's variables, parameters and the environment, never another scenario variable.
|
|
183
|
+
- **Subprotocols.** In `websocket.connection.subprotocols` only, an entry that resolves to an empty string, or whose reference is set nowhere, is left out. A scenario that does not set `token` above connects with no subprotocol. Anywhere else an unset reference is an error.
|
|
184
|
+
- **Typos.** A scenario variable that neither the scenario nor the `websocket` section references is an error at its key. Configuration variables are not checked, since some scenarios may not use them.
|
|
185
|
+
- **Errors.** A `websocket` problem that only some scenarios have is reported for each of them, for example `scenarios/a.yml:3:1: eval-config.yml:3:13 websocket.endpoint: ...`. One that every scenario has is a configuration error, reported once.
|
|
186
|
+
|
|
187
|
+
`--verbose` starts each scenario's transcript with `Connection: <endpoint>, <n> subprotocol(s)`, and the JSON report gives each scenario a `connection` with its endpoint and subprotocol count, so a scenario that connected without a token is visible.
|
|
128
188
|
|
|
129
189
|
### Redaction
|
|
130
190
|
|
|
131
191
|
Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
|
|
132
192
|
|
|
133
193
|
- the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
|
|
134
|
-
- the value of every `${
|
|
194
|
+
- the value of every `${name}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY` in any case, whether it comes from a variable, `--parameters` or the environment. `${user-token}` is one;
|
|
195
|
+
- the value of every variable with such a name, such as `token:`.
|
|
135
196
|
|
|
136
197
|
Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
|
|
137
198
|
|
|
@@ -152,7 +213,7 @@ turns:
|
|
|
152
213
|
response:
|
|
153
214
|
contains: [ORD-123]
|
|
154
215
|
notContains: [error]
|
|
155
|
-
|
|
216
|
+
should: Explains the delivery status.
|
|
156
217
|
toolUsed:
|
|
157
218
|
- name: getOrder
|
|
158
219
|
status: true
|
|
@@ -163,21 +224,21 @@ turns:
|
|
|
163
224
|
- getShipment
|
|
164
225
|
expect:
|
|
165
226
|
conversation:
|
|
166
|
-
|
|
227
|
+
should: Keeps the order ID across turns.
|
|
167
228
|
```
|
|
168
229
|
|
|
169
230
|
Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
|
|
170
231
|
|
|
171
232
|
| Expectation | Passes when |
|
|
172
233
|
|---|---|
|
|
173
|
-
| `response: <text>` | Shorthand for `response.
|
|
234
|
+
| `response: <text>` | Shorthand for `response.should`. |
|
|
174
235
|
| `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
|
|
175
236
|
| `response.notContains` | No entry appears in the response text. |
|
|
176
|
-
| `response.
|
|
237
|
+
| `response.should` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `should` together are one test. |
|
|
177
238
|
| `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
|
|
178
239
|
| `toolNotUsed` | No tool use in the turn has the name. Names only. |
|
|
179
240
|
| `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
|
|
180
|
-
| `expect.conversation.
|
|
241
|
+
| `expect.conversation.should` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
|
|
181
242
|
|
|
182
243
|
An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
|
|
183
244
|
|