@mappies/miniwat-eval 0.0.0-stage → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,131 @@
1
+ # PolyForm Noncommercial License 1.0.0
2
+
3
+ <https://polyformproject.org/licenses/noncommercial/1.0.0>
4
+
5
+ ## Acceptance
6
+
7
+ In order to get any license under these terms, you must agree
8
+ to them as both strict obligations and conditions to all
9
+ your licenses.
10
+
11
+ ## Copyright License
12
+
13
+ The licensor grants you a copyright license for the
14
+ software to do everything you might do with the software
15
+ that would otherwise infringe the licensor's copyright
16
+ in it for any permitted purpose. However, you may
17
+ only distribute the software according to [Distribution
18
+ License](#distribution-license) and make changes or new works
19
+ based on the software according to [Changes and New Works
20
+ License](#changes-and-new-works-license).
21
+
22
+ ## Distribution License
23
+
24
+ The licensor grants you an additional copyright license
25
+ to distribute copies of the software. Your license
26
+ to distribute covers distributing the software with
27
+ changes and new works permitted by [Changes and New Works
28
+ License](#changes-and-new-works-license).
29
+
30
+ ## Notices
31
+
32
+ You must ensure that anyone who gets a copy of any part of
33
+ the software from you also gets a copy of these terms or the
34
+ URL for them above, as well as copies of any plain-text lines
35
+ beginning with `Required Notice:` that the licensor provided
36
+ with the software. For example:
37
+
38
+ > Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
39
+
40
+ ## Changes and New Works License
41
+
42
+ The licensor grants you an additional copyright license to
43
+ make changes and new works based on the software for any
44
+ permitted purpose.
45
+
46
+ ## Patent License
47
+
48
+ The licensor grants you a patent license for the software that
49
+ covers patent claims the licensor can license, or becomes able
50
+ to license, that you would infringe by using the software.
51
+
52
+ ## Noncommercial Purposes
53
+
54
+ Any noncommercial purpose is a permitted purpose.
55
+
56
+ ## Personal Uses
57
+
58
+ Personal use for research, experiment, and testing for
59
+ the benefit of public knowledge, personal study, private
60
+ entertainment, hobby projects, amateur pursuits, or religious
61
+ observance, without any anticipated commercial application,
62
+ is use for a permitted purpose.
63
+
64
+ ## Noncommercial Organizations
65
+
66
+ Use by any charitable organization, educational institution,
67
+ public research organization, public safety or health
68
+ organization, environmental protection organization,
69
+ or government institution is use for a permitted purpose
70
+ regardless of the source of funding or obligations resulting
71
+ from the funding.
72
+
73
+ ## Fair Use
74
+
75
+ You may have "fair use" rights for the software under the
76
+ law. These terms do not limit them.
77
+
78
+ ## No Other Rights
79
+
80
+ These terms do not allow you to sublicense or transfer any of
81
+ your licenses to anyone else, or prevent the licensor from
82
+ granting licenses to anyone else. These terms do not imply
83
+ any other licenses.
84
+
85
+ ## Patent Defense
86
+
87
+ If you make any written claim that the software infringes or
88
+ contributes to infringement of any patent, your patent license
89
+ for the software granted under these terms ends immediately. If
90
+ your company makes such a claim, your patent license ends
91
+ immediately for work on behalf of your company.
92
+
93
+ ## Violations
94
+
95
+ The first time you are notified in writing that you have
96
+ violated any of these terms, or done anything with the software
97
+ not covered by your licenses, your licenses can nonetheless
98
+ continue if you come into full compliance with these terms,
99
+ and take practical steps to correct past violations, within
100
+ 32 days of receiving notice. Otherwise, all your licenses
101
+ end immediately.
102
+
103
+ ## No Liability
104
+
105
+ ***As far as the law allows, the software comes as is, without
106
+ any warranty or condition, and the licensor will not be liable
107
+ to you for any damages arising out of these terms or the use
108
+ or nature of the software, under any kind of legal claim.***
109
+
110
+ ## Definitions
111
+
112
+ The **licensor** is the individual or entity offering these
113
+ terms, and the **software** is the software the licensor makes
114
+ available under these terms.
115
+
116
+ **You** refers to the individual or entity agreeing to these
117
+ terms.
118
+
119
+ **Your company** is any legal entity, sole proprietorship,
120
+ or other kind of organization that you work for, plus all
121
+ organizations that have control over, are under the control of,
122
+ or are under common control with that organization. **Control**
123
+ means ownership of substantially all the assets of an entity,
124
+ or the power to direct its management and policies by vote,
125
+ contract, or otherwise. Control can be direct or indirect.
126
+
127
+ **Your licenses** are all the licenses granted to you for the
128
+ software under these terms.
129
+
130
+ **Use** means anything you do with the software requiring one
131
+ of your licenses.
package/README.md CHANGED
@@ -1,3 +1,248 @@
1
- # Temporary Holding Version
1
+ # miniwat-eval
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Black-box, YAML-driven evaluation for agents built on `miniwat-agent`. You write a conversation and what you expect from it in YAML. Miniwat Eval sends the turns to the agent over WebSocket, records what comes back, and checks it with literal, tool and evaluator-scored assertions. It repeats conversations to detect flaky behaviour, and reports the results with a CI exit code.
4
+
5
+ It talks to the host only through its WebSocket, the way a production client does. Conversations are real: every repetition repeats every tool action the agent takes, so run it against test environments and test data.
6
+
7
+ ## Install
8
+
9
+ ```bash
10
+ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
11
+ # or, for an Amazon Bedrock evaluator:
12
+ npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
13
+ ```
14
+
15
+ The provider is only needed when a scenario has semantic criteria. Node.js 22 or later is required.
16
+
17
+ ## Quick start
18
+
19
+ The repository's `examples/` folder has a working configuration and scenarios. From a clone, with an agent listening:
20
+
21
+ ```bash
22
+ cp examples/.env.example examples/.env # then fill it in
23
+ set -a; . examples/.env; set +a # miniwat-eval does not read .env files itself
24
+ npx miniwat-eval validate examples/scenarios
25
+ npx miniwat-eval run examples/scenarios --quick
26
+ ```
27
+
28
+ `validate` checks everything with no network calls. `run` validates the same way, then runs each scenario.
29
+
30
+ ## Commands
31
+
32
+ ```bash
33
+ miniwat-eval validate <target> [--config <path>]
34
+ miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
35
+ ```
36
+
37
+ `<target>` is a scenario file, or a directory searched recursively for `*.yml` and `*.yaml`. The configuration file and the report directory are skipped.
38
+
39
+ | Option | Effect |
40
+ |---|---|
41
+ | `--config <path>` | The configuration file. Without it, `eval-config.yml` is searched for upward from the target, stopping at the nearest directory with `.git` or `package.json`, then in the working directory. |
42
+ | `--repeat <n>` | Overrides `execution.repeat`. |
43
+ | `--min-pass <n>` | Overrides `execution.minPass`. |
44
+ | `--quick` | One attempt that must pass: `repeat 1`, `minPass 1`. Wins over `--repeat` and `--min-pass`. |
45
+ | `--concurrency <n>` | Overrides `execution.scenarioConcurrency`. |
46
+ | `--report <formats>` | Comma-separated `console`, `json`. Replaces `report.formats`. |
47
+ | `--verbose` | Adds every attempt's transcript to the console report. |
48
+
49
+ Option values are positive base-10 integers; `0`, `2.5` and `3x` are rejected. A problem with the configuration, a scenario or an option is printed as `file:line:column path: message` where it has a location, and nothing runs.
50
+
51
+ ### Exit codes
52
+
53
+ | Code | Meaning |
54
+ |---|---|
55
+ | 0 | Every scenario passed. FLAKY counts as passing unless `flakyPolicy: fail`. |
56
+ | 1 to 255 | The number of unsuccessful tests, capped at 255. Each test that is `FAIL` or `ERROR` across attempts counts once, and `FLAKY` counts with `flakyPolicy: fail`. A scenario that did not pass adds 1 when none of its tests counted. |
57
+ | 1 | Also: an invalid configuration, scenario or option, or a missing provider package. A report that could not be written raises the code to at least 1. |
58
+ | 130 | Interrupted by Ctrl+C, whatever the results. |
59
+
60
+ The JSON report records the exact count and status, since 1 and 130 can mean more than one thing.
61
+
62
+ ## Configuration
63
+
64
+ Each project has one `eval-config.yml`. Paths inside it resolve against its own directory. `examples/eval-config.yml` shows every key with a comment.
65
+
66
+ ```yaml
67
+ version: 1
68
+
69
+ websocket:
70
+ endpoint: ${MINIWAT_AGENT_ENDPOINT}
71
+ connection:
72
+ subprotocols:
73
+ - ${AGENT_ACCESS_TOKEN}
74
+ protocol:
75
+ request:
76
+ value:
77
+ name: add_session_message
78
+ payload: {}
79
+ inputPath: /payload/message
80
+ capabilities:
81
+ toolUses: true
82
+ toolResults: true
83
+
84
+ evaluator:
85
+ provider: openai
86
+ model: ${EVALUATOR_MODEL}
87
+ apiKey: ${OPENAI_API_KEY}
88
+ ```
89
+
90
+ | Key | Default | Notes |
91
+ |---|---|---|
92
+ | `version` | required | `1`. |
93
+ | `websocket.endpoint` | required | A `ws://` or `wss://` URL. |
94
+ | `websocket.connection.subprotocols` | `[]` | WebSocket subprotocols sent on connect, for example an access token. |
95
+ | `websocket.protocol.request.format` | `json` | The only format. |
96
+ | `websocket.protocol.request.value` | required | The request template, an object or array. |
97
+ | `websocket.protocol.request.inputPath` | required | A JSON Pointer such as `/payload/message`. Each turn's input is written there in a copy of `value`. Its parent must exist in `value`. |
98
+ | `websocket.protocol.capabilities.toolUses` | `false` | The host sends tool uses. Needed by `toolUsed`, `toolNotUsed` and `toolsInOrder`. |
99
+ | `websocket.protocol.capabilities.toolResults` | `false` | The host sends tool results. Needed by `toolUsed` entries with `status`. |
100
+ | `websocket.protocol.capabilities.tokenUsage` | `false` | The host sends token usage. Without it, agent tokens show as not supported. |
101
+ | `execution.scenarioConcurrency` | 3 | Scenarios run at once. Attempts and turns are always sequential. |
102
+ | `execution.repeat` | 1 | Attempts per scenario. All of them run. |
103
+ | `execution.minPass` | 1 | Passing attempts a scenario needs, from 1 to `repeat`. |
104
+ | `execution.continueOnFailure` | `true` | `true`: every turn runs and semantic checks run after the conversation. `false`: each turn's checks run as it completes, and the attempt stops at the first `FAIL`. |
105
+ | `execution.flakyPolicy` | `warn` | `fail` makes a `FLAKY` scenario count towards the exit code. |
106
+ | `execution.connectionRetries` | 2 | Retries of a failed connect, only before the first turn, with backoff from 250 ms doubling. |
107
+ | `execution.timeouts.connection` | 10000 | Milliseconds from connect to ready. |
108
+ | `execution.timeouts.turn` | 60000 | Milliseconds from sending a turn to its completion. |
109
+ | `execution.timeouts.conversation` | 180000 | Milliseconds for one attempt's connection, from connect to close. Semantic checks after close are not counted. |
110
+ | `evaluator.provider` | required | `openai` or `bedrock`. |
111
+ | `evaluator.model` | required | The provider's model ID. |
112
+ | `evaluator.apiKey` | OpenAI only | Must be an environment reference. |
113
+ | `evaluator.region` | Bedrock only | Credentials come from the AWS credential chain, for example `AWS_PROFILE`. |
114
+ | `evaluator.concurrency` | 3 | Evaluator calls at once, across all scenarios. |
115
+ | `evaluator.timeout` | 60000 | Milliseconds per call. |
116
+ | `evaluator.retries` | 2 | Retries of a call that failed, timed out or returned invalid output. A valid low score is not retried. |
117
+ | `evaluator.maxToolResultChars` | 4000 | In the evaluator's prompt, the current turn's tool result content is cut to this length; earlier turns' results keep only their status. `0` means no cut. Reports keep the full content. |
118
+ | `evaluator.thresholds.response` | 0.85 | A response score at or above this passes. |
119
+ | `evaluator.thresholds.conversation` | 0.85 | The same, for `expect.conversation`. |
120
+ | `report.directory` | `./reports` | Where report files are written. |
121
+ | `report.formats` | `[console]` | `console` and `json`. |
122
+
123
+ Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has semantic criteria.
124
+
125
+ ### Environment references
126
+
127
+ `${NAME}` in any string value, in the configuration or a scenario, is replaced with the environment variable `NAME` after the YAML is parsed, so a value containing YAML syntax stays one string. A missing variable is an error at its location; its value never appears in messages. `.env` files are not loaded.
128
+
129
+ ### Redaction
130
+
131
+ Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
132
+
133
+ - the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
134
+ - the value of every `${NAME}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY`, in any case.
135
+
136
+ Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
137
+
138
+ ## Scenarios
139
+
140
+ ```yaml
141
+ version: 1
142
+ name: Order lookup
143
+ description: Optional.
144
+ turns:
145
+ - input: Where is my order?
146
+ expect:
147
+ response: Asks the user for their order ID.
148
+ toolNotUsed:
149
+ - getOrder
150
+ - input: My order ID is ORD-123.
151
+ expect:
152
+ response:
153
+ contains: [ORD-123]
154
+ notContains: [error]
155
+ criteria: Explains the delivery status.
156
+ toolUsed:
157
+ - name: getOrder
158
+ status: true
159
+ arguments:
160
+ orderId: ORD-123
161
+ toolsInOrder:
162
+ - getOrder
163
+ - getShipment
164
+ expect:
165
+ conversation:
166
+ criteria: Keeps the order ID across turns.
167
+ ```
168
+
169
+ Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
170
+
171
+ | Expectation | Passes when |
172
+ |---|---|
173
+ | `response: <text>` | Shorthand for `response.criteria`. |
174
+ | `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
175
+ | `response.notContains` | No entry appears in the response text. |
176
+ | `response.criteria` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `criteria` together are one test. |
177
+ | `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
178
+ | `toolNotUsed` | No tool use in the turn has the name. Names only. |
179
+ | `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
180
+ | `expect.conversation.criteria` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
181
+
182
+ An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
183
+
184
+ The JSON Schemas in `schemas/` describe the configuration, scenarios and the JSON report, and can be used for editor completion.
185
+
186
+ ## Results
187
+
188
+ | Level | Statuses |
189
+ |---|---|
190
+ | Test, in one attempt | `PASS`, `FAIL` (the behaviour did not match), `ERROR` (it could not be checked: a missing capability, an evaluator failure), `SKIPPED` (not run because the turn or attempt stopped), `INTERRUPTED`. |
191
+ | Attempt | `INTERRUPTED`, else `FAIL` if any test failed, else `ERROR` if any test errored or was skipped, the attempt stopped on an error, or the host sent a payload after a turn had completed. Otherwise `PASS`. A failed close is recorded but changes nothing. |
192
+ | Test and scenario across attempts | `PASS` when at least `minPass` outcomes passed; `FLAKY` when that was reached but some failed; `ERROR` when it could still have been reached counting errors as passes; else `FAIL`. `INTERRUPTED` when any outcome was. |
193
+
194
+ A turn stops its attempt when the host reports an error (`statusCode >= 400` or an `error` body), the connection drops, the turn times out, or the host sends malformed data. Later turns are not sent and their tests are `SKIPPED`. The next attempt starts on a new connection.
195
+
196
+ ### Reports
197
+
198
+ The console prints one line per scenario as it finishes, in file order. A Problems section follows for each scenario that did not pass. It lists attempts that stopped and why, then each test that did not pass with its location, its counts across attempts and the details of each failing attempt, including evaluator scores and explanations. The report ends with:
199
+
200
+ - scenario and test counts;
201
+ - agent and evaluator token usage with coverage, such as `coverage 8/9, partial`, when not every call reported usage;
202
+ - the duration and the exit code.
203
+
204
+ `--verbose` adds every attempt's transcript. Status words are coloured on a terminal unless `NO_COLOR` is set.
205
+
206
+ `json` writes `miniwat-eval-report.json` to `report.directory`, replacing the previous one, and prints `Report: <path>` above the final line. It follows `schemas/report.schema.json` and holds:
207
+
208
+ - the run status, exit code and uncapped unsuccessful count;
209
+ - the effective settings and redacted configuration;
210
+ - every scenario with each test's status per attempt;
211
+ - every attempt's full transcript: inputs, responses, status messages, tool uses and complete tool results, token usage, errors, connection attempts, test details and evaluator verdicts.
212
+
213
+ Values that were not observed are left out rather than written as `null`.
214
+
215
+ ### Ctrl+C
216
+
217
+ The first Ctrl+C stops the run. No new scenario or attempt starts, active turns and evaluator calls are abandoned, and connections are closed. Unfinished work is `INTERRUPTED`, scenarios that never started are listed as `not started`, the requested reports are still written, and the exit code is 130. A second Ctrl+C ends the process at once.
218
+
219
+ ## Host requirements
220
+
221
+ Miniwat Eval uses `configurapi-handler-ws`'s `WebSocketClient` with reconnection off, and reads the payloads its `onPush` and `send()` callbacks deliver. A host must:
222
+
223
+ 1. **Sessions:** start a new conversation session for each new connection, and keep it for messages on that connection, without the client sending a session ID.
224
+ 2. **Turns:** accept the next turn after the previous one completed.
225
+ 3. **Replies:** send reply text as `statusCode: 200` payloads with a string body. They are joined exactly.
226
+ 4. **Completion:** end each successful turn with a `statusCode: 204` payload. Nothing else completes a turn.
227
+ 5. **Errors:** report a failed turn with `statusCode >= 400`, a `{ type: "error", message }` body, or a disconnect.
228
+ 6. **Optional payloads:** send any of these, with `statusCode: 200`, before the `204`:
229
+ - `{ type: "status", message }` for status messages;
230
+ - `{ type: "tooling", message }` carrying a `miniwat-agent` message with `toolUse` or `toolResult` entries;
231
+ - `{ type: "response", message }` whose `tokenUsage` is read.
232
+
233
+ Unknown body types and extra fields are ignored. A known type with a malformed body is an `ERROR`. Declare in `capabilities` which optional payloads the host sends.
234
+
235
+ ## Programmatic use
236
+
237
+ The package also exports its entities, loaders, runners and reporters. That API may change during 0.x; the CLI and file formats are the supported interface.
238
+
239
+ ## Limitations
240
+
241
+ - **No reconnection:** a dropped connection ends the attempt.
242
+ - **No HTML report:** it is planned for after V1. The JSON report holds everything a viewer would need.
243
+ - **No `.env` loading.**
244
+ - **Real side effects:** the agent's tool actions happen for real, once per repetition.
245
+
246
+ ## Licence
247
+
248
+ PolyForm Noncommercial 1.0.0