@mappies/miniwat-eval 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +131 -0
- package/README.md +247 -2
- package/dist/index.cjs +3453 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +941 -0
- package/dist/index.d.ts +941 -0
- package/dist/index.js +3343 -0
- package/dist/index.js.map +1 -0
- package/dist/main.js +3582 -0
- package/dist/main.js.map +1 -0
- package/package.json +76 -5
- package/schemas/conversation.schema.json +102 -0
- package/schemas/eval-config.schema.json +141 -0
- package/schemas/report.schema.json +326 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# PolyForm Noncommercial License 1.0.0
|
|
2
|
+
|
|
3
|
+
<https://polyformproject.org/licenses/noncommercial/1.0.0>
|
|
4
|
+
|
|
5
|
+
## Acceptance
|
|
6
|
+
|
|
7
|
+
In order to get any license under these terms, you must agree
|
|
8
|
+
to them as both strict obligations and conditions to all
|
|
9
|
+
your licenses.
|
|
10
|
+
|
|
11
|
+
## Copyright License
|
|
12
|
+
|
|
13
|
+
The licensor grants you a copyright license for the
|
|
14
|
+
software to do everything you might do with the software
|
|
15
|
+
that would otherwise infringe the licensor's copyright
|
|
16
|
+
in it for any permitted purpose. However, you may
|
|
17
|
+
only distribute the software according to [Distribution
|
|
18
|
+
License](#distribution-license) and make changes or new works
|
|
19
|
+
based on the software according to [Changes and New Works
|
|
20
|
+
License](#changes-and-new-works-license).
|
|
21
|
+
|
|
22
|
+
## Distribution License
|
|
23
|
+
|
|
24
|
+
The licensor grants you an additional copyright license
|
|
25
|
+
to distribute copies of the software. Your license
|
|
26
|
+
to distribute covers distributing the software with
|
|
27
|
+
changes and new works permitted by [Changes and New Works
|
|
28
|
+
License](#changes-and-new-works-license).
|
|
29
|
+
|
|
30
|
+
## Notices
|
|
31
|
+
|
|
32
|
+
You must ensure that anyone who gets a copy of any part of
|
|
33
|
+
the software from you also gets a copy of these terms or the
|
|
34
|
+
URL for them above, as well as copies of any plain-text lines
|
|
35
|
+
beginning with `Required Notice:` that the licensor provided
|
|
36
|
+
with the software. For example:
|
|
37
|
+
|
|
38
|
+
> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
|
|
39
|
+
|
|
40
|
+
## Changes and New Works License
|
|
41
|
+
|
|
42
|
+
The licensor grants you an additional copyright license to
|
|
43
|
+
make changes and new works based on the software for any
|
|
44
|
+
permitted purpose.
|
|
45
|
+
|
|
46
|
+
## Patent License
|
|
47
|
+
|
|
48
|
+
The licensor grants you a patent license for the software that
|
|
49
|
+
covers patent claims the licensor can license, or becomes able
|
|
50
|
+
to license, that you would infringe by using the software.
|
|
51
|
+
|
|
52
|
+
## Noncommercial Purposes
|
|
53
|
+
|
|
54
|
+
Any noncommercial purpose is a permitted purpose.
|
|
55
|
+
|
|
56
|
+
## Personal Uses
|
|
57
|
+
|
|
58
|
+
Personal use for research, experiment, and testing for
|
|
59
|
+
the benefit of public knowledge, personal study, private
|
|
60
|
+
entertainment, hobby projects, amateur pursuits, or religious
|
|
61
|
+
observance, without any anticipated commercial application,
|
|
62
|
+
is use for a permitted purpose.
|
|
63
|
+
|
|
64
|
+
## Noncommercial Organizations
|
|
65
|
+
|
|
66
|
+
Use by any charitable organization, educational institution,
|
|
67
|
+
public research organization, public safety or health
|
|
68
|
+
organization, environmental protection organization,
|
|
69
|
+
or government institution is use for a permitted purpose
|
|
70
|
+
regardless of the source of funding or obligations resulting
|
|
71
|
+
from the funding.
|
|
72
|
+
|
|
73
|
+
## Fair Use
|
|
74
|
+
|
|
75
|
+
You may have "fair use" rights for the software under the
|
|
76
|
+
law. These terms do not limit them.
|
|
77
|
+
|
|
78
|
+
## No Other Rights
|
|
79
|
+
|
|
80
|
+
These terms do not allow you to sublicense or transfer any of
|
|
81
|
+
your licenses to anyone else, or prevent the licensor from
|
|
82
|
+
granting licenses to anyone else. These terms do not imply
|
|
83
|
+
any other licenses.
|
|
84
|
+
|
|
85
|
+
## Patent Defense
|
|
86
|
+
|
|
87
|
+
If you make any written claim that the software infringes or
|
|
88
|
+
contributes to infringement of any patent, your patent license
|
|
89
|
+
for the software granted under these terms ends immediately. If
|
|
90
|
+
your company makes such a claim, your patent license ends
|
|
91
|
+
immediately for work on behalf of your company.
|
|
92
|
+
|
|
93
|
+
## Violations
|
|
94
|
+
|
|
95
|
+
The first time you are notified in writing that you have
|
|
96
|
+
violated any of these terms, or done anything with the software
|
|
97
|
+
not covered by your licenses, your licenses can nonetheless
|
|
98
|
+
continue if you come into full compliance with these terms,
|
|
99
|
+
and take practical steps to correct past violations, within
|
|
100
|
+
32 days of receiving notice. Otherwise, all your licenses
|
|
101
|
+
end immediately.
|
|
102
|
+
|
|
103
|
+
## No Liability
|
|
104
|
+
|
|
105
|
+
***As far as the law allows, the software comes as is, without
|
|
106
|
+
any warranty or condition, and the licensor will not be liable
|
|
107
|
+
to you for any damages arising out of these terms or the use
|
|
108
|
+
or nature of the software, under any kind of legal claim.***
|
|
109
|
+
|
|
110
|
+
## Definitions
|
|
111
|
+
|
|
112
|
+
The **licensor** is the individual or entity offering these
|
|
113
|
+
terms, and the **software** is the software the licensor makes
|
|
114
|
+
available under these terms.
|
|
115
|
+
|
|
116
|
+
**You** refers to the individual or entity agreeing to these
|
|
117
|
+
terms.
|
|
118
|
+
|
|
119
|
+
**Your company** is any legal entity, sole proprietorship,
|
|
120
|
+
or other kind of organization that you work for, plus all
|
|
121
|
+
organizations that have control over, are under the control of,
|
|
122
|
+
or are under common control with that organization. **Control**
|
|
123
|
+
means ownership of substantially all the assets of an entity,
|
|
124
|
+
or the power to direct its management and policies by vote,
|
|
125
|
+
contract, or otherwise. Control can be direct or indirect.
|
|
126
|
+
|
|
127
|
+
**Your licenses** are all the licenses granted to you for the
|
|
128
|
+
software under these terms.
|
|
129
|
+
|
|
130
|
+
**Use** means anything you do with the software requiring one
|
|
131
|
+
of your licenses.
|
package/README.md
CHANGED
|
@@ -1,3 +1,248 @@
|
|
|
1
|
-
#
|
|
1
|
+
# miniwat-eval
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Black-box, YAML-driven evaluation for agents built on `miniwat-agent`. You write a conversation and what you expect from it in YAML. Miniwat Eval sends the turns to the agent over WebSocket, records what comes back, and checks it with literal, tool and evaluator-scored assertions. It repeats conversations to detect flaky behaviour, and reports the results with a CI exit code.
|
|
4
|
+
|
|
5
|
+
It talks to the host only through its WebSocket, the way a production client does. Conversations are real: every repetition repeats every tool action the agent takes, so run it against test environments and test data.
|
|
6
|
+
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-openai
|
|
11
|
+
# or, for an Amazon Bedrock evaluator:
|
|
12
|
+
npm install --save-dev @mappies/miniwat-eval @mappies/miniwat-provider-bedrock
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The provider is only needed when a scenario has semantic criteria. Node.js 22 or later is required.
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
The repository's `examples/` folder has a working configuration and scenarios. From a clone, with an agent listening:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
cp examples/.env.example examples/.env # then fill it in
|
|
23
|
+
set -a; . examples/.env; set +a # miniwat-eval does not read .env files itself
|
|
24
|
+
npx miniwat-eval validate examples/scenarios
|
|
25
|
+
npx miniwat-eval run examples/scenarios --quick
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
`validate` checks everything with no network calls. `run` validates the same way, then runs each scenario.
|
|
29
|
+
|
|
30
|
+
## Commands
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
miniwat-eval validate <target> [--config <path>]
|
|
34
|
+
miniwat-eval run <target> [--config <path>] [--repeat <n>] [--min-pass <n>] [--quick] [--concurrency <n>] [--report <formats>] [--verbose]
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`<target>` is a scenario file, or a directory searched recursively for `*.yml` and `*.yaml`. The configuration file and the report directory are skipped.
|
|
38
|
+
|
|
39
|
+
| Option | Effect |
|
|
40
|
+
|---|---|
|
|
41
|
+
| `--config <path>` | The configuration file. Without it, `eval-config.yml` is searched for upward from the target, stopping at the nearest directory with `.git` or `package.json`, then in the working directory. |
|
|
42
|
+
| `--repeat <n>` | Overrides `execution.repeat`. |
|
|
43
|
+
| `--min-pass <n>` | Overrides `execution.minPass`. |
|
|
44
|
+
| `--quick` | One attempt that must pass: `repeat 1`, `minPass 1`. Wins over `--repeat` and `--min-pass`. |
|
|
45
|
+
| `--concurrency <n>` | Overrides `execution.scenarioConcurrency`. |
|
|
46
|
+
| `--report <formats>` | Comma-separated `console`, `json`. Replaces `report.formats`. |
|
|
47
|
+
| `--verbose` | Adds every attempt's transcript to the console report. |
|
|
48
|
+
|
|
49
|
+
Option values are positive base-10 integers; `0`, `2.5` and `3x` are rejected. A problem with the configuration, a scenario or an option is printed as `file:line:column path: message` where it has a location, and nothing runs.
|
|
50
|
+
|
|
51
|
+
### Exit codes
|
|
52
|
+
|
|
53
|
+
| Code | Meaning |
|
|
54
|
+
|---|---|
|
|
55
|
+
| 0 | Every scenario passed. FLAKY counts as passing unless `flakyPolicy: fail`. |
|
|
56
|
+
| 1 to 255 | The number of unsuccessful tests, capped at 255. Each test that is `FAIL` or `ERROR` across attempts counts once, and `FLAKY` counts with `flakyPolicy: fail`. A scenario that did not pass adds 1 when none of its tests counted. |
|
|
57
|
+
| 1 | Also: an invalid configuration, scenario or option, or a missing provider package. A report that could not be written raises the code to at least 1. |
|
|
58
|
+
| 130 | Interrupted by Ctrl+C, whatever the results. |
|
|
59
|
+
|
|
60
|
+
The JSON report records the exact count and status, since 1 and 130 can mean more than one thing.
|
|
61
|
+
|
|
62
|
+
## Configuration
|
|
63
|
+
|
|
64
|
+
Each project has one `eval-config.yml`. Paths inside it resolve against its own directory. `examples/eval-config.yml` shows every key with a comment.
|
|
65
|
+
|
|
66
|
+
```yaml
|
|
67
|
+
version: 1
|
|
68
|
+
|
|
69
|
+
websocket:
|
|
70
|
+
endpoint: ${MINIWAT_AGENT_ENDPOINT}
|
|
71
|
+
connection:
|
|
72
|
+
subprotocols:
|
|
73
|
+
- ${AGENT_ACCESS_TOKEN}
|
|
74
|
+
protocol:
|
|
75
|
+
request:
|
|
76
|
+
value:
|
|
77
|
+
name: add_session_message
|
|
78
|
+
payload: {}
|
|
79
|
+
inputPath: /payload/message
|
|
80
|
+
capabilities:
|
|
81
|
+
toolUses: true
|
|
82
|
+
toolResults: true
|
|
83
|
+
|
|
84
|
+
evaluator:
|
|
85
|
+
provider: openai
|
|
86
|
+
model: ${EVALUATOR_MODEL}
|
|
87
|
+
apiKey: ${OPENAI_API_KEY}
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
| Key | Default | Notes |
|
|
91
|
+
|---|---|---|
|
|
92
|
+
| `version` | required | `1`. |
|
|
93
|
+
| `websocket.endpoint` | required | A `ws://` or `wss://` URL. |
|
|
94
|
+
| `websocket.connection.subprotocols` | `[]` | WebSocket subprotocols sent on connect, for example an access token. |
|
|
95
|
+
| `websocket.protocol.request.format` | `json` | The only format. |
|
|
96
|
+
| `websocket.protocol.request.value` | required | The request template, an object or array. |
|
|
97
|
+
| `websocket.protocol.request.inputPath` | required | A JSON Pointer such as `/payload/message`. Each turn's input is written there in a copy of `value`. Its parent must exist in `value`. |
|
|
98
|
+
| `websocket.protocol.capabilities.toolUses` | `false` | The host sends tool uses. Needed by `toolUsed`, `toolNotUsed` and `toolsInOrder`. |
|
|
99
|
+
| `websocket.protocol.capabilities.toolResults` | `false` | The host sends tool results. Needed by `toolUsed` entries with `status`. |
|
|
100
|
+
| `websocket.protocol.capabilities.tokenUsage` | `false` | The host sends token usage. Without it, agent tokens show as not supported. |
|
|
101
|
+
| `execution.scenarioConcurrency` | 3 | Scenarios run at once. Attempts and turns are always sequential. |
|
|
102
|
+
| `execution.repeat` | 1 | Attempts per scenario. All of them run. |
|
|
103
|
+
| `execution.minPass` | 1 | Passing attempts a scenario needs, from 1 to `repeat`. |
|
|
104
|
+
| `execution.continueOnFailure` | `true` | `true`: every turn runs and semantic checks run after the conversation. `false`: each turn's checks run as it completes, and the attempt stops at the first `FAIL`. |
|
|
105
|
+
| `execution.flakyPolicy` | `warn` | `fail` makes a `FLAKY` scenario count towards the exit code. |
|
|
106
|
+
| `execution.connectionRetries` | 2 | Retries of a failed connect, only before the first turn, with backoff from 250 ms doubling. |
|
|
107
|
+
| `execution.timeouts.connection` | 10000 | Milliseconds from connect to ready. |
|
|
108
|
+
| `execution.timeouts.turn` | 60000 | Milliseconds from sending a turn to its completion. |
|
|
109
|
+
| `execution.timeouts.conversation` | 180000 | Milliseconds for one attempt's connection, from connect to close. Semantic checks after close are not counted. |
|
|
110
|
+
| `evaluator.provider` | required | `openai` or `bedrock`. |
|
|
111
|
+
| `evaluator.model` | required | The provider's model ID. |
|
|
112
|
+
| `evaluator.apiKey` | OpenAI only | Must be an environment reference. |
|
|
113
|
+
| `evaluator.region` | Bedrock only | Credentials come from the AWS credential chain, for example `AWS_PROFILE`. |
|
|
114
|
+
| `evaluator.concurrency` | 3 | Evaluator calls at once, across all scenarios. |
|
|
115
|
+
| `evaluator.timeout` | 60000 | Milliseconds per call. |
|
|
116
|
+
| `evaluator.retries` | 2 | Retries of a call that failed, timed out or returned invalid output. A valid low score is not retried. |
|
|
117
|
+
| `evaluator.maxToolResultChars` | 4000 | In the evaluator's prompt, the current turn's tool result content is cut to this length; earlier turns' results keep only their status. `0` means no cut. Reports keep the full content. |
|
|
118
|
+
| `evaluator.thresholds.response` | 0.85 | A response score at or above this passes. |
|
|
119
|
+
| `evaluator.thresholds.conversation` | 0.85 | The same, for `expect.conversation`. |
|
|
120
|
+
| `report.directory` | `./reports` | Where report files are written. |
|
|
121
|
+
| `report.formats` | `[console]` | `console` and `json`. |
|
|
122
|
+
|
|
123
|
+
Unknown keys are errors. The `evaluator` section is required, but its provider package is only loaded when a scenario has semantic criteria.
|
|
124
|
+
|
|
125
|
+
### Environment references
|
|
126
|
+
|
|
127
|
+
`${NAME}` in any string value, in the configuration or a scenario, is replaced with the environment variable `NAME` after the YAML is parsed, so a value containing YAML syntax stays one string. A missing variable is an error at its location; its value never appears in messages. `.env` files are not loaded.
|
|
128
|
+
|
|
129
|
+
### Redaction
|
|
130
|
+
|
|
131
|
+
Secrets are replaced with `[REDACTED]` in everything `run` prints and in report files. These are:
|
|
132
|
+
|
|
133
|
+
- the values of `evaluator.apiKey` and `websocket.connection.subprotocols`;
|
|
134
|
+
- the value of every `${NAME}` reference, in the configuration or a scenario, whose name contains `TOKEN`, `SECRET`, `PASSWORD` or `KEY`, in any case.
|
|
135
|
+
|
|
136
|
+
Only exact values are replaced, so text that does not contain a secret is never changed. Every occurrence is replaced, so a very short value, such as a one-letter placeholder, also changes ordinary words that contain it. The evaluator model still receives the agent's real responses.
|
|
137
|
+
|
|
138
|
+
## Scenarios
|
|
139
|
+
|
|
140
|
+
```yaml
|
|
141
|
+
version: 1
|
|
142
|
+
name: Order lookup
|
|
143
|
+
description: Optional.
|
|
144
|
+
turns:
|
|
145
|
+
- input: Where is my order?
|
|
146
|
+
expect:
|
|
147
|
+
response: Asks the user for their order ID.
|
|
148
|
+
toolNotUsed:
|
|
149
|
+
- getOrder
|
|
150
|
+
- input: My order ID is ORD-123.
|
|
151
|
+
expect:
|
|
152
|
+
response:
|
|
153
|
+
contains: [ORD-123]
|
|
154
|
+
notContains: [error]
|
|
155
|
+
criteria: Explains the delivery status.
|
|
156
|
+
toolUsed:
|
|
157
|
+
- name: getOrder
|
|
158
|
+
status: true
|
|
159
|
+
arguments:
|
|
160
|
+
orderId: ORD-123
|
|
161
|
+
toolsInOrder:
|
|
162
|
+
- getOrder
|
|
163
|
+
- getShipment
|
|
164
|
+
expect:
|
|
165
|
+
conversation:
|
|
166
|
+
criteria: Keeps the order ID across turns.
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Turns are sent in order on one connection, which is one host session. Each expectation is one test, reported with its YAML location, for example `orders.yml:16:11 turns[1].expect.toolUsed[0]`.
|
|
170
|
+
|
|
171
|
+
| Expectation | Passes when |
|
|
172
|
+
|---|---|
|
|
173
|
+
| `response: <text>` | Shorthand for `response.criteria`. |
|
|
174
|
+
| `response.contains` | Every entry appears in the response text. Literal and case-sensitive, with no normalisation. Text split across streamed chunks matches. |
|
|
175
|
+
| `response.notContains` | No entry appears in the response text. |
|
|
176
|
+
| `response.criteria` | The evaluator scores the response at or above `thresholds.response`, given the conversation so far. It is only called when `contains` and `notContains` pass. `contains`, `notContains` and `criteria` together are one test. |
|
|
177
|
+
| `toolUsed` | Each entry is matched to a different tool use in the same turn with that name. `arguments`, when given, must equal the tool input exactly: same keys, same types, arrays in order, `{}` only matches `{}`. `status: true` or `false` must equal the tool result's status. A string entry is name only. Entries are matched together, so overlapping entries are not lost to a greedy match. A failure shows the closest candidate and how its arguments differ. |
|
|
178
|
+
| `toolNotUsed` | No tool use in the turn has the name. Names only. |
|
|
179
|
+
| `toolsInOrder` | The entries appear as tool uses in this order within the turn, with anything in between. Entries may have `arguments`, matched as for `toolUsed`. It is one test. |
|
|
180
|
+
| `expect.conversation.criteria` | The evaluator scores the whole conversation at or above `thresholds.conversation`. It runs only when every turn completed; otherwise it is `SKIPPED`. |
|
|
181
|
+
|
|
182
|
+
An expectation whose capability is off is `ERROR`, not `PASS`. Tool expectations look only at their own turn. To test an approval, assert the proposed call with `status: false` on one turn and the approved call with `status: true` on the next (`examples/scenarios/approval-flow.yml`).
|
|
183
|
+
|
|
184
|
+
The JSON Schemas in `schemas/` describe the configuration, scenarios and the JSON report, and can be used for editor completion.
|
|
185
|
+
|
|
186
|
+
## Results
|
|
187
|
+
|
|
188
|
+
| Level | Statuses |
|
|
189
|
+
|---|---|
|
|
190
|
+
| Test, in one attempt | `PASS`, `FAIL` (the behaviour did not match), `ERROR` (it could not be checked: a missing capability, an evaluator failure), `SKIPPED` (not run because the turn or attempt stopped), `INTERRUPTED`. |
|
|
191
|
+
| Attempt | `INTERRUPTED`, else `FAIL` if any test failed, else `ERROR` if any test errored or was skipped, the attempt stopped on an error, or the host sent a payload after a turn had completed. Otherwise `PASS`. A failed close is recorded but changes nothing. |
|
|
192
|
+
| Test and scenario across attempts | `PASS` when at least `minPass` outcomes passed; `FLAKY` when that was reached but some failed; `ERROR` when it could still have been reached counting errors as passes; else `FAIL`. `INTERRUPTED` when any outcome was. |
|
|
193
|
+
|
|
194
|
+
A turn stops its attempt when the host reports an error (`statusCode >= 400` or an `error` body), the connection drops, the turn times out, or the host sends malformed data. Later turns are not sent and their tests are `SKIPPED`. The next attempt starts on a new connection.
|
|
195
|
+
|
|
196
|
+
### Reports
|
|
197
|
+
|
|
198
|
+
The console prints one line per scenario as it finishes, in file order. A Problems section follows for each scenario that did not pass. It lists attempts that stopped and why, then each test that did not pass with its location, its counts across attempts and the details of each failing attempt, including evaluator scores and explanations. The report ends with:
|
|
199
|
+
|
|
200
|
+
- scenario and test counts;
|
|
201
|
+
- agent and evaluator token usage with coverage, such as `coverage 8/9, partial`, when not every call reported usage;
|
|
202
|
+
- the duration and the exit code.
|
|
203
|
+
|
|
204
|
+
`--verbose` adds every attempt's transcript. Status words are coloured on a terminal unless `NO_COLOR` is set.
|
|
205
|
+
|
|
206
|
+
`json` writes `miniwat-eval-report.json` to `report.directory`, replacing the previous one, and prints `Report: <path>` above the final line. It follows `schemas/report.schema.json` and holds:
|
|
207
|
+
|
|
208
|
+
- the run status, exit code and uncapped unsuccessful count;
|
|
209
|
+
- the effective settings and redacted configuration;
|
|
210
|
+
- every scenario with each test's status per attempt;
|
|
211
|
+
- every attempt's full transcript: inputs, responses, status messages, tool uses and complete tool results, token usage, errors, connection attempts, test details and evaluator verdicts.
|
|
212
|
+
|
|
213
|
+
Values that were not observed are left out rather than written as `null`.
|
|
214
|
+
|
|
215
|
+
### Ctrl+C
|
|
216
|
+
|
|
217
|
+
The first Ctrl+C stops the run. No new scenario or attempt starts, active turns and evaluator calls are abandoned, and connections are closed. Unfinished work is `INTERRUPTED`, scenarios that never started are listed as `not started`, the requested reports are still written, and the exit code is 130. A second Ctrl+C ends the process at once.
|
|
218
|
+
|
|
219
|
+
## Host requirements
|
|
220
|
+
|
|
221
|
+
Miniwat Eval uses `configurapi-handler-ws`'s `WebSocketClient` with reconnection off, and reads the payloads its `onPush` and `send()` callbacks deliver. A host must:
|
|
222
|
+
|
|
223
|
+
1. **Sessions:** start a new conversation session for each new connection, and keep it for messages on that connection, without the client sending a session ID.
|
|
224
|
+
2. **Turns:** accept the next turn after the previous one completed.
|
|
225
|
+
3. **Replies:** send reply text as `statusCode: 200` payloads with a string body. They are joined exactly.
|
|
226
|
+
4. **Completion:** end each successful turn with a `statusCode: 204` payload. Nothing else completes a turn.
|
|
227
|
+
5. **Errors:** report a failed turn with `statusCode >= 400`, a `{ type: "error", message }` body, or a disconnect.
|
|
228
|
+
6. **Optional payloads:** send any of these, with `statusCode: 200`, before the `204`:
|
|
229
|
+
- `{ type: "status", message }` for status messages;
|
|
230
|
+
- `{ type: "tooling", message }` carrying a `miniwat-agent` message with `toolUse` or `toolResult` entries;
|
|
231
|
+
- `{ type: "response", message }` whose `tokenUsage` is read.
|
|
232
|
+
|
|
233
|
+
Unknown body types and extra fields are ignored. A known type with a malformed body is an `ERROR`. Declare in `capabilities` which optional payloads the host sends.
|
|
234
|
+
|
|
235
|
+
## Programmatic use
|
|
236
|
+
|
|
237
|
+
The package also exports its entities, loaders, runners and reporters. That API may change during 0.x; the CLI and file formats are the supported interface.
|
|
238
|
+
|
|
239
|
+
## Limitations
|
|
240
|
+
|
|
241
|
+
- **No reconnection:** a dropped connection ends the attempt.
|
|
242
|
+
- **No HTML report:** it is planned for after V1. The JSON report holds everything a viewer would need.
|
|
243
|
+
- **No `.env` loading.**
|
|
244
|
+
- **Real side effects:** the agent's tool actions happen for real, once per repetition.
|
|
245
|
+
|
|
246
|
+
## Licence
|
|
247
|
+
|
|
248
|
+
PolyForm Noncommercial 1.0.0
|