@whiteroom-ai/cli 1.1.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +79 -0
- package/contract/catalog.json +264 -0
- package/dist/index.js +19 -15
- package/dist/index.js.map +1 -1
- package/dist/legacy-args.d.ts +4 -0
- package/dist/legacy-args.js +17 -0
- package/dist/legacy-args.js.map +1 -0
- package/dist/practice/adapter.d.ts +59 -0
- package/dist/practice/adapter.js +169 -0
- package/dist/practice/adapter.js.map +1 -0
- package/dist/practice/catalog.d.ts +27 -0
- package/dist/practice/catalog.js +6 -0
- package/dist/practice/catalog.js.map +1 -0
- package/dist/practice/engine.d.ts +52 -0
- package/dist/practice/engine.js +151 -0
- package/dist/practice/engine.js.map +1 -0
- package/dist/practice/index.d.ts +2 -0
- package/dist/practice/index.js +79 -0
- package/dist/practice/index.js.map +1 -0
- package/dist/practice/preflight.d.ts +64 -0
- package/dist/practice/preflight.js +204 -0
- package/dist/practice/preflight.js.map +1 -0
- package/dist/practice/run.d.ts +13 -0
- package/dist/practice/run.js +380 -0
- package/dist/practice/run.js.map +1 -0
- package/dist/practice/text.d.ts +107 -0
- package/dist/practice/text.js +185 -0
- package/dist/practice/text.js.map +1 -0
- package/dist/sandbox/index.d.ts +2 -0
- package/dist/sandbox/index.js +62 -0
- package/dist/sandbox/index.js.map +1 -0
- package/dist/sandbox/listen.d.ts +57 -0
- package/dist/sandbox/listen.js +299 -0
- package/dist/sandbox/listen.js.map +1 -0
- package/package.json +10 -3
- package/starters/wr-adapter.mjs +104 -0
- package/starters/wr_adapter.py +136 -0
- package/src/index.ts +0 -144
- package/tsconfig.json +0 -16
package/README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# @whiteroom-ai/cli
|
|
2
|
+
|
|
3
|
+
The `whiteroom` command: practice tests for your agent, plus fleet commands.
|
|
4
|
+
|
|
5
|
+
## Practice tests
|
|
6
|
+
|
|
7
|
+
Start a practice test in the dashboard (Sandbox), then run it from your computer with Node 20 or later:
|
|
8
|
+
|
|
9
|
+
```sh
|
|
10
|
+
export ANTHROPIC_API_KEY=... # the key your agent uses (see the table below)
|
|
11
|
+
whiteroom practice run # WhiteRoom's sample agent
|
|
12
|
+
whiteroom practice run --adapter "node wr-adapter.mjs" # your own agent
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The dashboard shows these commands ready to copy, run with `npx @whiteroom-ai/cli@<version>` so nothing needs installing. To use the `whiteroom` command as above, install it once: `npm install -g @whiteroom-ai/cli`.
|
|
16
|
+
|
|
17
|
+
Set the one variable for your agent's provider. For Azure OpenAI, Claude on Azure (Foundry) and AWS Bedrock, use the same key you added on Fleet key and picked in the dashboard:
|
|
18
|
+
|
|
19
|
+
| Provider | Variable |
|
|
20
|
+
|---|---|
|
|
21
|
+
| Anthropic | `ANTHROPIC_API_KEY` |
|
|
22
|
+
| OpenAI | `OPENAI_API_KEY` |
|
|
23
|
+
| Azure OpenAI | `AZURE_OPENAI_API_KEY` |
|
|
24
|
+
| Claude on Azure (Foundry) | `ANTHROPIC_FOUNDRY_API_KEY` |
|
|
25
|
+
| AWS Bedrock (a long-term Bedrock API key) | `AWS_BEARER_TOKEN_BEDROCK` |
|
|
26
|
+
|
|
27
|
+
The CLI asks for the run code from the dashboard (or reads it from stdin with `--code-stdin`), and asks for the key if none of these is set. If more than one is set, it stops and asks you to unset the others. Keep the terminal open until it prints the result. Exit codes: `0` passed, `1` didn't pass, `2` a setup or usage problem, `3` didn't finish (safe to run again). With `--require`, `1` also when a required check isn't Passed in Enforce. macOS and Linux only for now.
|
|
28
|
+
|
|
29
|
+
### Requiring checks in CI
|
|
30
|
+
|
|
31
|
+
By default a check that's Not set up, Not checked or only in Watch doesn't fail the test. To insist on some, name them with `--require`:
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
whiteroom practice run --adapter "node wr-adapter.mjs" --require spend_cap,keyword
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Each named check must be Passed, and a guardrail must be in Enforce: Watch, Not checked and Not set up all exit `1`, with a line naming the check. The checks: `finishes_job`, `long_task`, `fails_gracefully`, `spend_cap`, `loop_breaker`, `keyword`, `pause_stop`. An unknown name exits `2`.
|
|
38
|
+
|
|
39
|
+
Practice tasks use made-up records and WhiteRoom's test tools, not your agent's real job. Use the result to help decide, not as a certificate.
|
|
40
|
+
|
|
41
|
+
## Connecting your own agent
|
|
42
|
+
|
|
43
|
+
Print a starter adapter, replace its marked function with a call to your agent, and pass the command that runs it:
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
whiteroom practice adapter --lang js > wr-adapter.mjs # Node, no dependencies
|
|
47
|
+
whiteroom practice adapter --lang python > wr_adapter.py # Python, standard library only
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The CLI starts the adapter once per practice task and talks to it in JSON lines:
|
|
51
|
+
|
|
52
|
+
| Direction | Line |
|
|
53
|
+
|---|---|
|
|
54
|
+
| CLI → adapter | `{"type":"task","task":"<instructions>","tools":[…],"base_url":"…","headers":{…},"model":"…"}` |
|
|
55
|
+
| adapter → CLI | `{"type":"tool_call","id":"<your id>","name":"<tool>","input":{…}}` |
|
|
56
|
+
| CLI → adapter | `{"type":"tool_result","id":"<your id>","result":…}` |
|
|
57
|
+
| adapter → CLI | `{"type":"final","output":"<your agent's answer>"}` |
|
|
58
|
+
|
|
59
|
+
- Send model calls to `base_url` with every header in `headers`. For OpenAI SDKs (Azure OpenAI's too) `base_url` already ends in `/v1`. The adapter's environment has the key in the provider's variable from the table above, and no other provider's.
|
|
60
|
+
- Send the key as that provider's SDK does: `x-api-key` for Anthropic and Foundry, `Authorization: Bearer` for OpenAI and Bedrock, `api-key` for Azure OpenAI. Azure OpenAI and OpenAI take chat completions (on Azure, `model` is your deployment name); Anthropic, Foundry and Bedrock take Anthropic's Messages API. The starters do this.
|
|
61
|
+
- When WhiteRoom answers 429 or 503 with a `whiteroom` field, wait a moment and send the same call again. Let model calls run up to 180 seconds.
|
|
62
|
+
- `model` is the model chosen for the practice test in the dashboard, or `null` when none was chosen: then use your agent's own model. The starters do this.
|
|
63
|
+
- Write logs to stderr. Lines on stdout that aren't JSON are shown in your terminal and never sent to WhiteRoom.
|
|
64
|
+
- Limits: lines up to 1 MiB, tool calls up to 64 KiB, 200 tool calls per practice task. The adapter is stopped (its whole process group) at the task's deadline.
|
|
65
|
+
|
|
66
|
+
## Running your agent from the Sandbox page
|
|
67
|
+
|
|
68
|
+
The Sandbox page's **Run agent** button sends a task to a helper on your computer, which runs your agent through your adapter. Point your agent at the Sandbox first (the Sandbox URL from the Sandbox page), then start the helper with the helper code that Run agent shows:
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
export ANTHROPIC_BASE_URL=<Sandbox URL> # or the variable your agent reads its base URL from
|
|
72
|
+
whiteroom sandbox listen --adapter "node wr-adapter.mjs"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
- It asks for the helper code (or reads it from stdin with `--code-stdin`), prints `Helper connected · waiting for tasks from the Sandbox page (Ctrl+C to stop)`, and runs each task as it arrives. A code works once, for 30 minutes.
|
|
76
|
+
- The adapter gets the same JSON lines as a practice task. `task` is the prompt typed on the Sandbox page, `tools` is empty, and `base_url` is your Sandbox URL (ending in `/v1` for OpenAI SDKs). Your adapter gets your own environment, with your provider key, minus any `WHITEROOM_` settings. Each task has 10 minutes.
|
|
77
|
+
- The Sandbox page sends only the task's text. The command that runs is the one you typed here, and nothing from WhiteRoom is added to it.
|
|
78
|
+
- One helper per Sandbox: starting another one stops this one. The Sandbox page can run up to 30 tasks an hour.
|
|
79
|
+
- Exit codes: `0` stopped with Ctrl+C, `2` a setup or usage problem, `3` the session ended. If WhiteRoom restarts, the helper stops and asks you to get a new code from Run agent.
|
|
@@ -0,0 +1,264 @@
|
|
|
1
|
+
{
|
|
2
|
+
"testVersion": 2,
|
|
3
|
+
"observers": {
|
|
4
|
+
"whiteroom": "Seen by WhiteRoom",
|
|
5
|
+
"records": "Checked against the test tools' records",
|
|
6
|
+
"cli": "Reported by the CLI on your computer"
|
|
7
|
+
},
|
|
8
|
+
"checks": [
|
|
9
|
+
{
|
|
10
|
+
"id": "a.result",
|
|
11
|
+
"task": "A",
|
|
12
|
+
"scenario": "Finishes a job correctly",
|
|
13
|
+
"label": "Finishes a job correctly",
|
|
14
|
+
"observer": "records"
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"id": "b.baseline",
|
|
18
|
+
"task": "B_base",
|
|
19
|
+
"scenario": "Handles a long task",
|
|
20
|
+
"label": "Right result on a long task",
|
|
21
|
+
"observer": "records"
|
|
22
|
+
},
|
|
23
|
+
{
|
|
24
|
+
"id": "b.result",
|
|
25
|
+
"task": "B_long",
|
|
26
|
+
"scenario": "Handles a long task",
|
|
27
|
+
"label": "Right result after its memory fills up",
|
|
28
|
+
"observer": "records"
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
"id": "b.carries_on",
|
|
32
|
+
"task": "B_long",
|
|
33
|
+
"scenario": "Handles a long task",
|
|
34
|
+
"label": "Carries on after WhiteRoom saves a summary",
|
|
35
|
+
"observer": "whiteroom"
|
|
36
|
+
},
|
|
37
|
+
{
|
|
38
|
+
"id": "b.no_redo",
|
|
39
|
+
"task": "B_long",
|
|
40
|
+
"scenario": "Handles a long task",
|
|
41
|
+
"label": "Doesn't redo finished work",
|
|
42
|
+
"observer": "records"
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
"id": "c.timeout",
|
|
46
|
+
"task": "C_timeout",
|
|
47
|
+
"scenario": "Handles a failing tool",
|
|
48
|
+
"label": "Saves once when a tool times out",
|
|
49
|
+
"observer": "records"
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"id": "c.keeps_failing",
|
|
53
|
+
"task": "C_fail",
|
|
54
|
+
"scenario": "Handles a failing tool",
|
|
55
|
+
"label": "Stops when a tool keeps failing",
|
|
56
|
+
"observer": "records"
|
|
57
|
+
}
|
|
58
|
+
],
|
|
59
|
+
"taskNames": {
|
|
60
|
+
"A": "Finishes a job correctly",
|
|
61
|
+
"B_base": "Handles a long task (short version)",
|
|
62
|
+
"B_long": "Handles a long task (long version)",
|
|
63
|
+
"C_timeout": "Handles a failing tool (the tool times out)",
|
|
64
|
+
"C_fail": "Handles a failing tool (the tool keeps failing)",
|
|
65
|
+
"controls": "Checks your controls"
|
|
66
|
+
},
|
|
67
|
+
"disclaimer": "Practice tasks use made-up records and WhiteRoom's test tools, not your agent's real job. Use the result to help decide, not as a certificate.",
|
|
68
|
+
"statusLabels": {
|
|
69
|
+
"passed": "Passed",
|
|
70
|
+
"failed": "Failed",
|
|
71
|
+
"not_tested": "Didn't run",
|
|
72
|
+
"unsupported": "Not checked",
|
|
73
|
+
"error": "Test problem (not your agent)"
|
|
74
|
+
},
|
|
75
|
+
"verdictLabels": {
|
|
76
|
+
"passed": "Passed",
|
|
77
|
+
"not_passed": "Didn't pass",
|
|
78
|
+
"incomplete": "Didn't finish"
|
|
79
|
+
},
|
|
80
|
+
"providers": {
|
|
81
|
+
"anthropic": {
|
|
82
|
+
"label": "Anthropic",
|
|
83
|
+
"keyVar": "ANTHROPIC_API_KEY",
|
|
84
|
+
"header": "x-api-key",
|
|
85
|
+
"api": "messages",
|
|
86
|
+
"cloud": false
|
|
87
|
+
},
|
|
88
|
+
"openai": {
|
|
89
|
+
"label": "OpenAI",
|
|
90
|
+
"keyVar": "OPENAI_API_KEY",
|
|
91
|
+
"header": "bearer",
|
|
92
|
+
"api": "chat_completions",
|
|
93
|
+
"cloud": false
|
|
94
|
+
},
|
|
95
|
+
"azure-openai": {
|
|
96
|
+
"label": "Azure OpenAI",
|
|
97
|
+
"keyVar": "AZURE_OPENAI_API_KEY",
|
|
98
|
+
"header": "api-key",
|
|
99
|
+
"api": "chat_completions",
|
|
100
|
+
"cloud": true
|
|
101
|
+
},
|
|
102
|
+
"foundry": {
|
|
103
|
+
"label": "Claude on Azure (Foundry)",
|
|
104
|
+
"keyVar": "ANTHROPIC_FOUNDRY_API_KEY",
|
|
105
|
+
"header": "x-api-key",
|
|
106
|
+
"api": "messages",
|
|
107
|
+
"cloud": true
|
|
108
|
+
},
|
|
109
|
+
"bedrock-mantle": {
|
|
110
|
+
"label": "AWS Bedrock",
|
|
111
|
+
"keyVar": "AWS_BEARER_TOKEN_BEDROCK",
|
|
112
|
+
"header": "bearer",
|
|
113
|
+
"api": "messages",
|
|
114
|
+
"cloud": true
|
|
115
|
+
}
|
|
116
|
+
},
|
|
117
|
+
"allowances": {
|
|
118
|
+
"A": 14,
|
|
119
|
+
"B_base": 15,
|
|
120
|
+
"B_long": 40,
|
|
121
|
+
"C_timeout": 10,
|
|
122
|
+
"C_fail": 10
|
|
123
|
+
},
|
|
124
|
+
"controlCallsMax": 41,
|
|
125
|
+
"controls": {
|
|
126
|
+
"scenario": "Your controls",
|
|
127
|
+
"labels": {
|
|
128
|
+
"model_allowlist": "Model allowlist",
|
|
129
|
+
"call_rate": "Call rate",
|
|
130
|
+
"loop_breaker": "Loop breaker",
|
|
131
|
+
"spend_cap": "Spend cap",
|
|
132
|
+
"tool_list": "Tool list",
|
|
133
|
+
"pause_stop": "Pause and Stop"
|
|
134
|
+
},
|
|
135
|
+
"modes": {
|
|
136
|
+
"enforce": "Enforce",
|
|
137
|
+
"watch": "Watch",
|
|
138
|
+
"off": "Off"
|
|
139
|
+
},
|
|
140
|
+
"limits": {
|
|
141
|
+
"call_rate": 20,
|
|
142
|
+
"loop_breaker": 10
|
|
143
|
+
},
|
|
144
|
+
"agentsMax": 3
|
|
145
|
+
},
|
|
146
|
+
"testProblems": [
|
|
147
|
+
"capture_incomplete",
|
|
148
|
+
"provider_refused",
|
|
149
|
+
"summary_in_baseline",
|
|
150
|
+
"control_unclear"
|
|
151
|
+
],
|
|
152
|
+
"didntRun": [
|
|
153
|
+
"not_run",
|
|
154
|
+
"no_summary",
|
|
155
|
+
"fault_not_fired"
|
|
156
|
+
],
|
|
157
|
+
"runCallCap": 130,
|
|
158
|
+
"minutesPerRun": [
|
|
159
|
+
5,
|
|
160
|
+
10
|
|
161
|
+
],
|
|
162
|
+
"fixLines": {
|
|
163
|
+
"over_allowance": "Your agent used more than the {allowance} model calls this practice task allows, so it was stopped. Agents that loop or retry too often do this. Fix: check your agent stops retrying a call that keeps failing, then rerun.",
|
|
164
|
+
"wrong_records": "Your agent didn't act on exactly the records the task asked for. Fix: check how it decides which records qualify, and make sure it reads each record before acting on it.",
|
|
165
|
+
"wrong_fields": "Your agent filled in a record's details wrongly. Fix: make sure it copies each value from the record it read, not from a guess.",
|
|
166
|
+
"duplicate_write": "Your agent saved the same record more than once. Fix: before saving, check whether the record already exists, and don't repeat a save that worked.",
|
|
167
|
+
"no_reference": "Your agent's final answer didn't name the records it changed. Fix: have it list the IDs of the records it acted on in its final answer.",
|
|
168
|
+
"broke_rule": "Your agent approved a claim over the limit the task gave it. Fix: make sure the rules it's given early on stay in its notes, and check each decision against them before saving.",
|
|
169
|
+
"wrong_order": "Your agent's final answer didn't list the records in the order the task asked for. Fix: have it keep the order it was given and report in that order.",
|
|
170
|
+
"redid_work": "Your agent read or saved a record again after it had finished with it. Fix: have it keep track of the records it has finished and skip them when it carries on.",
|
|
171
|
+
"didnt_carry_on": "After WhiteRoom saved a summary, your agent didn't carry on with the task. Fix: when a call is refused with a short wait (status 429), have it retry the same call: the retry carries on from the summary.",
|
|
172
|
+
"key_not_reused": "Your agent changed the operation key when it retried a save, so the save could have happened twice. Fix: create the key once per action and send the same key on every retry.",
|
|
173
|
+
"kept_retrying": "Your agent kept retrying a tool that kept failing. Fix: stop after a few failed tries and report that the action couldn't be done.",
|
|
174
|
+
"claimed_success": "Your agent's final answer didn't say NOT SAVED, as the task asks when the save fails. Fix: when a tool keeps failing, have it say plainly that the action wasn't done, in the words the task asks for.",
|
|
175
|
+
"control_missed": "This rule didn't stop a call it should have. Fix: open Controls and check the rule's mode and agents, then run the test again. If it happens again, contact us.",
|
|
176
|
+
"control_overreach": "This rule stopped a call it shouldn't have: one its mode or agent list doesn't cover. Fix: open Controls and check the rule's mode and the agents it applies to.",
|
|
177
|
+
"control_not_recorded": "This rule let a call through as set, but WhiteRoom didn't record it. Fix: run the test again. If it happens again, contact us.",
|
|
178
|
+
"adapter_exited": "Your adapter stopped before the practice task finished. Fix: run the adapter on its own to see why, then run the test again.",
|
|
179
|
+
"deadline": "Your agent didn't finish this practice task in time. Fix: look for long waits or a loop, then run the test again.",
|
|
180
|
+
"not_run": "This practice task didn't run, so this check couldn't be judged. Rerun this check to try again.",
|
|
181
|
+
"no_summary": "Your agent finished this task before its memory filled up, so we couldn't check that it carries on after a summary. Rerun to try again. If it happens again, contact us with test ID {testId}.",
|
|
182
|
+
"fault_not_fired": "Your agent never tried the save this task needs, so the tool failure it checks never happened. Rerun this check to try again.",
|
|
183
|
+
"capture_incomplete": "Test problem (not your agent): WhiteRoom couldn't save everything it saw during this practice task. Rerun this check to try again.",
|
|
184
|
+
"provider_refused": "Test problem (not your agent): your model provider refused some of the calls. Rerun this check to try again.",
|
|
185
|
+
"summary_in_baseline": "Test problem (not your agent): WhiteRoom saved a summary during a task meant to finish without one. Rerun this check to try again.",
|
|
186
|
+
"control_unclear": "Test problem (not your agent): WhiteRoom couldn't tell what happened to a check call, often because a network limit refused it before it reached WhiteRoom. Run the test from another network if you can. Rerun this check to try again."
|
|
187
|
+
},
|
|
188
|
+
"keyRefusals": {
|
|
189
|
+
"engine_access_key": "This key uses WhiteRoom's own cloud access, so there's no key of yours to test with. Add a provider key on Fleet key, then pick it.",
|
|
190
|
+
"other_host": "This key sends its calls to a host other than its provider's own. Practice tests don't support that.",
|
|
191
|
+
"unsupported_provider": "Practice tests don't support this key's provider.",
|
|
192
|
+
"short_term_key": "This is a short-term Bedrock key, which expires within hours. Add a long-term Bedrock API key on Fleet key, then start a new test and pick it.",
|
|
193
|
+
"bedrock_role": "This fleet reaches AWS Bedrock through an IAM role, which practice tests don't support.",
|
|
194
|
+
"key_not_found": "That key isn't on Fleet key any more. Pick another key.",
|
|
195
|
+
"model_required": "Enter the model your agent uses: the deployment name on Azure, or a Bedrock model ID like anthropic.claude-haiku-4-5."
|
|
196
|
+
},
|
|
197
|
+
"rows": [
|
|
198
|
+
{
|
|
199
|
+
"id": "finishes_job",
|
|
200
|
+
"name": "Finishes the job correctly",
|
|
201
|
+
"group": "The agent",
|
|
202
|
+
"members": [
|
|
203
|
+
"a."
|
|
204
|
+
]
|
|
205
|
+
},
|
|
206
|
+
{
|
|
207
|
+
"id": "long_task",
|
|
208
|
+
"name": "Survives a long task",
|
|
209
|
+
"group": "The agent",
|
|
210
|
+
"members": [
|
|
211
|
+
"b."
|
|
212
|
+
]
|
|
213
|
+
},
|
|
214
|
+
{
|
|
215
|
+
"id": "fails_gracefully",
|
|
216
|
+
"name": "Fails gracefully",
|
|
217
|
+
"group": "The agent",
|
|
218
|
+
"members": [
|
|
219
|
+
"c."
|
|
220
|
+
]
|
|
221
|
+
},
|
|
222
|
+
{
|
|
223
|
+
"id": "spend_cap",
|
|
224
|
+
"name": "Spend cap",
|
|
225
|
+
"group": "The guardrails",
|
|
226
|
+
"members": [
|
|
227
|
+
"ctl.spend_cap"
|
|
228
|
+
]
|
|
229
|
+
},
|
|
230
|
+
{
|
|
231
|
+
"id": "loop_breaker",
|
|
232
|
+
"name": "Loop breaker + call rate",
|
|
233
|
+
"group": "The guardrails",
|
|
234
|
+
"members": [
|
|
235
|
+
"ctl.loop_breaker",
|
|
236
|
+
"ctl.call_rate"
|
|
237
|
+
]
|
|
238
|
+
},
|
|
239
|
+
{
|
|
240
|
+
"id": "keyword",
|
|
241
|
+
"name": "Allowlists + keywords",
|
|
242
|
+
"group": "The guardrails",
|
|
243
|
+
"members": [
|
|
244
|
+
"ctl.model_allowlist",
|
|
245
|
+
"ctl.tool_list",
|
|
246
|
+
"ctl.keyword"
|
|
247
|
+
]
|
|
248
|
+
},
|
|
249
|
+
{
|
|
250
|
+
"id": "pause_stop",
|
|
251
|
+
"name": "Human pause/stop",
|
|
252
|
+
"group": "The guardrails",
|
|
253
|
+
"members": [
|
|
254
|
+
"ctl.pause_stop"
|
|
255
|
+
]
|
|
256
|
+
}
|
|
257
|
+
],
|
|
258
|
+
"rowStatusLabels": {
|
|
259
|
+
"passed": "Passed",
|
|
260
|
+
"failed": "Didn't pass",
|
|
261
|
+
"not_set_up": "Not set up",
|
|
262
|
+
"not_checked": "Not checked"
|
|
263
|
+
}
|
|
264
|
+
}
|
package/dist/index.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import {
|
|
2
|
+
import { parseArgs } from "./legacy-args.js";
|
|
3
3
|
const USAGE = `whiteroom — CLI for WhiteRoom agent governance
|
|
4
4
|
|
|
5
5
|
Usage:
|
|
@@ -15,31 +15,35 @@ Usage:
|
|
|
15
15
|
whiteroom report [--fleet <id>]
|
|
16
16
|
whiteroom fleets
|
|
17
17
|
whiteroom login <fleet_token>
|
|
18
|
+
whiteroom practice run [--adapter "<command>"] [--key-env <VAR>] [--code-stdin]
|
|
19
|
+
whiteroom practice adapter --lang js|python
|
|
20
|
+
whiteroom sandbox listen --adapter "<command>" [--code-stdin]
|
|
18
21
|
|
|
19
22
|
Options:
|
|
20
23
|
--url <base_url> Server URL (default: WHITEROOM_URL or http://localhost:3000)
|
|
21
24
|
--key <api_key> API key (default: WHITEROOM_API_KEY)
|
|
22
25
|
`;
|
|
23
|
-
function parseArgs(argv) {
|
|
24
|
-
const args = [];
|
|
25
|
-
const flags = {};
|
|
26
|
-
for (let i = 0; i < argv.length; i++) {
|
|
27
|
-
if (argv[i].startsWith("--") && i + 1 < argv.length) {
|
|
28
|
-
flags[argv[i].slice(2)] = argv[++i];
|
|
29
|
-
}
|
|
30
|
-
else {
|
|
31
|
-
args.push(argv[i]);
|
|
32
|
-
}
|
|
33
|
-
}
|
|
34
|
-
return { args, flags };
|
|
35
|
-
}
|
|
36
26
|
async function main() {
|
|
37
|
-
const
|
|
27
|
+
const argv = process.argv.slice(2);
|
|
28
|
+
// Practice tests never load the SDK (spec §9.5): plain fetch only.
|
|
29
|
+
if (argv[0] === "practice" || argv[0] === "readiness") {
|
|
30
|
+
const { practice } = await import("./practice/index.js");
|
|
31
|
+
process.exitCode = await practice(argv.slice(1));
|
|
32
|
+
return;
|
|
33
|
+
}
|
|
34
|
+
// The Sandbox helper is plain fetch too.
|
|
35
|
+
if (argv[0] === "sandbox") {
|
|
36
|
+
const { sandbox } = await import("./sandbox/index.js");
|
|
37
|
+
process.exitCode = await sandbox(argv.slice(1));
|
|
38
|
+
return;
|
|
39
|
+
}
|
|
40
|
+
const { args, flags } = parseArgs(argv);
|
|
38
41
|
const command = args[0];
|
|
39
42
|
if (!command || command === "help" || command === "--help") {
|
|
40
43
|
console.log(USAGE);
|
|
41
44
|
process.exit(0);
|
|
42
45
|
}
|
|
46
|
+
const { WhiteRoomClient, WhiteRoomError } = await import("@whiteroom-ai/sdk");
|
|
43
47
|
const client = new WhiteRoomClient({
|
|
44
48
|
baseUrl: flags.url ?? process.env.WHITEROOM_URL ?? "http://localhost:3000",
|
|
45
49
|
apiKey: flags.key ?? process.env.WHITEROOM_API_KEY,
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AAE7C,MAAM,KAAK,GAAG;;;;;;;;;;;;;;;;;;;;;;CAsBb,CAAC;AAEF,KAAK,UAAU,IAAI;IACjB,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IACnC,mEAAmE;IACnE,IAAI,IAAI,CAAC,CAAC,CAAC,KAAK,UAAU,IAAI,IAAI,CAAC,CAAC,CAAC,KAAK,WAAW,EAAE,CAAC;QACtD,MAAM,EAAE,QAAQ,EAAE,GAAG,MAAM,MAAM,CAAC,qBAAqB,CAAC,CAAC;QACzD,OAAO,CAAC,QAAQ,GAAG,MAAM,QAAQ,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;QACjD,OAAO;IACT,CAAC;IACD,yCAAyC;IACzC,IAAI,IAAI,CAAC,CAAC,CAAC,KAAK,SAAS,EAAE,CAAC;QAC1B,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACvD,OAAO,CAAC,QAAQ,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;QAChD,OAAO;IACT,CAAC;IACD,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;IACxC,MAAM,OAAO,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;IAExB,IAAI,CAAC,OAAO,IAAI,OAAO,KAAK,MAAM,IAAI,OAAO,KAAK,QAAQ,EAAE,CAAC;QAC3D,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;QACnB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,MAAM,EAAE,eAAe,EAAE,cAAc,EAAE,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,CAAC;IAC9E,MAAM,MAAM,GAAG,IAAI,eAAe,CAAC;QACjC,OAAO,EAAE,KAAK,CAAC,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,uBAAuB;QAC1E,MAAM,EAAE,KAAK,CAAC,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB;KACnD,CAAC,CAAC;IAEH,IAAI,CAAC;QACH,QAAQ,OAAO,EAAE,CAAC;YAChB,KAAK,QAAQ;gBACX,KAAK,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC;gBAC7B,MAAM;YAER,KAAK,UAAU;gBACb,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,KAAK,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,OAAO,EAAE,KAAK,CAAC,KAAK,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;gBACvF,MAAM;YAER,KAAK,MAAM;gBACT,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,SAAS,CAAC,CAAC;gBAC/B,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,SAAS,CAAC,CAAC;gBAC/B,KAAK,CAAC,MAAM,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBAC9D,MAAM;YAER,KAAK,WAAW;gBACd,KAAK,CAAC,MAAM,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBAC1C,MAAM;YAER,KAAK,OAAO;gBACV,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,KAAK,CAAC,MAAM,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBACrD,MAAM;YAER,KAAK,MAAM;gBACT,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,WAAW,CAAC,CAAC;gBACjC,KAAK,CACH,MAAM,MAAM,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,EAAE;oBAC1C,OAAO,EAAE,KAAK,CAAC,KAAK;oBACpB,YAAY,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS;oBAC/D,UAAU,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,SAAS;iBAC5D,CAAC,CACH,CAAC;gBACF,MAAM;YAER,KAAK,OAAO;gBACV,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,KAAK,CAAC,MAAM,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBACrD,MAAM;YAER,KAAK,UAAU;gBACb,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,YAAY,CAAC,CAAC;gBAClC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,KAAK,CAAC,MAAM,MAAM,CAAC,gBAAgB,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,OAAO,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;gBACjF,MAAM;YAER,KAAK,OAAO;gBACV,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC;gBAChC,KAAK,CAAC,MAAM,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBACpD,MAAM;YAER,KAAK,QAAQ;gBACX,KAAK,CAAC,MAAM,MAAM,CAAC,WAAW,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;gBAC7C,MAAM;YAER,KAAK,QAAQ;gBACX,KAAK,CAAC,MAAM,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC;gBACjC,MAAM;YAER,KAAK,OAAO;gBACV,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,aAAa,CAAC,CAAC;gBACnC,KAAK,CAAC,MAAM,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;gBACxC,MAAM;YAER;gBACE,OAAO,CAAC,KAAK,CAAC,oBAAoB,OAAO,IAAI,CAAC,CAAC;gBAC/C,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;gBACnB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,CAAC;IACH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,IAAI,CAAC,YAAY,cAAc,EAAE,CAAC;YAChC,OAAO,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC;QACD,MAAM,CAAC,CAAC;IACV,CAAC;AACH,CAAC;AAED,SAAS,UAAU,CAAC,KAAyB,EAAE,IAAY;IACzD,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,OAAO,CAAC,KAAK,CAAC,+BAA+B,IAAI,GAAG,CAAC,CAAC;QACtD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC;AAED,SAAS,KAAK,CAAC,IAAa;IAC1B,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;AAC7C,CAAC;AAED,IAAI,EAAE,CAAC"}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
// The fleet commands' argument parser, unchanged from before practice tests:
|
|
2
|
+
// every `--name` takes the next word as its value, whatever it looks like.
|
|
3
|
+
// (Practice commands use node:util parseArgs instead.)
|
|
4
|
+
export function parseArgs(argv) {
|
|
5
|
+
const args = [];
|
|
6
|
+
const flags = {};
|
|
7
|
+
for (let i = 0; i < argv.length; i++) {
|
|
8
|
+
if (argv[i].startsWith("--") && i + 1 < argv.length) {
|
|
9
|
+
flags[argv[i].slice(2)] = argv[++i];
|
|
10
|
+
}
|
|
11
|
+
else {
|
|
12
|
+
args.push(argv[i]);
|
|
13
|
+
}
|
|
14
|
+
}
|
|
15
|
+
return { args, flags };
|
|
16
|
+
}
|
|
17
|
+
//# sourceMappingURL=legacy-args.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"legacy-args.js","sourceRoot":"","sources":["../src/legacy-args.ts"],"names":[],"mappings":"AAAA,6EAA6E;AAC7E,2EAA2E;AAC3E,uDAAuD;AACvD,MAAM,UAAU,SAAS,CAAC,IAAc;IACtC,MAAM,IAAI,GAAa,EAAE,CAAC;IAC1B,MAAM,KAAK,GAA2B,EAAE,CAAC;IACzC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,IAAI,IAAI,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC;YACpD,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;QACtC,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;QACrB,CAAC;IACH,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;AACzB,CAAC"}
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
import { type ChildProcess } from "node:child_process";
|
|
2
|
+
export declare const MAX_LINE_BYTES: number;
|
|
3
|
+
export declare const MAX_TOOL_CALLS = 200;
|
|
4
|
+
/** A tool call's request to WhiteRoom must fit its 64 KiB body limit. */
|
|
5
|
+
export declare const MAX_TOOL_BODY_BYTES: number;
|
|
6
|
+
/** After SIGTERM, how long the group has before SIGKILL. */
|
|
7
|
+
export declare const KILL_GRACE_MS = 5000;
|
|
8
|
+
/** How to start the adapter: the user's own command through a shell, or the sample agent. */
|
|
9
|
+
export type Launch = {
|
|
10
|
+
shell: string;
|
|
11
|
+
} | {
|
|
12
|
+
file: string;
|
|
13
|
+
args: string[];
|
|
14
|
+
};
|
|
15
|
+
export interface ToolCall {
|
|
16
|
+
id: string;
|
|
17
|
+
name: string;
|
|
18
|
+
input: unknown;
|
|
19
|
+
}
|
|
20
|
+
/** How a practice task's adapter run ended. */
|
|
21
|
+
export type Outcome = {
|
|
22
|
+
kind: "final";
|
|
23
|
+
output: string;
|
|
24
|
+
} | {
|
|
25
|
+
kind: "exited";
|
|
26
|
+
how: string;
|
|
27
|
+
tail: string[];
|
|
28
|
+
} | {
|
|
29
|
+
kind: "broke";
|
|
30
|
+
why: string;
|
|
31
|
+
} | {
|
|
32
|
+
kind: "deadline";
|
|
33
|
+
} | {
|
|
34
|
+
kind: "stopped";
|
|
35
|
+
}
|
|
36
|
+
/** The CLI's tool callback ended the task (restart, deadline, session over). */
|
|
37
|
+
| {
|
|
38
|
+
kind: "ended";
|
|
39
|
+
by: unknown;
|
|
40
|
+
};
|
|
41
|
+
export type ToolAnswer = {
|
|
42
|
+
result: unknown;
|
|
43
|
+
} | {
|
|
44
|
+
end: unknown;
|
|
45
|
+
};
|
|
46
|
+
/** The adapter's environment: the CLI's, minus WhiteRoom's settings and every provider key, with only the run's key added. */
|
|
47
|
+
export declare function childEnv(parent: NodeJS.ProcessEnv, keyVar: string, key: string): NodeJS.ProcessEnv;
|
|
48
|
+
/** SIGTERM to the adapter's whole process group, then SIGKILL if anything is left after KILL_GRACE_MS. */
|
|
49
|
+
export declare function killGroup(child: ChildProcess): Promise<void>;
|
|
50
|
+
export interface RunOptions {
|
|
51
|
+
launch: Launch;
|
|
52
|
+
env: NodeJS.ProcessEnv;
|
|
53
|
+
/** The first line the adapter reads. */
|
|
54
|
+
message: unknown;
|
|
55
|
+
deadline: number;
|
|
56
|
+
signal: AbortSignal;
|
|
57
|
+
tool: (call: ToolCall) => Promise<ToolAnswer>;
|
|
58
|
+
}
|
|
59
|
+
export declare function runAdapter(o: RunOptions): Promise<Outcome>;
|