@beryl-so/cli 0.32.0 → 0.34.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +60 -59
- package/dist/commands/accounts.js +6 -6
- package/dist/commands/auth.js +5 -5
- package/dist/commands/config-vars.js +5 -5
- package/dist/commands/environments.js +78 -44
- package/dist/commands/explorations.js +3 -3
- package/dist/commands/groups.js +2 -2
- package/dist/commands/health.js +1 -1
- package/dist/commands/init.js +8 -8
- package/dist/commands/mailboxes.js +6 -6
- package/dist/commands/mcp.js +3 -3
- package/dist/commands/projects.js +5 -5
- package/dist/commands/runs.js +12 -12
- package/dist/commands/tests.js +86 -35
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,18 +1,18 @@
|
|
|
1
|
-
<!-- GENERATED by `npm run docs
|
|
1
|
+
<!-- GENERATED by `npm run docs`. Edit the command registry (src/commands/), not this file. -->
|
|
2
2
|
|
|
3
3
|
# beryl CLI
|
|
4
4
|
|
|
5
5
|
Beryl on the command line: create projects, watch the agent explore and author tests,
|
|
6
|
-
trigger runs in CI, manage everything the web app can
|
|
6
|
+
trigger runs in CI, manage everything the web app can, plus an MCP server (`beryl mcp`)
|
|
7
7
|
that exposes every command to coding agents.
|
|
8
8
|
|
|
9
|
-
## Quickstart
|
|
9
|
+
## Quickstart: one command
|
|
10
10
|
|
|
11
11
|
```bash
|
|
12
12
|
npx @beryl-so/cli@latest init
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
Signs you in (browser approval by default, emailed one-time code as fallback
|
|
15
|
+
Signs you in (browser approval by default, emailed one-time code as fallback; new
|
|
16
16
|
emails are signed up on the spot), wires the beryl + playwright MCP servers into Claude
|
|
17
17
|
Code or Cursor, installs the beryl-test authoring skill, and installs Playwright for
|
|
18
18
|
local runs. Safe to re-run.
|
|
@@ -27,18 +27,18 @@ npx @beryl-so/cli@latest … # or run one-off without installing
|
|
|
27
27
|
From the monorepo (development): `cd cli && npm install && npm run build && npm link`.
|
|
28
28
|
|
|
29
29
|
`beryl init` and `beryl mcp` warn (stderr, best-effort) when the running CLI is behind the
|
|
30
|
-
latest npm release
|
|
30
|
+
latest npm release; a stale MCP server silently exposes fewer tools. Set
|
|
31
31
|
`BERYL_NO_UPDATE_CHECK=1` to opt out.
|
|
32
32
|
|
|
33
33
|
## Authenticate
|
|
34
34
|
|
|
35
35
|
```bash
|
|
36
|
-
beryl login # opens the browser to approve
|
|
36
|
+
beryl login # opens the browser to approve; press Enter for an emailed code instead
|
|
37
37
|
beryl login --otp # skip the browser; emailed one-time code (signs up new emails too)
|
|
38
38
|
export BERYL_API_KEY=beryl_pat_… # CI: use a token from Account → API tokens
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
Config stores your token + API URL only
|
|
41
|
+
Config stores your token + API URL only; nothing else persists. Point a command at a
|
|
42
42
|
workspace/project with the `--workspace` / `--project` flags or the `BERYL_WORKSPACE` /
|
|
43
43
|
`BERYL_PROJECT` env vars; with exactly one, the CLI auto-picks it.
|
|
44
44
|
Precedence: flags > `BERYL_*` env > auto-pick-if-one.
|
|
@@ -59,18 +59,18 @@ claude mcp add beryl -- beryl mcp # or, with the CLI installe
|
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
Most non-interactive commands below are exposed as an MCP tool with the same name
|
|
62
|
-
(spaces and dashes become underscores)
|
|
62
|
+
(spaces and dashes become underscores); see the MCP tool column. `none` marks commands
|
|
63
63
|
kept CLI-only: account/billing/team admin, feedback, and cloud exploration.
|
|
64
64
|
|
|
65
65
|
## Commands
|
|
66
66
|
|
|
67
67
|
### init
|
|
68
68
|
|
|
69
|
-
Set up Beryl in this repo
|
|
69
|
+
Set up Beryl in this repo: sign in and wire up your coding agent
|
|
70
70
|
|
|
71
71
|
| Command | Summary | MCP tool |
|
|
72
72
|
| --- | --- | --- |
|
|
73
|
-
| `beryl init` | Set up Beryl in this repo
|
|
73
|
+
| `beryl init` | Set up Beryl in this repo: sign in and wire up your coding agent | none |
|
|
74
74
|
|
|
75
75
|
### guide
|
|
76
76
|
|
|
@@ -86,7 +86,7 @@ Authenticate the CLI with your Beryl account
|
|
|
86
86
|
|
|
87
87
|
| Command | Summary | MCP tool |
|
|
88
88
|
| --- | --- | --- |
|
|
89
|
-
| `beryl login` | Authenticate the CLI with your Beryl account |
|
|
89
|
+
| `beryl login` | Authenticate the CLI with your Beryl account | none |
|
|
90
90
|
|
|
91
91
|
### signup
|
|
92
92
|
|
|
@@ -143,9 +143,9 @@ Manage who belongs to a workspace and their roles.
|
|
|
143
143
|
|
|
144
144
|
| Command | Summary | MCP tool |
|
|
145
145
|
| --- | --- | --- |
|
|
146
|
-
| `beryl members list` | List workspace members |
|
|
147
|
-
| `beryl members set-role <user-id> <role>` | Change a member's role |
|
|
148
|
-
| `beryl members remove <user-id>` | Remove a member from the workspace |
|
|
146
|
+
| `beryl members list` | List workspace members | none |
|
|
147
|
+
| `beryl members set-role <user-id> <role>` | Change a member's role | none |
|
|
148
|
+
| `beryl members remove <user-id>` | Remove a member from the workspace | none |
|
|
149
149
|
|
|
150
150
|
### invites
|
|
151
151
|
|
|
@@ -153,16 +153,16 @@ Send, list, and revoke workspace invitations, and accept ones sent to you.
|
|
|
153
153
|
|
|
154
154
|
| Command | Summary | MCP tool |
|
|
155
155
|
| --- | --- | --- |
|
|
156
|
-
| `beryl invites send <email>` | Invite someone to the workspace by email |
|
|
157
|
-
| `beryl invites list` | List the workspace's outstanding invitations |
|
|
158
|
-
| `beryl invites revoke <invitation-id>` | Revoke a pending invitation |
|
|
159
|
-
| `beryl invites mine` | List invitations sent to you |
|
|
160
|
-
| `beryl invites accept <invitation>` | Accept an invitation (by id, or by the token from the invite email) |
|
|
161
|
-
| `beryl invites decline <invitation-id>` | Decline an invitation |
|
|
156
|
+
| `beryl invites send <email>` | Invite someone to the workspace by email | none |
|
|
157
|
+
| `beryl invites list` | List the workspace's outstanding invitations | none |
|
|
158
|
+
| `beryl invites revoke <invitation-id>` | Revoke a pending invitation | none |
|
|
159
|
+
| `beryl invites mine` | List invitations sent to you | none |
|
|
160
|
+
| `beryl invites accept <invitation>` | Accept an invitation (by id, or by the token from the invite email) | none |
|
|
161
|
+
| `beryl invites decline <invitation-id>` | Decline an invitation | none |
|
|
162
162
|
|
|
163
163
|
### projects
|
|
164
164
|
|
|
165
|
-
Create and manage projects
|
|
165
|
+
Create and manage projects: a site Beryl explores, authors tests for, and runs.
|
|
166
166
|
|
|
167
167
|
`beryl projects` with no subcommand runs `projects list`.
|
|
168
168
|
|
|
@@ -170,16 +170,16 @@ Create and manage projects — a site Beryl explores, authors tests for, and run
|
|
|
170
170
|
| --- | --- | --- |
|
|
171
171
|
| `beryl projects list` | List projects in the workspace | `projects_list` |
|
|
172
172
|
| `beryl projects get` | Show one project, including its current exploration state | `projects_get` |
|
|
173
|
-
| `beryl projects create [url]` | Create a project
|
|
173
|
+
| `beryl projects create [url]` | Create a project. With a URL the agent starts exploring; with just --name an empty one | `projects_create` |
|
|
174
174
|
| `beryl projects rename <name>` | Rename a project | `projects_rename` |
|
|
175
175
|
| `beryl projects delete` | Delete a project and all its tests and runs | `projects_delete` |
|
|
176
|
-
| `beryl projects re-explore` | Send the agent back in
|
|
176
|
+
| `beryl projects re-explore` | Send the agent back in to run/heal existing tests and discover new flows | none |
|
|
177
177
|
| `beryl projects report` | Aggregate quality report across recent runs (pass rates, flaky tests, trend) | `projects_report` |
|
|
178
178
|
| `beryl projects reusable-auth <url>` | Check whether a saved login can be reused for a URL before creating a project | `projects_reusable_auth` |
|
|
179
179
|
|
|
180
180
|
### envs
|
|
181
181
|
|
|
182
|
-
Manage a project's environments
|
|
182
|
+
Manage a project's environments: the URLs and auth Beryl runs tests against.
|
|
183
183
|
|
|
184
184
|
| Command | Summary | MCP tool |
|
|
185
185
|
| --- | --- | --- |
|
|
@@ -191,18 +191,18 @@ Manage a project's environments — the URLs and auth Beryl runs tests against.
|
|
|
191
191
|
|
|
192
192
|
### schedule
|
|
193
193
|
|
|
194
|
-
|
|
194
|
+
Manage the schedules on which Beryl runs a project's tests automatically. A project can hold many independent schedules; each has its own cadence (daily or weekly), local run time, timezone, and target groups (none = every active test).
|
|
195
195
|
|
|
196
196
|
| Command | Summary | MCP tool |
|
|
197
197
|
| --- | --- | --- |
|
|
198
|
-
| `beryl schedule
|
|
199
|
-
| `beryl schedule
|
|
200
|
-
| `beryl schedule
|
|
201
|
-
| `beryl schedule remove
|
|
198
|
+
| `beryl schedule list` | List the project's schedules | `schedule_list` |
|
|
199
|
+
| `beryl schedule add` | Add a schedule (daily, or weekly on a given day) | `schedule_add` |
|
|
200
|
+
| `beryl schedule update <schedule-id>` | Change a schedule's cadence, time, timezone, or groups | `schedule_update` |
|
|
201
|
+
| `beryl schedule remove <schedule-id>` | Remove a schedule (its tests and groups stay) | `schedule_remove` |
|
|
202
202
|
|
|
203
203
|
### tests
|
|
204
204
|
|
|
205
|
-
Author, inspect, version, and heal a project's tests
|
|
205
|
+
Author, inspect, version, and heal a project's tests: the checks Beryl runs on each run.
|
|
206
206
|
|
|
207
207
|
`beryl tests` with no subcommand runs `tests list`.
|
|
208
208
|
|
|
@@ -212,12 +212,13 @@ Author, inspect, version, and heal a project's tests — the checks Beryl runs o
|
|
|
212
212
|
| `beryl tests list` | List the project's tests with their latest result | `tests_list` |
|
|
213
213
|
| `beryl tests get <test-id>` | Show one test | `tests_get` |
|
|
214
214
|
| `beryl tests plan <test-id>` | Print a test's current step plan (JSON) | `tests_plan` |
|
|
215
|
-
| `beryl tests create` | Create a test case from a JSON action plan
|
|
215
|
+
| `beryl tests create` | Create a test case from a JSON action plan (for tests authored locally, e.g. by your coding agent) | `tests_create` |
|
|
216
216
|
| `beryl tests set-plan <test-id>` | Replace a test's step plan from a JSON file (creates a new version) | `tests_set_plan` |
|
|
217
217
|
| `beryl tests rename <test-id> <title>` | Rename a test | `tests_rename` |
|
|
218
218
|
| `beryl tests set-groups <test-id>` | Replace the groups a test belongs to | `tests_set_groups` |
|
|
219
|
+
| `beryl tests add-groups <test-ids...>` | Add groups to one or more tests, keeping the groups they already have | `tests_add_groups` |
|
|
219
220
|
| `beryl tests quarantine <test-id> <state>` | Mute a flaky test: it keeps running, but its failures stop failing the run | `tests_quarantine` |
|
|
220
|
-
| `beryl tests delete <test-
|
|
221
|
+
| `beryl tests delete <test-ids...>` | Delete tests, their version history, and their results | `tests_delete` |
|
|
221
222
|
| `beryl tests recompile <test-id>` | Validate + verify an edited plan against the live site before persisting | `tests_recompile` |
|
|
222
223
|
| `beryl tests versions <test-id>` | List a test's version history | `tests_versions` |
|
|
223
224
|
| `beryl tests version <test-id> <version-no>` | Show one specific version of a test (including its plan) | `tests_version` |
|
|
@@ -231,7 +232,7 @@ Author, inspect, version, and heal a project's tests — the checks Beryl runs o
|
|
|
231
232
|
|
|
232
233
|
### groups
|
|
233
234
|
|
|
234
|
-
Manage a project's test groups
|
|
235
|
+
Manage a project's test groups: labels a test can carry any number of, used to filter, run, or schedule a slice of the suite. Groups are created only here (or in Settings → Groups); assigning a test to an unknown name is an error.
|
|
235
236
|
|
|
236
237
|
`beryl groups` with no subcommand runs `groups list`.
|
|
237
238
|
|
|
@@ -240,7 +241,7 @@ Manage a project's test groups — labels a test can carry any number of, used t
|
|
|
240
241
|
| `beryl groups list` | List the project's test groups and how many tests each holds | `groups_list` |
|
|
241
242
|
| `beryl groups create <name>` | Create a group | `groups_create` |
|
|
242
243
|
| `beryl groups rename <group> <name>` | Rename a group (tests keep their membership) | `groups_rename` |
|
|
243
|
-
| `beryl groups delete <group>` | Delete a group
|
|
244
|
+
| `beryl groups delete <group>` | Delete a group: its tests stay; a schedule targeting only this group goes with it | `groups_delete` |
|
|
244
245
|
|
|
245
246
|
### runs
|
|
246
247
|
|
|
@@ -262,7 +263,7 @@ Trigger a run of a project's tests (e.g. in CI), then watch, inspect, and downlo
|
|
|
262
263
|
|
|
263
264
|
### health
|
|
264
265
|
|
|
265
|
-
Site Health
|
|
266
|
+
Site Health, or how your site reads to search engines and visitors: content, speed, mobile, links and security, graded from a real check of your live pages.
|
|
266
267
|
|
|
267
268
|
`beryl health` with no subcommand runs `health get`.
|
|
268
269
|
|
|
@@ -273,15 +274,15 @@ Site Health — how your site reads to search engines and visitors: content, spe
|
|
|
273
274
|
|
|
274
275
|
### explorations
|
|
275
276
|
|
|
276
|
-
Inspect the agent's exploration runs
|
|
277
|
+
Inspect the agent's exploration runs: how it crawled a site and authored its tests.
|
|
277
278
|
|
|
278
279
|
| Command | Summary | MCP tool |
|
|
279
280
|
| --- | --- | --- |
|
|
280
|
-
| `beryl explorations list` | List the agent's exploration passes for a project |
|
|
281
|
-
| `beryl explorations get <exploration-id>` | Show one exploration: authored tests, abandoned flows, coverage, frontier |
|
|
282
|
-
| `beryl explorations steps <exploration-id>` | List every step the agent took in an exploration |
|
|
283
|
-
| `beryl explorations cancel <exploration-id>` | Cancel an in-flight exploration |
|
|
284
|
-
| `beryl explorations watch <exploration-id>` | Stream an exploration live
|
|
281
|
+
| `beryl explorations list` | List the agent's exploration passes for a project | none |
|
|
282
|
+
| `beryl explorations get <exploration-id>` | Show one exploration: authored tests, abandoned flows, coverage, frontier | none |
|
|
283
|
+
| `beryl explorations steps <exploration-id>` | List every step the agent took in an exploration | none |
|
|
284
|
+
| `beryl explorations cancel <exploration-id>` | Cancel an in-flight exploration | none |
|
|
285
|
+
| `beryl explorations watch <exploration-id>` | Stream an exploration live: watch the agent explore and author tests | none |
|
|
285
286
|
|
|
286
287
|
### config
|
|
287
288
|
|
|
@@ -305,7 +306,7 @@ Manage the variables, secrets, and files the agent can use while exploring and r
|
|
|
305
306
|
|
|
306
307
|
### mailbox
|
|
307
308
|
|
|
308
|
-
The project's standing email addresses
|
|
309
|
+
The project's standing email addresses, where its tests receive sign-in mail.
|
|
309
310
|
|
|
310
311
|
`beryl mailbox` with no subcommand runs `mailbox get`.
|
|
311
312
|
|
|
@@ -320,14 +321,14 @@ The project's standing email addresses — where its tests receive sign-in mail.
|
|
|
320
321
|
|
|
321
322
|
### accounts
|
|
322
323
|
|
|
323
|
-
Durable identities on the site under test
|
|
324
|
+
Durable identities on the site under test: what an authenticated test signs in as.
|
|
324
325
|
|
|
325
326
|
`beryl accounts` with no subcommand runs `accounts list`.
|
|
326
327
|
|
|
327
328
|
| Command | Summary | MCP tool |
|
|
328
329
|
| --- | --- | --- |
|
|
329
330
|
| `beryl accounts list` | List the test accounts an environment's tests sign in as | `accounts_list` |
|
|
330
|
-
| `beryl accounts create` | Add a test account
|
|
331
|
+
| `beryl accounts create` | Add a test account: the customer's own, or one Beryl signs up | `accounts_create` |
|
|
331
332
|
| `beryl accounts provision <account-id>` | Prove a test account can get in, by replaying a plan that ends logged in | `accounts_provision` |
|
|
332
333
|
| `beryl accounts set-login <account-id>` | Store the sign-in plan a run replays once, plus the probe that proves it | `accounts_set_login` |
|
|
333
334
|
| `beryl accounts get-login <account-id>` | Read the stored sign-in plan, its probe, and the hash a safe write must cite | `accounts_get_login` |
|
|
@@ -341,9 +342,9 @@ View and update your personal account profile.
|
|
|
341
342
|
|
|
342
343
|
| Command | Summary | MCP tool |
|
|
343
344
|
| --- | --- | --- |
|
|
344
|
-
| `beryl account get` | Show your account profile |
|
|
345
|
-
| `beryl account update` | Update your profile |
|
|
346
|
-
| `beryl account deletion-preview` | Preview what deleting your account would remove or leave |
|
|
345
|
+
| `beryl account get` | Show your account profile | none |
|
|
346
|
+
| `beryl account update` | Update your profile | none |
|
|
347
|
+
| `beryl account deletion-preview` | Preview what deleting your account would remove or leave | none |
|
|
347
348
|
|
|
348
349
|
### feedback
|
|
349
350
|
|
|
@@ -351,7 +352,7 @@ Send product feedback to the Beryl team.
|
|
|
351
352
|
|
|
352
353
|
| Command | Summary | MCP tool |
|
|
353
354
|
| --- | --- | --- |
|
|
354
|
-
| `beryl feedback send <message>` | Send product feedback to the Beryl team |
|
|
355
|
+
| `beryl feedback send <message>` | Send product feedback to the Beryl team | none |
|
|
355
356
|
|
|
356
357
|
### billing
|
|
357
358
|
|
|
@@ -359,10 +360,10 @@ Review a workspace's plan usage, subscription, and invoices.
|
|
|
359
360
|
|
|
360
361
|
| Command | Summary | MCP tool |
|
|
361
362
|
| --- | --- | --- |
|
|
362
|
-
| `beryl billing usage` | Show plan usage: monthly cloud run minutes |
|
|
363
|
-
| `beryl billing subscription` | Show the workspace's subscription |
|
|
364
|
-
| `beryl billing invoices` | List recent invoices |
|
|
365
|
-
| `beryl billing portal` | Get a Stripe billing-portal link for the workspace |
|
|
363
|
+
| `beryl billing usage` | Show plan usage: monthly cloud run minutes | none |
|
|
364
|
+
| `beryl billing subscription` | Show the workspace's subscription | none |
|
|
365
|
+
| `beryl billing invoices` | List recent invoices | none |
|
|
366
|
+
| `beryl billing portal` | Get a Stripe billing-portal link for the workspace | none |
|
|
366
367
|
|
|
367
368
|
### version
|
|
368
369
|
|
|
@@ -374,18 +375,18 @@ Show the running CLI version, API URL, and Node version
|
|
|
374
375
|
|
|
375
376
|
### mcp
|
|
376
377
|
|
|
377
|
-
Run the Beryl MCP server (stdio)
|
|
378
|
+
Run the Beryl MCP server (stdio): every CLI command as an agent tool
|
|
378
379
|
|
|
379
380
|
| Command | Summary | MCP tool |
|
|
380
381
|
| --- | --- | --- |
|
|
381
|
-
| `beryl mcp` | Run the Beryl MCP server (stdio)
|
|
382
|
+
| `beryl mcp` | Run the Beryl MCP server (stdio): every CLI command as an agent tool | none |
|
|
382
383
|
|
|
383
384
|
Run `beryl <command> --help` for flags and examples.
|
|
384
385
|
|
|
385
386
|
## Global flags
|
|
386
387
|
|
|
387
|
-
- `-V, --version
|
|
388
|
-
- `--json
|
|
389
|
-
- `--api-url
|
|
390
|
-
- `--token
|
|
391
|
-
- `-h, --help
|
|
388
|
+
- `-V, --version`: Print the CLI version and exit
|
|
389
|
+
- `--json`: Print machine-readable JSON (NDJSON for streams)
|
|
390
|
+
- `--api-url`: API base URL (default https://api.beryl.so)
|
|
391
|
+
- `--token`: Personal access token (overrides config/BERYL_API_KEY)
|
|
392
|
+
- `-h, --help`: Show help
|
|
@@ -37,7 +37,7 @@ export const testAccountCommands = [
|
|
|
37
37
|
name: "accounts list",
|
|
38
38
|
summary: "List the test accounts an environment's tests sign in as",
|
|
39
39
|
groupDefault: true,
|
|
40
|
-
groupSummary: "Durable identities on the site under test
|
|
40
|
+
groupSummary: "Durable identities on the site under test: what an authenticated test signs in as.",
|
|
41
41
|
description: "Every test account on the project, with the identity label a plan's `auth_label` " +
|
|
42
42
|
"names. The row marked * is the default: what a plan gets when it sets " +
|
|
43
43
|
"requires_auth without an auth_label.",
|
|
@@ -59,10 +59,10 @@ export const testAccountCommands = [
|
|
|
59
59
|
},
|
|
60
60
|
{
|
|
61
61
|
name: "accounts create",
|
|
62
|
-
summary: "Add a test account
|
|
62
|
+
summary: "Add a test account: the customer's own, or one Beryl signs up",
|
|
63
63
|
description: "Two kinds. `--type user_provided` records a dedicated account you already have on " +
|
|
64
64
|
"the site: pass --email and --password, and it is usable immediately. `--type beryl` " +
|
|
65
|
-
"reserves one Beryl will sign up itself, addressed at the project mailbox
|
|
65
|
+
"reserves one Beryl will sign up itself, addressed at the project mailbox. It starts " +
|
|
66
66
|
"`pending` and becomes usable after `beryl accounts provision`. " +
|
|
67
67
|
"The first account an environment gets is its default whatever you pass.",
|
|
68
68
|
scope: "project",
|
|
@@ -124,11 +124,11 @@ export const testAccountCommands = [
|
|
|
124
124
|
name: "accounts provision",
|
|
125
125
|
summary: "Prove a test account can get in, by replaying a plan that ends logged in",
|
|
126
126
|
description: "Runs the plan once in a real browser and, if it passes, marks the account ready. " +
|
|
127
|
-
"The plan is an ordinary ActionPlan that ends logged in
|
|
127
|
+
"The plan is an ordinary ActionPlan that ends logged in: a SIGNUP when the account " +
|
|
128
128
|
"does not exist yet, or a SIGN-IN when it already does (you created it by hand, or " +
|
|
129
129
|
"the site already had it). Either proves the same thing, and a sign-in is what every " +
|
|
130
130
|
"later test will do anyway. Type {{mailbox_address}} into the email field and " +
|
|
131
|
-
"{{login_password}} into the password field
|
|
131
|
+
"{{login_password}} into the password field. Both resolve at replay time, and an " +
|
|
132
132
|
"`await_email` step reads the project mailbox, so a verification code works.",
|
|
133
133
|
scope: "project",
|
|
134
134
|
args: [{ name: "account-id", description: "Account id from `beryl accounts list`", required: true }],
|
|
@@ -170,7 +170,7 @@ export const testAccountCommands = [
|
|
|
170
170
|
"field; an emailed code or magic link arrives at the account's own address, so an " +
|
|
171
171
|
"`await_email` step reads it with no human involved.\n\n" +
|
|
172
172
|
"--probe is the liveness check: a two-step plan (goto a gated page, then a POSITIVE " +
|
|
173
|
-
"assertion that only holds when signed in
|
|
173
|
+
"assertion that only holds when signed in: the account menu, a 'Sign out' control). " +
|
|
174
174
|
"It is replayed in a fresh browser carrying only the captured session. It is " +
|
|
175
175
|
"REQUIRED, and not a formality: assertions like `hidden`, `count 0`, and a URL " +
|
|
176
176
|
"match on a redirect all pass against a logged-out page, so without a positive " +
|
package/dist/commands/auth.js
CHANGED
|
@@ -97,7 +97,7 @@ export const authCommands = [
|
|
|
97
97
|
summary: "Authenticate the CLI with your Beryl account",
|
|
98
98
|
description: "Signs in via your browser (a code you confirm at beryl.so) and stores a personal " +
|
|
99
99
|
"access token in the CLI config. Pass --otp (or --email) to skip the browser and " +
|
|
100
|
-
"sign in with an emailed one-time code instead
|
|
100
|
+
"sign in with an emailed one-time code instead. A new email gets an account " +
|
|
101
101
|
"created automatically. Pass --token to use an existing token from Account → API " +
|
|
102
102
|
"tokens. Pass --email plus --code (the 6 digits from the email, e.g. read from a " +
|
|
103
103
|
"`beryl inbox`) to complete the OTP flow without a prompt. In CI, prefer the " +
|
|
@@ -108,18 +108,18 @@ export const authCommands = [
|
|
|
108
108
|
{
|
|
109
109
|
name: "otp",
|
|
110
110
|
type: "boolean",
|
|
111
|
-
description: "Skip the browser
|
|
111
|
+
description: "Skip the browser and sign in with an emailed code",
|
|
112
112
|
},
|
|
113
113
|
{
|
|
114
114
|
name: "email",
|
|
115
115
|
type: "string",
|
|
116
116
|
description: "Email for the one-time code sign-in (default: your last sign-in, or your git " +
|
|
117
|
-
"user.email
|
|
117
|
+
"user.email, offered as the prompt default)",
|
|
118
118
|
},
|
|
119
119
|
{
|
|
120
120
|
name: "code",
|
|
121
121
|
type: "string",
|
|
122
|
-
description: "The emailed 6-digit code
|
|
122
|
+
description: "The emailed 6-digit code. Skips the prompt for non-interactive use " +
|
|
123
123
|
"(requires --email; also skips sending a fresh code, so pair it with a " +
|
|
124
124
|
"code already requested via `beryl login`, `beryl signup`, or the API)",
|
|
125
125
|
},
|
|
@@ -184,7 +184,7 @@ export const authCommands = [
|
|
|
184
184
|
name: "signup",
|
|
185
185
|
summary: "Register a new Beryl account (emails a 6-digit verification code)",
|
|
186
186
|
description: "Creates a passwordless account for the email and sends it a 6-digit code. " +
|
|
187
|
-
"Finish with `beryl login --email <addr> --code <the 6 digits
|
|
187
|
+
"Finish with `beryl login --email <addr> --code <the 6 digits>`, which verifies " +
|
|
188
188
|
"the account, creates its workspace, and signs the CLI in. With an inbox from " +
|
|
189
189
|
"`beryl inbox create` as the address, an agent can provision a fresh account " +
|
|
190
190
|
"end-to-end with no human at a prompt.",
|
|
@@ -57,8 +57,8 @@ export const configCommands = [
|
|
|
57
57
|
{
|
|
58
58
|
name: "config vars get",
|
|
59
59
|
summary: "Show one config variable",
|
|
60
|
-
description: "Returns the variable row with its value in plaintext (variables are not secret)
|
|
61
|
-
"
|
|
60
|
+
description: "Returns the variable row with its value in plaintext (variables are not secret). " +
|
|
61
|
+
"A sensitive value lives in `config secrets`, readable only via `config secrets " +
|
|
62
62
|
"get --reveal`.",
|
|
63
63
|
scope: "project",
|
|
64
64
|
args: [{ name: "key", description: "Variable key or id", required: true }],
|
|
@@ -116,15 +116,15 @@ export const configCommands = [
|
|
|
116
116
|
{
|
|
117
117
|
name: "config secrets get",
|
|
118
118
|
summary: "Show one secret's metadata, or reveal its value with --reveal",
|
|
119
|
-
description: "Returns metadata only by default; --reveal is a logged, member-gated decrypt
|
|
120
|
-
"unlike `config vars get`, which returns its value in plaintext.",
|
|
119
|
+
description: "Returns metadata only by default; --reveal is a logged, member-gated decrypt " +
|
|
120
|
+
"(unlike `config vars get`, which returns its value in plaintext).",
|
|
121
121
|
scope: "project",
|
|
122
122
|
args: [{ name: "key", description: "Secret key or id", required: true }],
|
|
123
123
|
flags: [
|
|
124
124
|
{
|
|
125
125
|
name: "reveal",
|
|
126
126
|
type: "boolean",
|
|
127
|
-
description: "Return the decrypted value
|
|
127
|
+
description: "Return the decrypted value: the explicit read that lets the agent drive a " +
|
|
128
128
|
"real login while authoring. Member-gated; every reveal is logged",
|
|
129
129
|
},
|
|
130
130
|
{
|
|
@@ -1,12 +1,24 @@
|
|
|
1
1
|
import { UsageError } from "../errors.js";
|
|
2
2
|
import { arg, findGroup, flagBool, flagNum, flagStr, projectPath } from "./util.js";
|
|
3
|
-
const
|
|
3
|
+
const SCHEDULE_GROUPS_FLAG = "Comma-separated group names or ids the schedule targets (the run covers the union of their active tests at fire time); omit to run every active test";
|
|
4
|
+
const SCHEDULE_CADENCE_FLAGS = [
|
|
5
|
+
{
|
|
6
|
+
name: "frequency",
|
|
7
|
+
type: "string",
|
|
8
|
+
enum: ["daily", "weekly"],
|
|
9
|
+
description: "How often (default daily)",
|
|
10
|
+
},
|
|
11
|
+
{ name: "day", type: "number", description: "Weekly only: day of week, 0=Monday … 6=Sunday" },
|
|
12
|
+
{ name: "hour", type: "number", description: "Hour of day 0-23" },
|
|
13
|
+
{ name: "minute", type: "number", description: "Minute 0-59" },
|
|
14
|
+
{ name: "tz", type: "string", description: "IANA timezone (e.g. America/Los_Angeles)" },
|
|
15
|
+
];
|
|
4
16
|
export const environmentCommands = [
|
|
5
17
|
{
|
|
6
18
|
name: "envs list",
|
|
7
19
|
summary: "List a project's environments",
|
|
8
20
|
scope: "project",
|
|
9
|
-
groupSummary: "Manage a project's environments
|
|
21
|
+
groupSummary: "Manage a project's environments: the URLs and auth Beryl runs tests against.",
|
|
10
22
|
async run(ctx, input) {
|
|
11
23
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
12
24
|
return { data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/environments`) };
|
|
@@ -92,88 +104,110 @@ export const environmentCommands = [
|
|
|
92
104
|
},
|
|
93
105
|
},
|
|
94
106
|
{
|
|
95
|
-
name: "schedule
|
|
96
|
-
summary: "
|
|
107
|
+
name: "schedule list",
|
|
108
|
+
summary: "List the project's schedules",
|
|
97
109
|
scope: "project",
|
|
98
|
-
groupSummary: "
|
|
110
|
+
groupSummary: "Manage the schedules on which Beryl runs a project's tests automatically. A project can hold many independent schedules; each has its own cadence (daily or weekly), local run time, timezone, and target groups (none = every active test).",
|
|
99
111
|
async run(ctx, input) {
|
|
100
112
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
101
|
-
return { data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/
|
|
113
|
+
return { data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/schedules`) };
|
|
102
114
|
},
|
|
103
115
|
},
|
|
104
116
|
{
|
|
105
|
-
name: "schedule
|
|
106
|
-
summary: "
|
|
117
|
+
name: "schedule add",
|
|
118
|
+
summary: "Add a schedule (daily, or weekly on a given day)",
|
|
107
119
|
scope: "project",
|
|
108
120
|
flags: [
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
type: "string",
|
|
112
|
-
enum: ["daily", "weekly"],
|
|
113
|
-
description: "How often (default daily)",
|
|
114
|
-
},
|
|
115
|
-
{ name: "day", type: "number", description: "Weekly only: day of week, 0=Monday … 6=Sunday" },
|
|
116
|
-
{ name: "hour", type: "number", description: "Hour of day 0-23" },
|
|
117
|
-
{ name: "minute", type: "number", description: "Minute 0-59" },
|
|
118
|
-
{ name: "tz", type: "string", description: "IANA timezone (e.g. America/Los_Angeles)" },
|
|
119
|
-
{ name: "group", type: "string", description: SCHEDULE_GROUP_FLAG },
|
|
121
|
+
...SCHEDULE_CADENCE_FLAGS,
|
|
122
|
+
{ name: "groups", type: "string", description: SCHEDULE_GROUPS_FLAG },
|
|
120
123
|
],
|
|
121
124
|
examples: [
|
|
122
|
-
"beryl schedule
|
|
123
|
-
|
|
125
|
+
"beryl schedule add --frequency daily --hour 6 --tz UTC",
|
|
126
|
+
'beryl schedule add --groups "Smoke,Checkout" --frequency weekly --day 0 --hour 9 --minute 15 --tz UTC',
|
|
124
127
|
],
|
|
125
128
|
async run(ctx, input) {
|
|
126
129
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
127
130
|
return {
|
|
128
|
-
data: await ctx.client.
|
|
129
|
-
enabled: true,
|
|
131
|
+
data: await ctx.client.post(`${projectPath(workspaceId, projectId)}/schedules`, {
|
|
130
132
|
frequency: flagStr(input, "frequency") ?? "daily",
|
|
131
133
|
day_of_week: flagNum(input, "day") ?? null,
|
|
132
134
|
run_hour: flagNum(input, "hour"),
|
|
133
135
|
run_minute: flagNum(input, "minute"),
|
|
134
136
|
timezone: flagStr(input, "tz"),
|
|
137
|
+
group_ids: await resolveGroupIds(ctx, input, workspaceId, projectId),
|
|
135
138
|
}),
|
|
136
139
|
};
|
|
137
140
|
},
|
|
138
141
|
},
|
|
139
142
|
{
|
|
140
|
-
name: "schedule
|
|
141
|
-
summary: "
|
|
143
|
+
name: "schedule update",
|
|
144
|
+
summary: "Change a schedule's cadence, time, timezone, or groups",
|
|
142
145
|
scope: "project",
|
|
143
|
-
|
|
146
|
+
args: [{ name: "schedule-id", description: "Schedule id (see schedule list)", required: true }],
|
|
147
|
+
flags: [
|
|
148
|
+
...SCHEDULE_CADENCE_FLAGS,
|
|
149
|
+
{ name: "groups", type: "string", description: SCHEDULE_GROUPS_FLAG },
|
|
150
|
+
{
|
|
151
|
+
name: "all-tests",
|
|
152
|
+
type: "boolean",
|
|
153
|
+
description: "Target every active test (clears the schedule's groups)",
|
|
154
|
+
},
|
|
155
|
+
],
|
|
144
156
|
async run(ctx, input) {
|
|
145
157
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
158
|
+
const scheduleId = arg(input, "schedule-id");
|
|
159
|
+
const schedules = (await ctx.client.get(`${projectPath(workspaceId, projectId)}/schedules`));
|
|
160
|
+
const existing = schedules.find((sch) => sch.id === scheduleId);
|
|
161
|
+
if (!existing)
|
|
162
|
+
throw new UsageError(`No schedule '${scheduleId}' in this project.`);
|
|
163
|
+
if (flagBool(input, "all-tests") && flagStr(input, "groups"))
|
|
164
|
+
throw new UsageError("--all-tests and --groups are mutually exclusive");
|
|
165
|
+
const groupIds = flagBool(input, "all-tests")
|
|
166
|
+
? []
|
|
167
|
+
: flagStr(input, "groups")
|
|
168
|
+
? await resolveGroupIds(ctx, input, workspaceId, projectId)
|
|
169
|
+
: existing.group_ids;
|
|
146
170
|
return {
|
|
147
|
-
data: await ctx.client.put(
|
|
148
|
-
|
|
171
|
+
data: await ctx.client.put(`${projectPath(workspaceId, projectId)}/schedules/${scheduleId}`, {
|
|
172
|
+
frequency: flagStr(input, "frequency") ?? existing.frequency,
|
|
173
|
+
day_of_week: flagNum(input, "day") ?? existing.day_of_week,
|
|
174
|
+
run_hour: flagNum(input, "hour") ?? existing.run_hour,
|
|
175
|
+
run_minute: flagNum(input, "minute") ?? existing.run_minute,
|
|
176
|
+
timezone: flagStr(input, "tz") ?? existing.timezone,
|
|
177
|
+
group_ids: groupIds,
|
|
149
178
|
}),
|
|
150
179
|
};
|
|
151
180
|
},
|
|
152
181
|
},
|
|
153
182
|
{
|
|
154
183
|
name: "schedule remove",
|
|
155
|
-
summary: "Remove a
|
|
184
|
+
summary: "Remove a schedule (its tests and groups stay)",
|
|
156
185
|
scope: "project",
|
|
157
|
-
|
|
186
|
+
args: [{ name: "schedule-id", description: "Schedule id (see schedule list)", required: true }],
|
|
158
187
|
async run(ctx, input) {
|
|
159
188
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
160
|
-
|
|
161
|
-
throw new UsageError("--group is required");
|
|
162
|
-
await ctx.client.del(await schedulePath(ctx, input, workspaceId, projectId));
|
|
189
|
+
await ctx.client.del(`${projectPath(workspaceId, projectId)}/schedules/${arg(input, "schedule-id")}`);
|
|
163
190
|
return { human: "Removed." };
|
|
164
191
|
},
|
|
165
192
|
},
|
|
166
193
|
];
|
|
167
|
-
async function
|
|
168
|
-
const
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
194
|
+
async function resolveGroupIds(ctx, input, workspaceId, projectId) {
|
|
195
|
+
const raw = flagStr(input, "groups");
|
|
196
|
+
if (!raw)
|
|
197
|
+
return [];
|
|
198
|
+
const refs = raw
|
|
199
|
+
.split(",")
|
|
200
|
+
.map((r) => r.trim())
|
|
201
|
+
.filter(Boolean);
|
|
202
|
+
if (refs.length === 0)
|
|
203
|
+
return [];
|
|
172
204
|
const groups = (await ctx.client.get(`${projectPath(workspaceId, projectId)}/groups`));
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
205
|
+
return refs.map((ref) => {
|
|
206
|
+
const group = findGroup(groups, ref);
|
|
207
|
+
if (!group) {
|
|
208
|
+
const names = groups.map((g) => g.name).join(", ") || "none yet";
|
|
209
|
+
throw new UsageError(`No group '${ref}' in this project (existing: ${names}).`);
|
|
210
|
+
}
|
|
211
|
+
return group.id;
|
|
212
|
+
});
|
|
179
213
|
}
|
|
@@ -3,7 +3,7 @@ import { watchExploration } from "./watch.js";
|
|
|
3
3
|
export const explorationCommands = [
|
|
4
4
|
{
|
|
5
5
|
name: "explorations list",
|
|
6
|
-
groupSummary: "Inspect the agent's exploration runs
|
|
6
|
+
groupSummary: "Inspect the agent's exploration runs: how it crawled a site and authored its tests.",
|
|
7
7
|
summary: "List the agent's exploration passes for a project",
|
|
8
8
|
scope: "project",
|
|
9
9
|
async run(ctx, input) {
|
|
@@ -38,7 +38,7 @@ export const explorationCommands = [
|
|
|
38
38
|
{
|
|
39
39
|
name: "explorations cancel",
|
|
40
40
|
summary: "Cancel an in-flight exploration",
|
|
41
|
-
description: "Stops a running exploration without touching the project or its tests
|
|
41
|
+
description: "Stops a running exploration without touching the project or its tests: the " +
|
|
42
42
|
"lever for a runaway/non-convergent pass burning agent budget.",
|
|
43
43
|
scope: "project",
|
|
44
44
|
args: [{ name: "exploration-id", description: "Exploration id", required: true }],
|
|
@@ -51,7 +51,7 @@ export const explorationCommands = [
|
|
|
51
51
|
},
|
|
52
52
|
{
|
|
53
53
|
name: "explorations watch",
|
|
54
|
-
summary: "Stream an exploration live
|
|
54
|
+
summary: "Stream an exploration live: watch the agent explore and author tests",
|
|
55
55
|
description: "Replays every recorded step on connect, then follows live until the exploration " +
|
|
56
56
|
"completes or fails.",
|
|
57
57
|
scope: "project",
|
package/dist/commands/groups.js
CHANGED
|
@@ -15,7 +15,7 @@ export const groupCommands = [
|
|
|
15
15
|
summary: "List the project's test groups and how many tests each holds",
|
|
16
16
|
scope: "project",
|
|
17
17
|
groupDefault: true,
|
|
18
|
-
groupSummary: "Manage a project's test groups
|
|
18
|
+
groupSummary: "Manage a project's test groups: labels a test can carry any number of, used to filter, run, or schedule a slice of the suite. Groups are created only here (or in Settings → Groups); assigning a test to an unknown name is an error.",
|
|
19
19
|
async run(ctx, input) {
|
|
20
20
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
21
21
|
return { data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/groups`) };
|
|
@@ -54,7 +54,7 @@ export const groupCommands = [
|
|
|
54
54
|
},
|
|
55
55
|
{
|
|
56
56
|
name: "groups delete",
|
|
57
|
-
summary: "Delete a group
|
|
57
|
+
summary: "Delete a group: its tests stay; a schedule targeting only this group goes with it",
|
|
58
58
|
scope: "project",
|
|
59
59
|
args: [{ name: "group", description: "Group name or id", required: true }],
|
|
60
60
|
flags: [{ name: "force", type: "boolean", description: "Skip the confirmation prompt" }],
|
package/dist/commands/health.js
CHANGED
|
@@ -8,7 +8,7 @@ export const healthCommands = [
|
|
|
8
8
|
{
|
|
9
9
|
name: "health get",
|
|
10
10
|
groupDefault: true,
|
|
11
|
-
groupSummary: "Site Health
|
|
11
|
+
groupSummary: "Site Health, or how your site reads to search engines and visitors: content, " +
|
|
12
12
|
"speed, mobile, links and security, graded from a real check of your live pages.",
|
|
13
13
|
summary: "Show the latest Site Health report for a project environment",
|
|
14
14
|
description: "A check runs automatically when a project or environment gets its URL. While one " +
|
package/dist/commands/init.js
CHANGED
|
@@ -147,20 +147,20 @@ async function ensureLocalPlaywright(ctx, cwd) {
|
|
|
147
147
|
export const initCommands = [
|
|
148
148
|
{
|
|
149
149
|
name: "init",
|
|
150
|
-
summary: "Set up Beryl in this repo
|
|
150
|
+
summary: "Set up Beryl in this repo: sign in and wire up your coding agent",
|
|
151
151
|
description: "One-command onboarding: signs you in (browser confirm, or an emailed one-time code) " +
|
|
152
152
|
"and wires up your coding " +
|
|
153
|
-
"agent. Nothing is detected and nothing is conditional
|
|
153
|
+
"agent. Nothing is detected and nothing is conditional: every run wires the beryl AND " +
|
|
154
154
|
"playwright MCP servers, writes the authoring skill (user scope: your home " +
|
|
155
155
|
".agents/skills/ + .claude/skills/, so it follows you into every session; --scope " +
|
|
156
156
|
"project: the repo's own dirs, committed for teammates), and installs @playwright/test " +
|
|
157
|
-
"+ chromium if missing
|
|
157
|
+
"+ chromium if missing, since browser authoring and local runs depend on it. By default the " +
|
|
158
158
|
"servers are wired per-user (matching where your login token lives) via `claude mcp add " +
|
|
159
159
|
"-s user`; pass --scope project to write a committed .mcp.json for a shared repo " +
|
|
160
|
-
"instead. No workspace/project pin and no URL prompt
|
|
160
|
+
"instead. No workspace/project pin and no URL prompt: ask Claude to write tests for " +
|
|
161
161
|
"your site and it resolves the workspace, project, and URL. Safe to re-run: every run " +
|
|
162
162
|
"refreshes the authoring skill to this CLI's version (overwriting an older or edited " +
|
|
163
|
-
"copy
|
|
163
|
+
"copy; `beryl guide` prints the same content), and skips whatever else is already set " +
|
|
164
164
|
"up.",
|
|
165
165
|
interactive: true,
|
|
166
166
|
flags: [
|
|
@@ -170,7 +170,7 @@ export const initCommands = [
|
|
|
170
170
|
enum: ["user", "project"],
|
|
171
171
|
description: "Where to wire the MCP servers. `user` (default) configures them per-user (matching " +
|
|
172
172
|
"where your Beryl login token lives) via `claude mcp add -s user`. `project` writes a " +
|
|
173
|
-
"committed .mcp.json for a shared repo
|
|
173
|
+
"committed .mcp.json for a shared repo. Every teammate still runs `beryl login` to " +
|
|
174
174
|
"authenticate",
|
|
175
175
|
},
|
|
176
176
|
{
|
|
@@ -285,8 +285,8 @@ export const initCommands = [
|
|
|
285
285
|
description: "The full guide to authoring durable, healable tests: the ActionPlan shape, outcome " +
|
|
286
286
|
"assertions, natural-language intent, test accounts and the project mailbox " +
|
|
287
287
|
"({{login_email}}, {{mailbox_address}}, {{inbox_address}} + await_email), and the " +
|
|
288
|
-
"local run-fix loop. Same content as the beryl-test skill `beryl init` installs
|
|
289
|
-
`
|
|
288
|
+
"local run-fix loop. Same content as the beryl-test skill `beryl init` installs. " +
|
|
289
|
+
`Skip if a loaded beryl-test skill states v${cliVersion()}; else call this first ` +
|
|
290
290
|
"(works without logging in).",
|
|
291
291
|
examples: ["beryl guide"],
|
|
292
292
|
async run() {
|
|
@@ -7,10 +7,10 @@ export const mailboxCommands = [
|
|
|
7
7
|
name: "mailbox get",
|
|
8
8
|
summary: "The project's mailbox address",
|
|
9
9
|
groupDefault: true,
|
|
10
|
-
groupSummary: "The project's standing email addresses
|
|
10
|
+
groupSummary: "The project's standing email addresses, where its tests receive sign-in mail.",
|
|
11
11
|
description: "Returns the address {{mailbox_address}} resolves to, creating it on first ask. " +
|
|
12
12
|
"Every test that reads mail receives here. A test needing an address the site has " +
|
|
13
|
-
"never seen cites {{inbox_address}} instead
|
|
13
|
+
"never seen cites {{inbox_address}} instead. That renders a `+tag` alias of this " +
|
|
14
14
|
"same mailbox, so a signup stays repeatable without a second address to manage.",
|
|
15
15
|
scope: "project",
|
|
16
16
|
async run(ctx, input) {
|
|
@@ -46,7 +46,7 @@ export const mailboxCommands = [
|
|
|
46
46
|
name: "mailbox create",
|
|
47
47
|
summary: "Add a second mailbox to the project",
|
|
48
48
|
description: "Only needed when a flow requires two genuinely separate inboxes at the same time " +
|
|
49
|
-
"
|
|
49
|
+
"(both sides of an invite handshake). For an address the site has not seen before, " +
|
|
50
50
|
"cite {{inbox_address}} in the plan instead: it aliases the existing mailbox and " +
|
|
51
51
|
"costs nothing to manage.",
|
|
52
52
|
scope: "project",
|
|
@@ -88,12 +88,12 @@ export const mailboxCommands = [
|
|
|
88
88
|
name: "mailbox read",
|
|
89
89
|
summary: "Read the latest email in a mailbox (waits for one to arrive)",
|
|
90
90
|
description: "Waits up to --timeout-s for a matching email and returns it (one blocking request; " +
|
|
91
|
-
"the server caps the wait at 50s
|
|
91
|
+
"the server caps the wait at 50s, so re-run to keep waiting). With --extract-code, " +
|
|
92
92
|
"also asks the server to pull the one-time code out of the email (AI-assisted when " +
|
|
93
93
|
"the email is ambiguous; `code` is null if none was found). Use " +
|
|
94
94
|
"--recipient-contains to read only one `+tag` alias's mail when several identities " +
|
|
95
95
|
"share the mailbox. Exits non-zero if nothing arrives before the timeout. Waits for " +
|
|
96
|
-
"and returns ONE latest matching email
|
|
96
|
+
"and returns ONE latest matching email. `mailbox emails` lists what has already " +
|
|
97
97
|
"arrived, without waiting.",
|
|
98
98
|
scope: "project",
|
|
99
99
|
args: [{ name: "mailbox-id", description: "Mailbox id from `beryl mailbox list`", required: true }],
|
|
@@ -154,7 +154,7 @@ export const mailboxCommands = [
|
|
|
154
154
|
{
|
|
155
155
|
name: "mailbox emails",
|
|
156
156
|
summary: "List the emails a mailbox has received",
|
|
157
|
-
description: "Returns the already-received emails without waiting
|
|
157
|
+
description: "Returns the already-received emails without waiting. `mailbox read` blocks for " +
|
|
158
158
|
"a matching one and returns just it.",
|
|
159
159
|
scope: "project",
|
|
160
160
|
args: [{ name: "mailbox-id", description: "Mailbox id from `beryl mailbox list`", required: true }],
|
package/dist/commands/mcp.js
CHANGED
|
@@ -3,9 +3,9 @@ export const mcpCommands = [
|
|
|
3
3
|
{
|
|
4
4
|
name: "version",
|
|
5
5
|
summary: "Show the running CLI version, API URL, and Node version",
|
|
6
|
-
description: "Answers \"which build am I actually talking to?\"
|
|
6
|
+
description: "Answers \"which build am I actually talking to?\" (the one question a long-lived " +
|
|
7
7
|
"`beryl mcp` process can't otherwise answer, since it loads source at spawn and never " +
|
|
8
|
-
"hot-reloads. Needs no login, so it still works when a token is missing or broken. " +
|
|
8
|
+
"hot-reloads). Needs no login, so it still works when a token is missing or broken. " +
|
|
9
9
|
"Also reports whether the running build is behind npm's `latest` (best-effort: " +
|
|
10
10
|
"`update_available` is null when the registry can't be reached, and the check is " +
|
|
11
11
|
"skipped entirely under BERYL_NO_UPDATE_CHECK).",
|
|
@@ -26,7 +26,7 @@ export const mcpCommands = [
|
|
|
26
26
|
},
|
|
27
27
|
{
|
|
28
28
|
name: "mcp",
|
|
29
|
-
summary: "Run the Beryl MCP server (stdio)
|
|
29
|
+
summary: "Run the Beryl MCP server (stdio): every CLI command as an agent tool",
|
|
30
30
|
description: "Exposes the CLI's commands as MCP tools over stdio, so coding agents (Claude Code, " +
|
|
31
31
|
"Cursor, …) can create projects, trigger runs, watch the agent, and edit tests. " +
|
|
32
32
|
"Authenticate via BERYL_API_KEY or a prior `beryl login`.",
|
|
@@ -37,7 +37,7 @@ export const projectCommands = [
|
|
|
37
37
|
summary: "List projects in the workspace",
|
|
38
38
|
scope: "workspace",
|
|
39
39
|
groupDefault: true,
|
|
40
|
-
groupSummary: "Create and manage projects
|
|
40
|
+
groupSummary: "Create and manage projects: a site Beryl explores, authors tests for, and runs.",
|
|
41
41
|
async run(ctx, input) {
|
|
42
42
|
const ws = await ctx.requireWorkspace(input);
|
|
43
43
|
const rows = (await ctx.client.get(`/workspaces/${ws}/projects`));
|
|
@@ -70,7 +70,7 @@ export const projectCommands = [
|
|
|
70
70
|
},
|
|
71
71
|
{
|
|
72
72
|
name: "projects create",
|
|
73
|
-
summary: "Create a project
|
|
73
|
+
summary: "Create a project. With a URL the agent starts exploring; with just --name an empty one",
|
|
74
74
|
scope: "workspace",
|
|
75
75
|
args: [
|
|
76
76
|
{
|
|
@@ -91,13 +91,13 @@ export const projectCommands = [
|
|
|
91
91
|
type: "string",
|
|
92
92
|
enum: ["public", "gated"],
|
|
93
93
|
description: "Whether the site needs a login (gated) or not (public). Default: detected from the " +
|
|
94
|
-
"site
|
|
94
|
+
"site, and only asked when detection is genuinely unsure",
|
|
95
95
|
},
|
|
96
96
|
{ name: "force-new-login", type: "boolean", description: "Ignore any reusable saved login" },
|
|
97
97
|
{
|
|
98
98
|
name: "no-explore",
|
|
99
99
|
type: "boolean",
|
|
100
|
-
description: "Create the project without starting the cloud exploration
|
|
100
|
+
description: "Create the project without starting the cloud exploration. Author tests yourself " +
|
|
101
101
|
"via `beryl tests create` or your coding agent over MCP",
|
|
102
102
|
},
|
|
103
103
|
{ name: "watch", type: "boolean", description: "Stream the agent's exploration live" },
|
|
@@ -190,7 +190,7 @@ export const projectCommands = [
|
|
|
190
190
|
},
|
|
191
191
|
{
|
|
192
192
|
name: "projects re-explore",
|
|
193
|
-
summary: "Send the agent back in
|
|
193
|
+
summary: "Send the agent back in to run/heal existing tests and discover new flows",
|
|
194
194
|
scope: "project",
|
|
195
195
|
flags: [
|
|
196
196
|
{ name: "watch", type: "boolean", description: "Stream the agent's exploration live" },
|
package/dist/commands/runs.js
CHANGED
|
@@ -50,7 +50,7 @@ export const runCommands = [
|
|
|
50
50
|
name: "runs trigger",
|
|
51
51
|
summary: "Trigger a test run (whole suite, a subset, or one environment)",
|
|
52
52
|
description: "Runs execute in Beryl's cloud. With --watch the CLI streams live progress and " +
|
|
53
|
-
"exits 0 only if every test passed
|
|
53
|
+
"exits 0 only if every test passed, so wire it straight into CI. A run with a " +
|
|
54
54
|
"heal-eligible failure completes only after Beryl has tried to heal it: a repaired " +
|
|
55
55
|
"test is re-run inside the same run and counts as passed (reported as `healed`), so " +
|
|
56
56
|
"the final counts, the report and the completion email all reflect the repair.",
|
|
@@ -60,7 +60,7 @@ export const runCommands = [
|
|
|
60
60
|
{
|
|
61
61
|
name: "group",
|
|
62
62
|
type: "string",
|
|
63
|
-
description: "Run only the active tests in this group (by name)
|
|
63
|
+
description: "Run only the active tests in this group (by name), resolved to ids before the run, " +
|
|
64
64
|
"so the run records exactly what it ran. Cannot be combined with --test.",
|
|
65
65
|
},
|
|
66
66
|
{ name: "env", type: "string", description: "Environment id to run against" },
|
|
@@ -116,7 +116,7 @@ export const runCommands = [
|
|
|
116
116
|
{
|
|
117
117
|
name: "runs local",
|
|
118
118
|
summary: "Run tests on your machine with your own Playwright; results sync to Beryl",
|
|
119
|
-
description: "Unlike `runs trigger`, the browser runs on YOUR machine
|
|
119
|
+
description: "Unlike `runs trigger`, the browser runs on YOUR machine: each test's rendered spec is " +
|
|
120
120
|
"fetched and run with your local @playwright/test and the Chromium binary it drives. " +
|
|
121
121
|
"Both are checked once before any spec is fetched, so a machine that can't run tests " +
|
|
122
122
|
"says so once instead of failing every test (on a terminal the CLI offers to install " +
|
|
@@ -129,12 +129,12 @@ export const runCommands = [
|
|
|
129
129
|
"does, and scrubbed from any error text or DOM snapshot before results upload; a test " +
|
|
130
130
|
"whose account has no stored password is skipped with the exact fix-it command. When the " +
|
|
131
131
|
"run finishes, the results and replay artifacts are imported into Beryl as a normal run " +
|
|
132
|
-
"(trigger source `local`)
|
|
132
|
+
"(trigger source `local`), so history, replay, and reports all work; pass --no-sync to " +
|
|
133
133
|
"keep a run entirely off the record while iterating. Session-mode tests behave as " +
|
|
134
134
|
"in the cloud: their account signs in once per invocation and every session-mode " +
|
|
135
135
|
"test rides that session; a failed sign-in fails those tests with the same " +
|
|
136
136
|
"SESSION_* reason a cloud run reports. Only a test that depends on a session Beryl " +
|
|
137
|
-
"captured server-side is skipped, with a note
|
|
137
|
+
"captured server-side is skipped, with a note. Run those with `runs trigger`. Point " +
|
|
138
138
|
"--url-override at a local dev server or preview, and --dir to keep specs, artifacts, " +
|
|
139
139
|
"and reports on disk. Exits 0 only if every executed test passed.",
|
|
140
140
|
scope: "project",
|
|
@@ -164,7 +164,7 @@ export const runCommands = [
|
|
|
164
164
|
{
|
|
165
165
|
name: "env",
|
|
166
166
|
type: "string",
|
|
167
|
-
description: "Environment id to run against and attach the imported run to
|
|
167
|
+
description: "Environment id to run against and attach the imported run to. Use for a " +
|
|
168
168
|
"standing environment; --url-override is for a throwaway host",
|
|
169
169
|
},
|
|
170
170
|
{
|
|
@@ -531,8 +531,8 @@ export const runCommands = [
|
|
|
531
531
|
summary: "Show one run with its per-test results",
|
|
532
532
|
description: "Over MCP the failure screenshots come back as viewable image content, so an agent can " +
|
|
533
533
|
"look at the page that broke instead of guessing from the error string. Set screenshots " +
|
|
534
|
-
"to false to skip fetching them. Ignored outside MCP
|
|
535
|
-
"Returns the run row with its per-test results
|
|
534
|
+
"to false to skip fetching them. Ignored outside MCP (the terminal cannot show an image). " +
|
|
535
|
+
"Returns the run row with its per-test results. `runs report` returns the generated " +
|
|
536
536
|
"report document, `runs explain` an AI explanation of one failed result.",
|
|
537
537
|
scope: "project",
|
|
538
538
|
args: [{ name: "run-id", description: "Run id", required: true }],
|
|
@@ -585,7 +585,7 @@ export const runCommands = [
|
|
|
585
585
|
{
|
|
586
586
|
name: "runs report",
|
|
587
587
|
summary: "Show the generated report for a run",
|
|
588
|
-
description: "Returns the run's stored generated report (404 until it has been generated)
|
|
588
|
+
description: "Returns the run's stored generated report (404 until it has been generated). " +
|
|
589
589
|
"`runs get` returns the raw run row with per-test results.",
|
|
590
590
|
scope: "project",
|
|
591
591
|
args: [{ name: "run-id", description: "Run id", required: true }],
|
|
@@ -599,8 +599,8 @@ export const runCommands = [
|
|
|
599
599
|
{
|
|
600
600
|
name: "runs download",
|
|
601
601
|
summary: "Download a run's results, with its artifacts, to disk",
|
|
602
|
-
description: "With --dir, fetches the artifact bytes
|
|
603
|
-
"trace zip, and the filmstrip frames of the failing tests
|
|
602
|
+
description: "With --dir, fetches the artifact bytes (screenshots, DOM snapshots, the Playwright " +
|
|
603
|
+
"trace zip, and the filmstrip frames of the failing tests) into <dir>/<test-result-id>/ " +
|
|
604
604
|
"alongside a run.json manifest. Artifact URLs are short-lived, so download rather than " +
|
|
605
605
|
"stash them. With --out (or neither), writes only the JSON manifest.",
|
|
606
606
|
scope: "project",
|
|
@@ -667,7 +667,7 @@ export const runCommands = [
|
|
|
667
667
|
name: "runs explain",
|
|
668
668
|
summary: "Explain, with AI, why a test result failed",
|
|
669
669
|
description: "Takes a single test-RESULT id (not a run id) and returns an AI failure " +
|
|
670
|
-
"explanation for that result
|
|
670
|
+
"explanation for that result. `runs get` lists a run's results and their ids.",
|
|
671
671
|
scope: "project",
|
|
672
672
|
args: [{ name: "result-id", description: "Test result id (from `beryl runs get`)", required: true }],
|
|
673
673
|
async run(ctx, input) {
|
package/dist/commands/tests.js
CHANGED
|
@@ -80,7 +80,7 @@ export const testCommands = [
|
|
|
80
80
|
{
|
|
81
81
|
name: "tests lint",
|
|
82
82
|
summary: "Validate a plan JSON file offline, before sending it to the server",
|
|
83
|
-
description: "Checks a plan against the published ActionPlan JSON Schema
|
|
83
|
+
description: "Checks a plan against the published ActionPlan JSON Schema: every action's required " +
|
|
84
84
|
"fields, plus the two structural rules (the first EXECUTED step must be a goto, and at " +
|
|
85
85
|
"least one step across before + steps must be an expect). Runs entirely locally, so a " +
|
|
86
86
|
"malformed plan fails here instead of costing a server round-trip. " +
|
|
@@ -110,7 +110,7 @@ export const testCommands = [
|
|
|
110
110
|
"unless --page is given; pass --page to walk a large project a slice at a time.",
|
|
111
111
|
scope: "project",
|
|
112
112
|
groupDefault: true,
|
|
113
|
-
groupSummary: "Author, inspect, version, and heal a project's tests
|
|
113
|
+
groupSummary: "Author, inspect, version, and heal a project's tests: the checks Beryl runs on each run.",
|
|
114
114
|
flags: [
|
|
115
115
|
{ name: "env", type: "string", description: "Filter by environment id" },
|
|
116
116
|
{
|
|
@@ -139,6 +139,11 @@ export const testCommands = [
|
|
|
139
139
|
type: "boolean",
|
|
140
140
|
description: "With --page, show only tests carrying no group",
|
|
141
141
|
},
|
|
142
|
+
{
|
|
143
|
+
name: "q",
|
|
144
|
+
type: "string",
|
|
145
|
+
description: "With --page, show only tests whose title contains this text (case-insensitive)",
|
|
146
|
+
},
|
|
142
147
|
],
|
|
143
148
|
async run(ctx, input) {
|
|
144
149
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
@@ -159,6 +164,7 @@ export const testCommands = [
|
|
|
159
164
|
page_size: flagNum(input, "page-size"),
|
|
160
165
|
status: flagStr(input, "status"),
|
|
161
166
|
group: flagBool(input, "ungrouped") ? "__ungrouped__" : group,
|
|
167
|
+
q: flagStr(input, "q"),
|
|
162
168
|
environment_id: flagStr(input, "env"),
|
|
163
169
|
})).items;
|
|
164
170
|
}
|
|
@@ -172,7 +178,7 @@ export const testCommands = [
|
|
|
172
178
|
name: "tests get",
|
|
173
179
|
summary: "Show one test",
|
|
174
180
|
description: "Returns the test's metadata row (status, flags, per-environment last result), not " +
|
|
175
|
-
"the plan
|
|
181
|
+
"the plan. `tests plan` prints the stored JSON plan, `tests script` the rendered " +
|
|
176
182
|
"Playwright spec.",
|
|
177
183
|
scope: "project",
|
|
178
184
|
args: [{ name: "test-id", description: "Test id", required: true }],
|
|
@@ -184,7 +190,7 @@ export const testCommands = [
|
|
|
184
190
|
{
|
|
185
191
|
name: "tests plan",
|
|
186
192
|
summary: "Print a test's current step plan (JSON)",
|
|
187
|
-
description: "Returns the stored json_plan of the test's current version
|
|
193
|
+
description: "Returns the stored json_plan of the test's current version. `tests get` returns " +
|
|
188
194
|
"the metadata row, `tests script` the rendered Playwright spec.",
|
|
189
195
|
scope: "project",
|
|
190
196
|
args: [{ name: "test-id", description: "Test id", required: true }],
|
|
@@ -198,30 +204,30 @@ export const testCommands = [
|
|
|
198
204
|
},
|
|
199
205
|
{
|
|
200
206
|
name: "tests create",
|
|
201
|
-
summary: "Create a test case from a JSON action plan
|
|
207
|
+
summary: "Create a test case from a JSON action plan (for tests authored locally, e.g. by your coding agent)",
|
|
202
208
|
description: "The plan is a JSON object whose steps are {action, selector, url, value, ...}: the first " +
|
|
203
209
|
"EXECUTED step must be a goto, and at least one step must be an expect. Before anything is " +
|
|
204
210
|
"banked, the plan is proven by replaying it in a browser ON YOUR MACHINE with your local " +
|
|
205
|
-
"@playwright/test: the server renders the spec (`tests/compile`), the CLI runs it
|
|
211
|
+
"@playwright/test: the server renders the spec (`tests/compile`), the CLI runs it (minting " +
|
|
206
212
|
"a run inbox for await_email steps and resolving the saved login exactly as a cloud run " +
|
|
207
|
-
"would
|
|
213
|
+
"would), and only a green replay creates the test (bound to the replayed plan by its hash). " +
|
|
208
214
|
"This holds over MCP too: the replay runs on the machine hosting the MCP server, never on " +
|
|
209
215
|
"Beryl's; if @playwright/test is missing there the tool returns the install commands (on a " +
|
|
210
216
|
"terminal the CLI offers to install it). " +
|
|
211
217
|
"A red replay banks NOTHING: the failure evidence comes back (over MCP the screenshot is " +
|
|
212
218
|
"image content), you fix the plan file and re-run. The proving run is imported as the " +
|
|
213
219
|
"test's first run (--no-sync to skip). A plan that signs in with a session Beryl captured " +
|
|
214
|
-
"server-side cannot replay locally (that session never leaves Beryl's cloud)
|
|
220
|
+
"server-side cannot replay locally (that session never leaves Beryl's cloud). It falls " +
|
|
215
221
|
"back to server-side verification automatically, and says so. A session-mode plan replays locally " +
|
|
216
222
|
"fine: the server renders it with its account's stored sign-in steps in front, so " +
|
|
217
223
|
"the same identity is exercised on your machine. Optional `before` and `after` arrays hold setup and teardown " +
|
|
218
224
|
"steps: `after` runs even when a main step fails, which is how a create/update/delete test " +
|
|
219
225
|
"cleans up the record it made on the runs that go red. " +
|
|
220
|
-
"Recovery: a 409 `duplicate_title` carries existing_test_id + existing_plan_hash
|
|
226
|
+
"Recovery: a 409 `duplicate_title` carries existing_test_id + existing_plan_hash, so " +
|
|
221
227
|
"reconcile with that test (`tests get` / `tests set-plan`), don't rename-and-retry; a " +
|
|
222
228
|
"409 `plan_hash_mismatch` means the submitted plan is not the bytes that were replayed " +
|
|
223
|
-
"
|
|
224
|
-
"saturated
|
|
229
|
+
"(re-run `tests create`); a 429 with Retry-After 30 means the verify slots are " +
|
|
230
|
+
"saturated (wait and retry).",
|
|
225
231
|
scope: "project",
|
|
226
232
|
flags: [
|
|
227
233
|
{ name: "title", type: "string", required: true, description: "Title for the new test" },
|
|
@@ -229,26 +235,26 @@ export const testCommands = [
|
|
|
229
235
|
{
|
|
230
236
|
name: "description",
|
|
231
237
|
type: "string",
|
|
232
|
-
description: "
|
|
238
|
+
description: "One to three sentences stating what this test proves: the immutable outcome Beryl's healing " +
|
|
233
239
|
"checks against. State the purpose, not the steps; the one observable signal that's true " +
|
|
234
240
|
"only if the flow worked.",
|
|
235
241
|
},
|
|
236
242
|
{
|
|
237
243
|
name: "no-verify",
|
|
238
244
|
type: "boolean",
|
|
239
|
-
description: "Skip verification entirely
|
|
245
|
+
description: "Skip verification entirely: bank the authored plan as-is, unproven",
|
|
240
246
|
},
|
|
241
247
|
{
|
|
242
248
|
name: "url-override",
|
|
243
249
|
type: "string",
|
|
244
250
|
description: "Replay against this base URL instead of the environment's (e.g. http://localhost:3000). " +
|
|
245
|
-
"The banked test is then unproven against its real environment
|
|
251
|
+
"The banked test is then unproven against its real environment, and the CLI says so.",
|
|
246
252
|
},
|
|
247
253
|
{
|
|
248
254
|
name: "group",
|
|
249
255
|
type: "strings",
|
|
250
256
|
description: "Put the test in an existing project group (by name), repeatable. An unknown " +
|
|
251
|
-
"name is an error
|
|
257
|
+
"name is an error (create groups with `beryl groups create`). Omit for no group.",
|
|
252
258
|
},
|
|
253
259
|
{ name: "env", type: "string", description: "Environment id to compile and prove against" },
|
|
254
260
|
{
|
|
@@ -480,9 +486,9 @@ export const testCommands = [
|
|
|
480
486
|
name: "tests set-plan",
|
|
481
487
|
summary: "Replace a test's step plan from a JSON file (creates a new version)",
|
|
482
488
|
description: "Accepts the same plan shape as `tests create`, including the optional `before` and " +
|
|
483
|
-
"`after` sections
|
|
489
|
+
"`after` sections. `after` runs on pass and on fail, so cleanup happens even when the " +
|
|
484
490
|
"test goes red. Pass `--description` when the re-authored plan changes what the test " +
|
|
485
|
-
"proves; omit it to keep the test's existing intent. Saves the edit with NO replay
|
|
491
|
+
"proves; omit it to keep the test's existing intent. Saves the edit with NO replay: " +
|
|
486
492
|
"it rides into the next run unproven; `tests recompile` is the verify-first " +
|
|
487
493
|
"alternative.",
|
|
488
494
|
scope: "project",
|
|
@@ -492,7 +498,7 @@ export const testCommands = [
|
|
|
492
498
|
{
|
|
493
499
|
name: "description",
|
|
494
500
|
type: "string",
|
|
495
|
-
description: "
|
|
501
|
+
description: "One to three sentences stating what this test proves: the immutable outcome Beryl's healing " +
|
|
496
502
|
"checks against. State the purpose, not the steps; the one observable signal that's true " +
|
|
497
503
|
"only if the flow worked. Omit to keep the test's existing intent.",
|
|
498
504
|
},
|
|
@@ -527,7 +533,7 @@ export const testCommands = [
|
|
|
527
533
|
name: "tests set-groups",
|
|
528
534
|
summary: "Replace the groups a test belongs to",
|
|
529
535
|
description: "Groups are project-defined labels used to filter, run, or schedule a slice of the " +
|
|
530
|
-
"suite (`beryl runs trigger --group <name>`). This REPLACES the whole set
|
|
536
|
+
"suite (`beryl runs trigger --group <name>`). This REPLACES the whole set: pass every " +
|
|
531
537
|
"group you want, or none to clear it. Names must already exist in the project " +
|
|
532
538
|
"(`beryl groups list`); an unknown name is an error, never a new group.",
|
|
533
539
|
scope: "project",
|
|
@@ -552,16 +558,49 @@ export const testCommands = [
|
|
|
552
558
|
};
|
|
553
559
|
},
|
|
554
560
|
},
|
|
561
|
+
{
|
|
562
|
+
name: "tests add-groups",
|
|
563
|
+
summary: "Add groups to one or more tests, keeping the groups they already have",
|
|
564
|
+
description: "The additive counterpart of `tests set-groups`: every listed test gains the given " +
|
|
565
|
+
"groups and loses none, so you don't need to know what each test already carries. " +
|
|
566
|
+
"Names must already exist in the project (`beryl groups list`); an unknown name is " +
|
|
567
|
+
"an error, never a new group.",
|
|
568
|
+
scope: "project",
|
|
569
|
+
args: [{ name: "test-ids", description: "One or more test ids", required: true, variadic: true }],
|
|
570
|
+
flags: [
|
|
571
|
+
{
|
|
572
|
+
name: "group",
|
|
573
|
+
type: "strings",
|
|
574
|
+
description: "Group name to add, repeatable (at least one)",
|
|
575
|
+
},
|
|
576
|
+
],
|
|
577
|
+
examples: ["beryl tests add-groups 4f… 9a… --group Smoke", "beryl tests add-groups 4f… --group Checkout --group Smoke"],
|
|
578
|
+
async run(ctx, input) {
|
|
579
|
+
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
580
|
+
const add = flagStrings(input, "group");
|
|
581
|
+
if (!add)
|
|
582
|
+
throw new UsageError("Pass at least one --group");
|
|
583
|
+
const ids = argList(input, "test-ids");
|
|
584
|
+
const data = (await ctx.client.post(`${projectPath(workspaceId, projectId)}/tests/bulk-groups`, {
|
|
585
|
+
test_case_ids: ids,
|
|
586
|
+
add,
|
|
587
|
+
}));
|
|
588
|
+
return {
|
|
589
|
+
data,
|
|
590
|
+
human: `Added ${add.join(", ")} to ${data.updated} of ${ids.length} test${ids.length === 1 ? "" : "s"} (the rest already had them).`,
|
|
591
|
+
};
|
|
592
|
+
},
|
|
593
|
+
},
|
|
555
594
|
{
|
|
556
595
|
name: "tests quarantine",
|
|
557
596
|
summary: "Mute a flaky test: it keeps running, but its failures stop failing the run",
|
|
558
|
-
description: "A quarantined test still executes and its result is still recorded and visible
|
|
597
|
+
description: "A quarantined test still executes and its result is still recorded and visible. Its " +
|
|
559
598
|
"red lands in the run's quarantined_count and gates neither the run's verdict nor exit " +
|
|
560
599
|
"codes, so it can't red-light a deploy. Use " +
|
|
561
600
|
"it on a persistently flaky test instead of deleting it (which destroys the history) or " +
|
|
562
601
|
"asking support to deactivate it (which stops it running at all). After 5 consecutive " +
|
|
563
|
-
"clean passes the test reports rehab_ready
|
|
564
|
-
"itself. `off` un-quarantines.",
|
|
602
|
+
"clean passes the test reports rehab_ready (advisory only, nothing un-quarantines " +
|
|
603
|
+
"itself). `off` un-quarantines.",
|
|
565
604
|
scope: "project",
|
|
566
605
|
args: [
|
|
567
606
|
{ name: "test-id", description: "Test id", required: true },
|
|
@@ -581,16 +620,28 @@ export const testCommands = [
|
|
|
581
620
|
},
|
|
582
621
|
{
|
|
583
622
|
name: "tests delete",
|
|
584
|
-
summary: "Delete
|
|
623
|
+
summary: "Delete tests, their version history, and their results",
|
|
624
|
+
description: "Several ids are deleted together in one transaction: an unknown id fails the whole " +
|
|
625
|
+
"call and nothing is deleted.",
|
|
585
626
|
scope: "project",
|
|
586
|
-
args: [{ name: "test-
|
|
627
|
+
args: [{ name: "test-ids", description: "One or more test ids", required: true, variadic: true }],
|
|
587
628
|
flags: [{ name: "force", type: "boolean", description: "Skip the confirmation prompt" }],
|
|
629
|
+
examples: ["beryl tests delete 4f… --force", "beryl tests delete 4f… 9a… 1c…"],
|
|
588
630
|
async run(ctx, input) {
|
|
589
631
|
const { workspaceId, projectId } = await ctx.requireProject(input);
|
|
590
|
-
const
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
632
|
+
const ids = argList(input, "test-ids");
|
|
633
|
+
const single = ids.length === 1 ? ids[0] : undefined;
|
|
634
|
+
const what = single ? `test ${single} and all its history` : `${ids.length} tests and all their history`;
|
|
635
|
+
await ctx.confirm(`Delete ${what}?`, flagBool(input, "force"));
|
|
636
|
+
if (single) {
|
|
637
|
+
await ctx.client.del(testPath(workspaceId, projectId, single));
|
|
638
|
+
}
|
|
639
|
+
else {
|
|
640
|
+
await ctx.client.post(`${projectPath(workspaceId, projectId)}/tests/bulk-delete`, {
|
|
641
|
+
test_case_ids: ids,
|
|
642
|
+
});
|
|
643
|
+
}
|
|
644
|
+
return { human: ids.length === 1 ? "Deleted." : `Deleted ${ids.length} tests.` };
|
|
594
645
|
},
|
|
595
646
|
},
|
|
596
647
|
{
|
|
@@ -598,11 +649,11 @@ export const testCommands = [
|
|
|
598
649
|
summary: "Validate + verify an edited plan against the live site before persisting",
|
|
599
650
|
description: "Unlike `tests set-plan` (which saves the edit and lets it ride into the next run), " +
|
|
600
651
|
"this replays the edited plan against the live site before anything persists. A " +
|
|
601
|
-
"deterministic replay failure (verdict `drop`) REJECTS the edit
|
|
602
|
-
"the prior plan stays live
|
|
652
|
+
"deterministic replay failure (verdict `drop`) REJECTS the edit (persisted:false, " +
|
|
653
|
+
"the prior plan stays live) and returns the failure evidence (over MCP the " +
|
|
603
654
|
"screenshot is image content). Verdict `flag` (the runner errored, no verdict on " +
|
|
604
655
|
"the flow) persists the plan but reports it unverified. Returns 429 with " +
|
|
605
|
-
"Retry-After 30 when the 2 inline-verify slots are saturated
|
|
656
|
+
"Retry-After 30 when the 2 inline-verify slots are saturated (wait and retry).",
|
|
606
657
|
scope: "project",
|
|
607
658
|
args: [{ name: "test-id", description: "Test id", required: true }],
|
|
608
659
|
flags: [
|
|
@@ -671,7 +722,7 @@ export const testCommands = [
|
|
|
671
722
|
{
|
|
672
723
|
name: "tests restore",
|
|
673
724
|
summary: "Restore a test to an earlier version",
|
|
674
|
-
description: "Copies the named older version's plan forward as a NEW head version
|
|
725
|
+
description: "Copies the named older version's plan forward as a NEW head version, unlike " +
|
|
675
726
|
"`tests reset`, which flips authored_by back to `system` and leaves the plan " +
|
|
676
727
|
"untouched.",
|
|
677
728
|
scope: "project",
|
|
@@ -690,7 +741,7 @@ export const testCommands = [
|
|
|
690
741
|
name: "tests reset",
|
|
691
742
|
summary: "Discard user edits and return the test to its latest system-authored version",
|
|
692
743
|
description: "Flips authored_by back to `system` WITHOUT changing the plan (the next " +
|
|
693
|
-
"regeneration overwrites it)
|
|
744
|
+
"regeneration overwrites it), unlike `tests restore`, which copies an older " +
|
|
694
745
|
"version's plan forward as a new version.",
|
|
695
746
|
scope: "project",
|
|
696
747
|
args: [{ name: "test-id", description: "Test id", required: true }],
|
|
@@ -739,9 +790,9 @@ export const testCommands = [
|
|
|
739
790
|
name: "tests script",
|
|
740
791
|
summary: "Print the rendered Playwright spec for a test (or an unbanked plan file)",
|
|
741
792
|
description: "With a test id, fetches the banked test's rendered .spec.ts. With --file, compiles a " +
|
|
742
|
-
"plan JSON that has NOT been banked yet
|
|
793
|
+
"plan JSON that has NOT been banked yet (the same render `tests create` proves locally) " +
|
|
743
794
|
"so you can inspect exactly what would run before creating anything. This returns the " +
|
|
744
|
-
"executable spec
|
|
795
|
+
"executable spec. `tests plan` returns the stored JSON plan it is rendered from, " +
|
|
745
796
|
"`tests get` the metadata row.",
|
|
746
797
|
scope: "project",
|
|
747
798
|
args: [{ name: "test-id", description: "Test id (omit when passing --file)" }],
|
package/package.json
CHANGED