@beryl-so/cli 0.64.0 → 0.73.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  # beryl CLI
4
4
 
5
- Beryl on the command line: create projects, watch the agent explore and author tests,
5
+ Beryl on the command line: create projects, author tests,
6
6
  trigger runs in CI, manage everything the web app can, plus an MCP server (`beryl mcp`)
7
7
  that exposes every command to coding agents.
8
8
 
@@ -78,11 +78,11 @@ Set up Beryl in this repo: sign in and wire up your coding agent
78
78
 
79
79
  ### guide
80
80
 
81
- Print the Beryl test-authoring guide
81
+ Print the Beryl test guide
82
82
 
83
83
  | Command | Summary | MCP tool |
84
84
  | --- | --- | --- |
85
- | `beryl guide` | Print the Beryl test-authoring guide | `guide` |
85
+ | `beryl guide` | Print the Beryl test guide | `guide` |
86
86
 
87
87
  ### login
88
88
 
@@ -166,7 +166,7 @@ Send, list, and revoke workspace invitations, and accept ones sent to you.
166
166
 
167
167
  ### projects
168
168
 
169
- Create and manage projects: a site Beryl explores, authors tests for, and runs.
169
+ Create and manage projects: a site you author tests for and run.
170
170
 
171
171
  `beryl projects` with no subcommand runs `projects list`.
172
172
 
@@ -174,12 +174,14 @@ Create and manage projects: a site Beryl explores, authors tests for, and runs.
174
174
  | --- | --- | --- |
175
175
  | `beryl projects list` | List projects in the workspace | `projects_list` |
176
176
  | `beryl projects get` | Show one project, including its current exploration state | `projects_get` |
177
- | `beryl projects create [url]` | Create a project. With a URL the agent starts exploring; with just --name an empty one | `projects_create` |
177
+ | `beryl projects create [url]` | Create a project for a URL, or an empty one with just --name | `projects_create` |
178
178
  | `beryl projects rename <name>` | Rename a project | `projects_rename` |
179
179
  | `beryl projects quarantine [state]` | Turn auto quarantine on or off and tune when it mutes and releases a test | `projects_quarantine` |
180
+ | `beryl projects surfaces <surfaces...>` | Set which surfaces are enabled for this project | `projects_surfaces` |
181
+ | `beryl projects findings-gate <mode>` | Turn the findings gate on each run on or off | `projects_findings_gate` |
180
182
  | `beryl projects delete` | Delete a project and all its tests and runs | `projects_delete` |
181
183
  | `beryl projects re-explore` | Send the agent back in to run/heal existing tests and discover new flows | none |
182
- | `beryl projects report` | Aggregate quality report across recent runs (pass rates, flaky tests, trend) | `projects_report` |
184
+ | `beryl projects report` | Aggregate quality report across recent runs (pass rates, per-test history, trend) | `projects_report` |
183
185
  | `beryl projects reusable-auth <url>` | Check whether a saved login can be reused for a URL before creating a project | `projects_reusable_auth` |
184
186
 
185
187
  ### envs
@@ -217,9 +219,10 @@ Author, inspect, version, and heal a project's tests: the checks Beryl runs on e
217
219
  | `beryl tests list` | List the project's tests with their latest result | `tests_list` |
218
220
  | `beryl tests get <test-id>` | Show one test | `tests_get` |
219
221
  | `beryl tests plan <test-id>` | Print a test's current step plan (JSON) | `tests_plan` |
220
- | `beryl tests create` | Create a test case from a JSON action plan (for tests authored locally, e.g. by your coding agent) | `tests_create` |
222
+ | `beryl tests create` | Retired: tests are now created by the cloud test-creation job (`jobs create`) | `tests_create` |
221
223
  | `beryl tests set-plan <test-id>` | Replace a test's step plan from a JSON file (creates a new version) | `tests_set_plan` |
222
224
  | `beryl tests rename <test-id> <title>` | Rename a test | `tests_rename` |
225
+ | `beryl tests set-intent <test-id> <intent>` | Edit what a test proves, the target heal judges against | `tests_set_intent` |
223
226
  | `beryl tests set-groups <test-id>` | Replace the groups a test belongs to | `tests_set_groups` |
224
227
  | `beryl tests set-access <test-id> <access>` | Mark a test read or write | `tests_set_access` |
225
228
  | `beryl tests add-groups <test-ids...>` | Add groups to one or more tests, keeping the groups they already have | `tests_add_groups` |
@@ -286,6 +289,18 @@ Site Health, or how your site reads to search engines and visitors: content, spe
286
289
  | `beryl health tracker teams` | List the teams of the connected issue tracker | `health_tracker_teams` |
287
290
  | `beryl health tracker projects` | List a tracker team's projects | `health_tracker_projects` |
288
291
 
292
+ ### jobs
293
+
294
+ Test-creation jobs: the cloud agent reads a document and authors tests from it.
295
+
296
+ | Command | Summary | MCP tool |
297
+ | --- | --- | --- |
298
+ | `beryl jobs create [file]` | Create tests from a document (PRD, release notes) on Beryl's cloud | `jobs_create` |
299
+ | `beryl jobs list` | List the project's test-creation jobs | `jobs_list` |
300
+ | `beryl jobs get <job-id>` | Show a job's status and the tests it created | `jobs_get` |
301
+ | `beryl jobs clone <job-id>` | Run a job again: queue a new job with the same document and environment | `jobs_clone` |
302
+ | `beryl jobs delete <job-id>` | Delete a job; the flows and tests it created stay | `jobs_delete` |
303
+
289
304
  ### explorations
290
305
 
291
306
  Inspect the agent's exploration runs: how it crawled a site and authored its tests.
@@ -308,7 +323,7 @@ Manage the variables, secrets, and files the agent can use while exploring and r
308
323
  | `beryl config vars set <key> <value>` | Create or update a config variable | `config_vars_set` |
309
324
  | `beryl config vars get <key>` | Show one config variable | `config_vars_get` |
310
325
  | `beryl config vars delete <key>` | Delete a config variable | `config_vars_delete` |
311
- | `beryl config secrets list` | List the project's secrets (values are never returned) | `config_secrets_list` |
326
+ | `beryl config secrets list` | List the project's secrets (values are never returned); a test account's password is managed on the account, not here | `config_secrets_list` |
312
327
  | `beryl config secrets set <key> <value>` | Create a secret (write-only; re-setting a key replaces it) | `config_secrets_set` |
313
328
  | `beryl config secrets get <key>` | Show one secret's metadata, or reveal its value with --reveal | `config_secrets_get` |
314
329
  | `beryl config secrets delete <key>` | Delete a secret | `config_secrets_delete` |
@@ -354,7 +369,9 @@ Durable identities on the site under test: what an authenticated test signs in a
354
369
  | Command | Summary | MCP tool |
355
370
  | --- | --- | --- |
356
371
  | `beryl accounts list` | List the test accounts an environment's tests sign in as | `accounts_list` |
372
+ | `beryl accounts get <account-id>` | Show one test account: origin, lifetime, sign-in state | `accounts_get` |
357
373
  | `beryl accounts create` | Add a test account: the customer's own, or one Beryl signs up | `accounts_create` |
374
+ | `beryl accounts capture` | Store a signed-in session as a captured test account | `accounts_capture` |
358
375
  | `beryl accounts signup <account-id>` | Store a test account's signup plan and sign it up on its environment | `accounts_signup` |
359
376
  | `beryl accounts set-login <account-id>` | Store the sign-in plan a run replays once, plus the probe that proves it | `accounts_set_login` |
360
377
  | `beryl accounts get-login <account-id>` | Read the stored sign-in plan, its probe, and the hash a safe write must cite | `accounts_get_login` |
@@ -278,13 +278,15 @@ export function mcpInstructions() {
278
278
  "version, and whether this build is behind npm's latest). " +
279
279
  "Beryl authors, runs, and heals end-to-end tests for any web app: tests are JSON " +
280
280
  "action plans replayed in real cloud browsers, signing in as a durable test " +
281
- "account whose mail arrives at the project's own mailbox — so signup/OTP/" +
281
+ "account whose mail arrives at the project's own mailbox, so signup/OTP/" +
282
282
  "magic-link flows are self-contained, with no human login needed. " +
283
- "The full authoring guide (plan shape, outcome assertions, email/OTP wiring, " +
284
- "run-fix loop) ships as both the beryl-test skill and the `guide` tool — same " +
283
+ "New tests are authored only by Beryl's cloud agent: give it a document describing " +
284
+ "the flows with `jobs_create`; there is no local authoring. " +
285
+ "The guide (plan shape, outcome assertions, accounts, email/OTP wiring, the local " +
286
+ "run-fix loop) ships as both the beryl-test skill and the `guide` tool, same " +
285
287
  `content. If a beryl-test skill stating v${version} is already loaded, do not ` +
286
288
  "call `guide`; if no beryl-test skill is available or it states another version, " +
287
- "call `guide` before authoring your first test plan. " +
289
+ "call `guide` before editing a test plan or a sign-in plan. " +
288
290
  "An account can hold several workspaces and projects: call `projects_list` first " +
289
291
  "and pass `project` (an id is enough) on every project-scoped tool, or the call fails.");
290
292
  }
@@ -1,4 +1,4 @@
1
- // The `beryl-test` authoring skill, installed by `beryl init` into
1
+ // The `beryl-test` skill, installed by `beryl init` into
2
2
  // `.agents/skills/beryl-test/SKILL.md` (vendor-neutral, editor-agnostic) — and, when
3
3
  // claude-code is a selected editor, ALSO into the `.claude/skills/beryl-test/SKILL.md`
4
4
  // that Claude Code actually indexes (see `writeSkills` in `commands/init.ts`). Kept as an
@@ -17,23 +17,24 @@ export const BERYL_TEST_SKILL_DIR = "beryl-test";
17
17
  // nor `.claude/skills/`, so `init` writes them their own pointer instead. Thin on purpose:
18
18
  // it sends the agent to the `guide` MCP tool rather than inlining the 500-line skill, and
19
19
  // `guide` ships with the CLI so the real guidance never drifts from the installed version.
20
- const BERYL_GUIDE_POINTER = `When writing, editing, or debugging Beryl tests in this repo, call the \`guide\`
21
- MCP tool first. It returns the full authoring guide: the ActionPlan shape, outcome
22
- assertions, test accounts and the project mailbox, and the local run-fix loop.
20
+ const BERYL_GUIDE_POINTER = `When creating, running, editing, or debugging Beryl tests in this repo, call the
21
+ \`guide\` MCP tool first. It returns the full guide: where new tests come from, the
22
+ ActionPlan shape, outcome assertions, test accounts and the project mailbox, and the
23
+ local run-fix loop.
23
24
 
24
- Then use the Beryl MCP tools: \`projects_list\` to find the project, and \`tests_create\`
25
- to author a plan and replay it locally before it is banked.
25
+ Then use the Beryl MCP tools: \`projects_list\` to find the project, \`jobs_create\` to
26
+ have Beryl's cloud agent author tests from a document, and \`runs_local\` to run them here.
26
27
 
27
28
  The \`guide\` tool ships with the CLI and always matches the installed version, so treat
28
29
  it as the source of truth over anything cached here.`;
29
30
  // Cursor reads its own `.cursor/rules/*.mdc`, auto-loaded when alwaysApply is set.
30
31
  export const BERYL_CURSOR_RULE_FILENAME = "beryl.mdc";
31
32
  export const BERYL_CURSOR_RULE = `---
32
- description: Authoring and editing Beryl end-to-end tests
33
+ description: Creating, running and editing Beryl end-to-end tests
33
34
  alwaysApply: true
34
35
  ---
35
36
 
36
- # Beryl test authoring
37
+ # Beryl tests
37
38
 
38
39
  ${BERYL_GUIDE_POINTER}
39
40
  `;
@@ -43,7 +44,7 @@ ${BERYL_GUIDE_POINTER}
43
44
  export const BERYL_COPILOT_MARK_BEGIN = "<!-- BEGIN beryl (managed by beryl init) -->";
44
45
  export const BERYL_COPILOT_MARK_END = "<!-- END beryl -->";
45
46
  export const BERYL_COPILOT_SECTION = `${BERYL_COPILOT_MARK_BEGIN}
46
- ## Beryl test authoring
47
+ ## Beryl tests
47
48
 
48
49
  ${BERYL_GUIDE_POINTER}
49
50
  ${BERYL_COPILOT_MARK_END}`;
@@ -89,78 +90,78 @@ export const BERYL_TEST_SKILL_OTP_EXAMPLE_PLAN = {
89
90
  };
90
91
  export const BERYL_TEST_SKILL = `---
91
92
  name: beryl-test
92
- description: Author durable, healable end-to-end tests for a web app with Beryl. Use when writing, running, or fixing a Beryl test locally with your own coding agent, drafting the plan over the Playwright MCP, writing the natural-language intent, and running it with \`runs_local\`.
93
+ description: Create, run, and fix durable, healable end-to-end tests for a web app with Beryl. Use when asking Beryl's cloud agent to create tests from a document (\`jobs_create\`), running tests locally with \`runs_local\`, setting up test accounts, or fixing a banked test's plan with your own coding agent.
93
94
  ---
94
95
 
95
- # Authoring Beryl tests
96
+ # Beryl tests
96
97
 
97
- beryl-test v${cliVersion()} — installed by \`beryl init\`; the \`guide\` tool always returns the current version of this guide.
98
+ beryl-test v${cliVersion()}, installed by \`beryl init\`; the \`guide\` tool always returns the current version of this guide.
98
99
 
99
100
  A Beryl test drives a real browser through a flow and asserts the flow reached its
100
- meaningful outcome. \`beryl init\` wired two MCP servers: **beryl** (the tools you author
101
- with: \`accounts_list\`, \`tests_create\`, \`runs_local\`, and the rest) and **playwright**
102
- (drive a real browser). You do everything below by calling those tools. Beryl **heals**
103
- tests as the app's markup drifts, but only when you supply the three things that make a
104
- test durable: a real **outcome assertion**, a strong **natural-language intent**, and the
105
- **local run-fix loop**.
101
+ meaningful outcome. \`beryl init\` wired two MCP servers: **beryl** (\`jobs_create\`,
102
+ \`accounts_list\`, \`runs_local\`, and the rest) and **playwright** (drive a real
103
+ browser). You do everything below by calling those tools. Beryl **heals** tests as the
104
+ app's markup drifts, but only when a test has the three things that make it durable: a
105
+ real **outcome assertion**, a strong **natural-language intent**, and the **local
106
+ run-fix loop**.
106
107
 
107
108
  ## 0. Start here
108
109
 
109
- **Everything happens on your machine first**: you drive the flow here, and
110
- \`tests_create\` proves the plan here too, so a working local browser IS the loop.
111
- Keep \`tests_create\`'s verify on (banking unproven is a trap); replay mechanics and the
112
- cloud-session exception are in the \`tests_create\` tool's own description.
110
+ **New tests are authored only by Beryl's cloud agent.** There is no local authoring: you
111
+ do not draft a new plan here. A human writes a document (a PRD, release notes, a list of
112
+ flows) and \`jobs_create\` hands it to the agent, which explores the environment and
113
+ authors one test per scenario and surface. \`jobs_get\` shows the job's status, the tests
114
+ it produced and any scenario it skipped. The document is sent to the LLM that drives the
115
+ agent, so it must not hold anything the user would not share with it.
116
+
117
+ What you do locally: set up the accounts tests sign in as (§1), run tests on this machine
118
+ (§4), and fix a banked test's plan when it needs it (§2).
113
119
 
114
120
  Set up (once): if you cannot call the Beryl tools yet, run \`beryl init\` in a terminal to
115
- install the browser and wire the tools. Then, before the first plan:
121
+ install the browser and wire the tools. Then, before the first job:
116
122
 
117
123
  - \`envs_list\`: the root URL gotos resolve against (§2). If it is empty, set it with
118
- \`envs_update\`.
124
+ \`envs_update\` to a URL Beryl's cloud can reach, never \`localhost\`, \`127.0.0.1\`
125
+ or a private IP (refused). For a local dev server, pass \`url-override\` to
126
+ \`runs_local\` instead.
119
127
  - \`accounts_list\`: who do authenticated tests sign in as (§1)? Store its sign-in with
120
128
  \`accounts_set_login\`, prove it with \`accounts_check\`.
121
129
  - \`mailbox_get\`: the address the account receives mail at (§5).
122
130
 
123
- \`accounts_list\` comes FIRST: standing account vs new identity changes the plan you
124
- write (discovering it afterwards means rewriting), and an authenticated test cannot be
125
- created until its account's login plan is stored.
126
-
127
- Then, per test:
128
-
129
- 1. **Drive the flow** in a real browser over the Playwright MCP, never from imagination (§2).
130
- 2. **Write the ActionPlan**, with one real outcome assertion (§2).
131
- 3. **Lint it** with \`tests_lint\` (the plan plus its \`read\`/\`write\` access): schema and
132
- access check, offline.
133
- 4. **Bank it** with \`tests_create\` (the title, the plan, its \`access\`, and the
134
- \`description\` intent): it replays the plan locally, banks only on green, records no
135
- run; a red replay hands back the failure to fix and re-run (§3 for the intent, §4 for
136
- the loop).
137
- 5. When the batch is banked, run the new ids together ONCE with \`runs_local\` (or
131
+ \`accounts_list\` comes FIRST: an authenticated test cannot be saved until its account's
132
+ login plan is stored.
133
+
134
+ Then:
135
+
136
+ 1. **Create tests** with \`jobs_create\` (the document, and the environment to explore).
137
+ 2. When the job is done, run the new ids together ONCE with \`runs_local\` (or
138
138
  \`runs_trigger\` for the cloud) so they land as one run with real first results.
139
+ 3. **A test is red or wrong?** Fix its plan with the local run-fix loop (§4).
139
140
 
140
141
  ## 1. First: who do your tests sign in as?
141
142
 
142
143
  An authenticated test **signs in as an account that already exists** — it does not sign
143
- one up. Start every authoring session with \`accounts_list\`:
144
+ one up. Start every session with \`accounts_list\`:
144
145
 
145
146
  \`\`\`
146
- label email type login status
147
- default * qa@acme.test user_provided password ready
148
- admin qa+admin@x7k2p9.email.beryl.so beryl otp ready
147
+ label email type login status
148
+ default * qa@acme.test provided password ready
149
+ admin qa+admin@x7k2p9.email.beryl.so provisioned (permanent) otp ready
149
150
  \`\`\`
150
151
 
151
152
  Author the sign-in as ordinary opening steps citing the reserved handles — fill
152
153
  \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell.
153
- Set \`requires_auth: true\` AND \`auth_mode\` — both, always; \`auth_mode\` has NO
154
- default and create rejects a \`requires_auth\` plan without it. \`"inline"\` = the plan
154
+ Set \`requires_auth: true\` AND \`auth_mode\`, both, always; \`auth_mode\` has NO
155
+ default and a save rejects a \`requires_auth\` plan without it. \`"inline"\` = the plan
155
156
  signs itself in like this; \`"session"\` = no sign-in steps, rides the account's
156
157
  once-per-run session (§ Session mode). \`auth_label\` picks a non-default identity
157
158
  (the \`*\` row is the default). Handles resolve at run time to the named account, so
158
159
  one plan stays correct across environments.
159
160
 
160
- **The account needs a stored login plan before any authenticated test can be created.**
161
+ **The account needs a stored login plan before any authenticated test can be saved.**
161
162
  A run replays that plan to produce the session its tests ride, and replays it again
162
- when the session expires, so \`tests_create\` rejects a \`requires_auth\` plan whose
163
- account has none, naming the fix. Store it once per account with
163
+ when the session expires, so Beryl rejects a \`requires_auth\` plan whose account has
164
+ none, naming the fix. Store it once per account with
164
165
  \`accounts_set_login\` (§ Session mode).
165
166
 
166
167
  **NEVER paste a real email or password into a plan.** Handles resolve at run time —
@@ -237,9 +238,9 @@ An empty \`accounts_list\` means nothing is set up. In order of preference:
237
238
  let the plan \`await_email\` through; later sign-ins read the same mailbox.
238
239
  4. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
239
240
 
240
- A project with no accounts falls back to the \`LOGIN_EMAIL\` variable +
241
- \`LOGIN_PASSWORD\` secret with the same handle resolution, so existing tests are
242
- unaffected — but new work should create an account.
241
+ Every saved login is a test account: \`{{login_email}}\` and \`{{login_password}}\` resolve
242
+ only from the environment's account, and a login password is never set with
243
+ \`config secrets set\`. The webapp's Test accounts tab edits the same accounts.
243
244
 
244
245
  ### Session mode: sign in once per run, not once per test
245
246
 
@@ -262,26 +263,21 @@ descriptions.
262
263
  Then write the tests with \`requires_auth: true\`, \`auth_mode: "session"\`, and NO
263
264
  sign-in steps.
264
265
 
265
- **Creates share the sign-in.** \`tests_create\` replays session-mode plans against the
266
- account's established session (reused while it stays live), so a batch of creates costs
267
- at most one sign-in — but still create one at a time: racing creates race the same
268
- mailbox for mail.
269
-
270
266
  An expired session is never your problem: every run proves the stored session and, on a
271
267
  miss, signs in again from the login plan. A "reconnect" message only appears when there
272
268
  is no stored login plan to refresh from.
273
269
 
274
270
  When \`accounts_check\` fails, its description explains which half broke and each
275
- \`SESSION_*\` reason. The one that changes your authoring: an account marked
271
+ \`SESSION_*\` reason. The one that changes your plans: an account marked
276
272
  unsupported (its session cannot survive a fresh browser) is not your bug, but it IS
277
- your move — re-author its session-mode tests with \`auth_mode: "inline"\` and their own
273
+ your move: switch its session-mode tests to \`auth_mode: "inline"\` and their own
278
274
  sign-in steps.
279
275
 
280
276
  ### Two identities in one test
281
277
 
282
- Invite-a-teammate-and-accept-as-them needs two identities mid-flow — not bankable as
283
- one test. Bank the half the app shows to account A ("the invitation is listed as
284
- pending", "the share link is issued") — a real, strong outcome. While AUTHORING you can
278
+ Invite-a-teammate-and-accept-as-them needs two identities mid-flow, so it is not bankable
279
+ as one test. Bank the half the app shows to account A ("the invitation is listed as
280
+ pending", "the share link is issued"): a real, strong outcome. While fixing a plan you can
285
281
  drive the full handshake live: a second account (\`accounts_create\`, labelled), or a
286
282
  second mailbox (\`mailbox_create\`, read with \`mailbox_read\`).
287
283
 
@@ -289,11 +285,14 @@ SSO-only sites are not a dead end: with a dedicated Google test account they fol
289
285
  item 2 like any password account. The captured session is the FALLBACK — no test account
290
286
  can be provided, or Google challenges the scripted sign-in: a human signs in once
291
287
  through a live browser view in the Beryl webapp, Beryl stores the session encrypted
292
- server-side and injects it into cloud runs, and \`tests_create\` verifies such plans
293
- server-side, \`runs_trigger\` runs them, \`runs_local\` skips them (the session never
294
- leaves Beryl's cloud).
288
+ server-side and injects it into cloud runs: \`runs_trigger\` runs such tests,
289
+ \`runs_local\` skips them (the session never leaves Beryl's cloud).
290
+
291
+ ## 2. Fixing a test's plan over the Playwright MCP
295
292
 
296
- ## 2. Author locally over the Playwright MCP
293
+ New tests come from \`jobs_create\` (§0). You write plan JSON only to fix a banked test
294
+ (\`tests_plan\` returns its current plan) or to store an account's sign-in or signup
295
+ plan (§1). Either way, the rules below hold.
297
296
 
298
297
  1. **Drive the flow in a real browser first.** Walk it by hand over the Playwright MCP
299
298
  — log in, fill, submit — acting on elements by their accessibility ref from the
@@ -355,17 +354,13 @@ ${JSON.stringify(BERYL_TEST_SKILL_EXAMPLE_PLAN, null, 2)
355
354
  .map((line) => ` ${line}`)
356
355
  .join("\n")}
357
356
  \`\`\`
358
- 3. **Validate offline, then create.** Lint with \`tests_lint\` (schema check, no network),
359
- then bank with \`tests_create\` (the title, the plan, and the \`description\` intent, see
360
- §3). \`tests_create\` banks the plan only on a green local replay; a red one hands back
361
- the failure to fix and re-run. It records no run. Replay mechanics (where it runs over
362
- MCP, its verify option, the cloud-session exception) are in the \`tests_create\` tool's
363
- own description. Inspect the exact spec with \`tests_script\`.
357
+ 3. **Validate offline, then save.** Lint with \`tests_lint\` (schema check, no network),
358
+ inspect the exact spec with \`tests_script\`, then save with \`tests_recompile\`
359
+ (replays the edit against the live site first and rejects it on a red replay) or
360
+ \`tests_set_plan\` (saves with no replay; the next run proves it). Pass a new
361
+ \`description\` only when the edit changes what the test proves (§3).
364
362
  Full ActionPlan JSON Schema:
365
363
  https://api.beryl.so/api/v1/schemas/action-plan.schema.json.
366
- Groups are optional and never block: after the test is banked, if \`groups_list\`
367
- shows any, offer them once (\`tests_set_groups\`); no answer = leave it ungrouped and
368
- move on. Never create a group yourself, that is the user's call.
369
364
 
370
365
  ### Testing a mobile or tablet viewport
371
366
  A test runs at a desktop viewport unless you say otherwise. To cover a responsive layout,
@@ -374,7 +369,7 @@ full Chromium device emulation, not a resize: the viewport, a mobile Chrome user
374
369
  touch and the device pixel ratio all change together, so a hamburger or a nav that only
375
370
  exists on small screens actually renders and is testable. \`mobile\` emulates a Pixel-class
376
371
  phone, \`tablet\` a Galaxy-class tablet; both keep a Chrome user-agent so the emulated UA
377
- matches the engine. Author the flow at that size over the Playwright MCP (resize the MCP
372
+ matches the engine. Drive the flow at that size over the Playwright MCP (resize the MCP
378
373
  browser to match) so the refs you act on are the ones the small-screen layout renders. The
379
374
  account, session and teardown rules are unchanged; \`device\` only sets the frame it runs in.
380
375
 
@@ -406,9 +401,9 @@ observable proof the flow worked.
406
401
  - **Don't work around a real app failure to make a test go green.** A genuinely broken
407
402
  flow is a finding to report, not something to paper over with a weaker assertion.
408
403
 
409
- ### Traps when authoring against a real app
404
+ ### Traps when writing a plan against a real app
410
405
 
411
- Every one of these has produced a wrong plan or a red \`tests_create\`. Check them.
406
+ Every one of these has produced a wrong plan or a red run. Check them.
412
407
 
413
408
  1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
414
409
  profile. Author while signed in and you never see the gate — you will mark gated
@@ -440,8 +435,9 @@ Every one of these has produced a wrong plan or a red \`tests_create\`. Check th
440
435
 
441
436
  ## 3. Writing the natural-language intent
442
437
 
443
- Give the intent as the \`description\` of \`tests_create\` (or \`tests_set_plan\`
444
- when you re-author). 1-3 sentences. This is the immutable anchor from §6.
438
+ The cloud agent drafts each new test's intent. When a fix changes what a test proves,
439
+ pass the new intent as the \`description\` of \`tests_set_plan\`. 1-3 sentences. This is
440
+ the immutable anchor from §6.
445
441
 
446
442
  - **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
447
443
  clicks submit" — the trajectory will change. Instead: *what does a green run prove is
@@ -464,10 +460,10 @@ with the Playwright runner bundled in the CLI: no cloud, no waiting for a schedu
464
460
  Run it un-synced with a report directory against one test id while you iterate, or with
465
461
  no id to run the whole suite and record the results in Beryl.
466
462
 
467
- The loop: draft → \`tests_lint\` → \`tests_create\` (proves, records nothing) → \`runs_local\`
468
- un-synced while iterating → read the run's \`report.json\` to see which step or assertion
469
- failed and why → fix the plan → \`tests_set_plan\` → run again. When the batch is done, one
470
- synced \`runs_local\` (or \`runs_trigger\`) records it as a single run. Its options,
463
+ The loop: \`runs_local\` un-synced against the test id → read the run's \`report.json\` to
464
+ see which step or assertion failed and why → fix the plan (§2) → \`tests_lint\` →
465
+ \`tests_set_plan\` → run again. When the batch is done, one synced \`runs_local\` (or
466
+ \`runs_trigger\`) records it as a single run. Its options,
471
467
  session-mode behavior, and \`await_email\` / authenticated-test handling are in the
472
468
  \`runs_local\` tool's own description.
473
469
 
@@ -525,12 +521,9 @@ For a magic-link flow, replace the code steps with
525
521
 
526
522
  Two caveats:
527
523
 
528
- - \`tests_create\` verifies an \`await_email\` plan like any other: the local
529
- replay receives at the project mailbox and answers each step over the API, so the
530
- app's mail really is received and extracted before the test is accepted. The replay
531
- signs up / sends mail FOR REAL, so turn \`tests_create\`'s verify off only if that side
532
- effect is unwanted. \`runs_local\` serves \`await_email\` the same way, so the local loop
533
- covers OTP/signup flows end to end.
524
+ - \`runs_local\` serves \`await_email\` like a cloud run: it receives at the project
525
+ mailbox and answers each step over the API, so the local loop covers OTP/signup flows
526
+ end to end. It signs up / sends mail FOR REAL, like every run.
534
527
  - §2's outcome discipline still applies: the green signal is the post-verification
535
528
  state (the welcome screen, the dashboard), not "an email arrived".
536
529
 
@@ -575,8 +568,8 @@ Rules:
575
568
 
576
569
  - Always tear down what the test connected in \`after\`: disconnect the integration in
577
570
  the app.
578
- - These steps run only in Beryl's cloud: \`tests_create\` verifies server-side and
579
- \`runs_local\` skips them, so run them with \`runs_trigger\`.
571
+ - These steps run only in Beryl's cloud: \`runs_local\` skips them, so run them with
572
+ \`runs_trigger\`.
580
573
  - A setup error saying Beryl's access "is being restored" is on Beryl's side: do not
581
574
  edit the test.
582
575
  - Never add steps that browse Slack itself; the actions above do all of it.