@beryl-so/cli 0.17.0 → 0.21.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -288,17 +288,37 @@ Send run outcomes to a Slack channel via a per-project incoming webhook.
288
288
  | `beryl slack clear` | Remove the project's Slack webhook (stops all alerts) | — |
289
289
  | `beryl slack test` | Post a sample alert to the configured webhook | — |
290
290
 
291
- ### inbox
291
+ ### mailbox
292
292
 
293
- Email inboxes for testing flows that send mail signups, OTPs, receipts.
293
+ The project's standing email addresses where its tests receive sign-in mail.
294
+
295
+ `beryl mailbox` with no subcommand runs `mailbox get`.
296
+
297
+ | Command | Summary | MCP tool |
298
+ | --- | --- | --- |
299
+ | `beryl mailbox get` | The project's mailbox address | `mailbox_get` |
300
+ | `beryl mailbox list` | List the project's mailboxes | `mailbox_list` |
301
+ | `beryl mailbox create` | Add a second mailbox to the project | `mailbox_create` |
302
+ | `beryl mailbox delete <mailbox-id>` | Delete a mailbox and every email it has received | `mailbox_delete` |
303
+ | `beryl mailbox read <mailbox-id>` | Read the latest email in a mailbox (waits for one to arrive) | `mailbox_read` |
304
+ | `beryl mailbox emails <mailbox-id>` | List the emails a mailbox has received | `mailbox_emails` |
305
+
306
+ ### accounts
307
+
308
+ Durable identities on the site under test — what an authenticated test signs in as.
309
+
310
+ `beryl accounts` with no subcommand runs `accounts list`.
294
311
 
295
312
  | Command | Summary | MCP tool |
296
313
  | --- | --- | --- |
297
- | `beryl inbox create` | Mint an email inbox that Beryl receives mail for | `inbox_create` |
298
- | `beryl inbox list` | List the workspace's inboxes, newest first | `inbox_list` |
299
- | `beryl inbox delete <inbox-id>` | Delete an inbox and every email it has received | `inbox_delete` |
300
- | `beryl inbox read <inbox-id>` | Read the latest email from an inbox (waits for one to arrive) | `inbox_read` |
301
- | `beryl inbox emails <inbox-id>` | List the emails an inbox has received | `inbox_emails` |
314
+ | `beryl accounts list` | List the test accounts an environment's tests sign in as | `accounts_list` |
315
+ | `beryl accounts create` | Add a test account — the customer's own, or one Beryl signs up | `accounts_create` |
316
+ | `beryl accounts provision <account-id>` | Prove a test account can get in, by replaying a plan that ends logged in | `accounts_provision` |
317
+ | `beryl accounts set-login <account-id>` | Store the sign-in plan a run replays once, plus the probe that proves it | `accounts_set_login` |
318
+ | `beryl accounts get-login <account-id>` | Read the stored sign-in plan, its probe, and the hash a safe write must cite | `accounts_get_login` |
319
+ | `beryl accounts check <account-id>` | Sign in now and prove the session survives into a fresh browser | `accounts_check` |
320
+ | `beryl accounts update <account-id>` | Change a test account's password, login method, or default flag | `accounts_update` |
321
+ | `beryl accounts delete <account-id>` | Delete a test account | `accounts_delete` |
302
322
 
303
323
  ### account
304
324
 
@@ -329,6 +349,14 @@ Review a workspace's plan usage, subscription, and invoices.
329
349
  | `beryl billing invoices` | List recent invoices | — |
330
350
  | `beryl billing portal` | Get a Stripe billing-portal link for the workspace | — |
331
351
 
352
+ ### version
353
+
354
+ Show the running CLI version, API URL, and Node version
355
+
356
+ | Command | Summary | MCP tool |
357
+ | --- | --- | --- |
358
+ | `beryl version` | Show the running CLI version, API URL, and Node version | `version` |
359
+
332
360
  ### mcp
333
361
 
334
362
  Run the Beryl MCP server (stdio) — every CLI command as an agent tool
@@ -230,6 +230,8 @@ export async function runCli(argv) {
230
230
  // flag we don't recognise as global — that one belongs to the command.
231
231
  const words = [];
232
232
  const wordIndices = [];
233
+ let sawGlobalFlag = false;
234
+ let sawVersionFlag = false;
233
235
  for (let i = 0; i < argv.length; i++) {
234
236
  const tok = argv[i];
235
237
  if (tok === "--")
@@ -238,6 +240,9 @@ export async function runCli(argv) {
238
240
  const span = globalFlagSpan(argv, i);
239
241
  if (span === 0)
240
242
  break;
243
+ sawGlobalFlag = true;
244
+ if (tok === "--version" || tok === "-V")
245
+ sawVersionFlag = true;
241
246
  i += span - 1;
242
247
  continue;
243
248
  }
@@ -246,7 +251,16 @@ export async function runCli(argv) {
246
251
  }
247
252
  let rest = argv;
248
253
  let restWordIndices = wordIndices;
249
- if (argv.includes("--version") || argv.includes("-V") || words[0] === "version") {
254
+ // `-V`, `--version`, and a plain `beryl version` keep printing just the number: that
255
+ // one-line output is a contract CI users parse, so it stays intercepted here rather than
256
+ // going through the registry (which would render the full key/value block instead).
257
+ // Any global flag alongside the `version` word (`--json`, `--api-url`, `--help`, …) means
258
+ // the caller wants more than the number, so it falls through to the `version` spec — the
259
+ // same build picture the `version` MCP tool returns (api_url, Node, staleness). Only flags
260
+ // the scanner above actually recognised count, so it honours `--` the same way (tokens
261
+ // after it are inert positionals, not a request for the spec).
262
+ const versionWord = words[0] === "version" && words.length === 1;
263
+ if (sawVersionFlag || (versionWord && !sawGlobalFlag)) {
250
264
  process.stdout.write(cliVersion() + "\n");
251
265
  return EXIT_OK;
252
266
  }
@@ -1,8 +1,11 @@
1
1
  import { Server } from "@modelcontextprotocol/sdk/server/index.js";
2
2
  import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
3
3
  import { CallToolRequestSchema, ListToolsRequestSchema, } from "@modelcontextprotocol/sdk/types.js";
4
+ import fs from "node:fs";
5
+ import { loadConfig } from "../config.js";
4
6
  import { createContext } from "../context.js";
5
7
  import { CliError } from "../errors.js";
8
+ import { ApiClient } from "../http.js";
6
9
  import { commands } from "../registry/index.js";
7
10
  import { cliVersion, warnIfStale } from "../version-check.js";
8
11
  export function toolName(spec) {
@@ -71,15 +74,50 @@ function toInput(spec, params) {
71
74
  }
72
75
  return { args, flags };
73
76
  }
77
+ let authCache;
78
+ /** The credentials as they are on disk RIGHT NOW, not as they were at spawn.
79
+ *
80
+ * This server is long-lived and never re-execs, so reading the token once would leave a
81
+ * `beryl login` performed afterwards invisible to it forever — and the 401 it then
82
+ * returns tells you to run the login you just ran, which is unescapable without knowing
83
+ * to reconnect. Keyed on mtime so the common path is a stat, not a rebuilt client. */
84
+ export function currentAuth(fallback) {
85
+ let mtimeMs = 0;
86
+ try {
87
+ mtimeMs = fs.statSync(fallback.config.globalConfigPath).mtimeMs;
88
+ }
89
+ catch {
90
+ // No config file (env-var auth, or never logged in) — nothing to watch.
91
+ }
92
+ if (authCache?.mtimeMs === mtimeMs)
93
+ return authCache.auth;
94
+ const config = loadConfig();
95
+ const auth = {
96
+ client: new ApiClient(config.apiUrl, config.token),
97
+ config,
98
+ };
99
+ authCache = { mtimeMs, auth };
100
+ return auth;
101
+ }
102
+ export function __resetAuthCacheForTests() {
103
+ authCache = undefined;
104
+ }
74
105
  export async function serveMcp(baseCtx) {
75
106
  // Fire-and-forget staleness warning: a stale MCP server silently exposes fewer
76
107
  // tools, and stderr is the one channel a stdio MCP server can safely log to.
77
108
  void warnIfStale(cliVersion(), (msg) => console.error(msg));
78
109
  const server = new Server({ name: "beryl", version: cliVersion() }, {
79
110
  capabilities: { tools: {} },
80
- instructions: "Beryl authors, runs, and heals end-to-end tests for any web app: tests are JSON " +
81
- "action plans replayed in real cloud browsers, with per-run email inboxes that make " +
82
- "signup/OTP/magic-link flows fully self-contained (no human login needed). Before " +
111
+ // The running version is stated up front because this server is long-lived and never
112
+ // hot-reloads: a session can sit on a days-old build while `@latest` has moved, and
113
+ // "that tool doesn't exist for me" is indistinguishable from a bug without it. Saying
114
+ // it here means the model knows without spending a `version` tool call.
115
+ instructions: `Beryl CLI v${cliVersion()} (call the \`version\` tool for the API URL, Node ` +
116
+ "version, and whether this build is behind npm's latest). " +
117
+ "Beryl authors, runs, and heals end-to-end tests for any web app: tests are JSON " +
118
+ "action plans replayed in real cloud browsers, signing in as a durable test " +
119
+ "account whose mail arrives at the project's own mailbox — so signup/OTP/" +
120
+ "magic-link flows are self-contained, with no human login needed. Before " +
83
121
  "authoring your first test plan, call the `guide` tool — it returns the full " +
84
122
  "authoring guide (plan shape, outcome assertions, email/OTP wiring, run-fix loop).",
85
123
  });
@@ -105,9 +143,10 @@ export async function serveMcp(baseCtx) {
105
143
  if (lines.length > 400)
106
144
  lines.splice(0, lines.length - 400);
107
145
  };
146
+ const auth = currentAuth(baseCtx);
108
147
  const ctx = createContext({
109
- client: baseCtx.client,
110
- config: baseCtx.config,
148
+ client: auth.client,
149
+ config: auth.config,
111
150
  json: true,
112
151
  interactive: false,
113
152
  mcp: true,
@@ -57,7 +57,182 @@ because Beryl can **heal** them — but only when you give it what it needs to.
57
57
  Read this before authoring. The three ideas that make a test durable: a real **outcome
58
58
  assertion**, a strong **natural-language intent**, and the **local run-fix loop**.
59
59
 
60
- ## 1. Author locally over the Playwright MCP
60
+ ## 0. Start here
61
+
62
+ **Everything happens on your machine first.** You drive the flow in a real browser here,
63
+ and \`tests create\` replays the plan here — over MCP that means the machine running the
64
+ MCP server, never Beryl's. A red replay banks nothing. Two exceptions to "proven
65
+ locally": a plan that depends on a session only Beryl's cloud holds is verified
66
+ server-side instead (create tells you when), and \`--no-verify\` banks unproven — avoid
67
+ it. So a broken local browser is not a detail you can skip past; it is the whole loop.
68
+
69
+ Before the first plan:
70
+
71
+ \`\`\`
72
+ npm i -D @playwright/test && npx playwright install chromium # once, per project
73
+ beryl accounts list # who do authenticated tests sign in as? (§1)
74
+ beryl accounts set-login <id> --file … --probe … # store its sign-in — required (§1)
75
+ beryl accounts check <id> # does that stored sign-in still work? (§1)
76
+ beryl mailbox get # the address they receive mail at (§5)
77
+ \`\`\`
78
+
79
+ \`accounts list\` comes FIRST, before you author anything. It decides whether a flow signs
80
+ in as a standing account or acquires a new identity, and that choice changes the plan you
81
+ write — discovering it afterwards means rewriting. If a flow needs signing in, the
82
+ account's login plan has to be stored before you can create the test at all.
83
+
84
+ Then, per test:
85
+
86
+ 1. **Drive the flow** in a real browser over the Playwright MCP — never author from
87
+ imagination (§2).
88
+ 2. **Write the ActionPlan**, with one real outcome assertion (§2).
89
+ 3. \`beryl tests lint --file plan.json\` — schema check, offline, no network.
90
+ 4. \`beryl tests create --title … --file … --description "<the intent>"\` — replays the plan
91
+ locally and banks it only if it goes green. A red replay banks nothing and hands back
92
+ the failure; fix the file and re-run (§3 for the intent, §4 for the loop).
93
+ 5. \`beryl runs local\` — re-run banked tests on your machine while iterating.
94
+ 6. \`beryl runs trigger\` — hand it to Beryl's cloud, on demand or on a schedule.
95
+
96
+ ## 1. First: who do your tests sign in as?
97
+
98
+ An authenticated test **signs in as an account that already exists** — it does not sign
99
+ one up. Start every authoring session with \`beryl accounts list\`:
100
+
101
+ \`\`\`
102
+ label email type login status
103
+ default * qa@acme.test user_provided password ready
104
+ admin qa+admin@x7k2p9.email.beryl.so beryl otp ready
105
+ \`\`\`
106
+
107
+ Author the sign-in as ordinary opening steps citing the reserved handles — fill
108
+ \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell. Set
109
+ \`requires_auth: true\` AND \`auth_mode\` — both, always. \`auth_mode\` has NO default and
110
+ a \`requires_auth\` plan without it is rejected at create: \`"inline"\` when the plan
111
+ signs itself in like this, \`"session"\` when it carries no sign-in steps and rides its
112
+ account's once-per-run session (§ Session mode). Add \`auth_label\` only to pick a
113
+ non-default identity (the row marked \`*\` is what a plan gets otherwise). The handles
114
+ resolve at run time to whichever account the plan named, so one plan stays correct
115
+ across environments.
116
+
117
+ **The account needs a stored login plan before any authenticated test can be created.**
118
+ That plan is what a run replays to produce the session its tests ride, and replays again
119
+ when the session expires — so \`tests create\` rejects a \`requires_auth\` plan whose
120
+ account has none, and names the command. Store it once per account with
121
+ \`beryl accounts set-login\` (§ Session mode); everything after that is just authoring.
122
+
123
+ **NEVER paste a real email or password into a plan.** The handles resolve at run time —
124
+ the password never lands in the rendered spec and is scrubbed from artifacts. A pasted
125
+ value is baked into the test forever and rots on the next rotation.
126
+
127
+ ### Why this matters for coverage
128
+
129
+ The account persists between runs, so it accumulates real data — and that is the point. A
130
+ standing account reaches what a fresh signup never could: a populated list, filters with
131
+ something to filter, run history, a dashboard with numbers in it. **When a flow needs
132
+ pre-existing data, that is the signal to use the test account** rather than building the
133
+ data inside the test.
134
+
135
+ The trade is drift — run 200 has 200 of everything run 1 created. So:
136
+
137
+ - **Assert relatively, never absolutely.** "The row I just created is present"
138
+ (\`expect.persisted\` on a \`{{unique}}\`-named row), not "there are 3 rows". A count
139
+ assertion is true in week one and false in week four with nobody having touched it.
140
+ - **Clean up what you create** — put the delete in the plan's \`after\` section, which runs
141
+ on pass AND on fail, unlike a trailing step inside \`steps\`.
142
+
143
+ ### If there is no account yet
144
+
145
+ An empty \`accounts list\` means nothing is set up. In order of preference:
146
+
147
+ 1. **The user has a dedicated test account** → bank it:
148
+ \`beryl accounts create --email <email> --password <password>\`. A DEDICATED test
149
+ account only — never a real user's.
150
+ 2. **No credentials, but the app has a signup form** → let Beryl make one.
151
+ \`beryl accounts create --type beryl\`, then prove it with
152
+ \`beryl accounts provision <id> --file <plan.json>\`. The plan is any ActionPlan that
153
+ ends LOGGED IN — a signup filling \`{{mailbox_address}}\` and \`{{login_password}}\`
154
+ when the account is new, or a **sign-in** when it already exists (you made it by hand,
155
+ or a previous provision succeeded and the record was lost). Both prove the same thing.
156
+ If the app has no password sign-in, pass \`--login-method otp\` (or \`magic_link\`)
157
+ and let the plan \`await_email\` its way through — later sign-ins read the same mailbox.
158
+ 3. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
159
+
160
+ A project with no accounts still falls back to the \`LOGIN_EMAIL\` variable +
161
+ \`LOGIN_PASSWORD\` secret and the handles resolve the same way, so existing tests are
162
+ unaffected — but new work should create an account.
163
+
164
+ ### Session mode: sign in once per run, not once per test
165
+
166
+ Every \`requires_auth\` plan declares its \`auth_mode\` — there is no default. With
167
+ \`"inline"\` the test carries its own sign-in steps and re-types them on every run; that
168
+ is fine and always available.
169
+
170
+ \`auth_mode: "session"\` moves the sign-in out of the test. The account signs in ONCE at
171
+ the start of the run, the resulting browser session is proved live, and every
172
+ \`requires_auth\` test in that run rides it — session-mode tests carry no sign-in steps
173
+ at all, and inline ones simply start already signed in. The test then starts where the
174
+ flow it actually tests begins — no login preamble in the plan, in the replay, or in the
175
+ failure evidence. Nothing has to be turned on for it: every run signs in its accounts and
176
+ hands out the sessions. What a session-mode test DOES need is a stored login plan on its
177
+ account — with none, there is nothing to sign in with, and the test fails at setup with
178
+ \`SESSION_NO_LOGIN_PLAN\` rather than falling back, because it has no sign-in steps of its
179
+ own to fall back to.
180
+
181
+ Set it up once per account:
182
+
183
+ \`\`\`
184
+ beryl accounts set-login <id> --file signin.json --probe probe.json
185
+ beryl accounts check <id> # signs in NOW and proves it — do not skip this
186
+ \`\`\`
187
+
188
+ - **\`--file\`** is the SIGN-IN plan (not the signup): opens the login page, fills
189
+ \`{{login_email}}\` / \`{{login_password}}\`, \`await_email\`s a code if the account
190
+ needs one, and asserts the logged-in shell.
191
+ - **\`--probe\`** is two steps: goto a gated page, then a **positive** assertion that only
192
+ holds when signed in — \`expect.visible\` on the account menu or a "Sign out" control.
193
+ It is required. \`hidden\`, \`count 0\` and a URL match on a redirect ALL pass against a
194
+ logged-out page, so without a positive signal a dead session runs every test logged-out
195
+ and the run still reports green.
196
+
197
+ Then write the tests with \`requires_auth: true\` and \`auth_mode: "session"\`, and NO
198
+ sign-in steps.
199
+
200
+ **Create them one at a time.** Locally there is no run to share a session across, so
201
+ \`tests create\` renders each session-mode test with the account's sign-in in front of it —
202
+ every create performs a real sign-in. Two at once would both trigger a code and race for
203
+ the newest mail in the same mailbox. In the cloud that cost disappears: the whole run
204
+ shares one sign-in.
205
+
206
+ Reading before writing matters here: the login plan self-heals, so
207
+ \`beryl accounts get-login <id>\` first and pass its \`login_plan_hash\` back as
208
+ \`--base-hash\` — a blind overwrite would clobber a repair you never saw.
209
+
210
+ An expired session is never your problem to notice. Beryl proves the stored session at
211
+ the start of every run and, if it no longer works, treats that as a cache miss and signs
212
+ in again from the login plan. Nothing asks a human to reconnect — that message only
213
+ appears for a project with no stored login plan to refresh from.
214
+
215
+ What to expect when it does not work: \`accounts check\` returning
216
+ \`SESSION_LOGIN_FAILED\` means the sign-in plan itself is wrong — fix it.
217
+ \`SESSION_PROOF_FAILED\` means signing in worked but the session could not be carried into
218
+ a fresh browser, because this app keeps its credential somewhere unextractable. That is not
219
+ your bug, but it IS your move: the account is marked unsupported, and its session-mode
220
+ tests fail at setup with \`SESSION_UNSUPPORTED\` on every run until you re-author them
221
+ with \`auth_mode: "inline"\` and their own sign-in steps. Inline tests are unaffected.
222
+
223
+ \`beryl runs local\` works on session-mode tests too — locally the server renders the
224
+ account's sign-in steps in front of the test, so it proves the same thing on your machine.
225
+
226
+ ### Two identities in one test
227
+
228
+ Still not bankable as one test: invite-a-teammate-and-accept-as-them needs two identities
229
+ mid-flow. Bank the half the app shows to account A ("the invitation is listed as pending",
230
+ "the share link is issued") — a real, strong outcome. While AUTHORING you can drive the
231
+ full handshake live: create a second account (\`accounts create --label member\`), or add a
232
+ second mailbox (\`mailbox create --label invitee\`) and read it (\`mailbox read\`).
233
+
234
+ SSO-only sites (no email+password form at all) remain webapp territory.
235
+ ## 2. Author locally over the Playwright MCP
61
236
 
62
237
  1. **Drive the flow in a real browser first.** Use the Playwright MCP to open the app and
63
238
  walk the flow by hand — log in, fill the form, submit, whatever the flow is. You act on
@@ -126,7 +301,7 @@ observable proof the flow worked. Get this right and everything else follows.
126
301
  \`/thank-you\`) and no distinctive destination content is available.
127
302
  - The usual outcome kinds: \`visible\` (the success element showed up), \`have_text\` (an
128
303
  element's text matches), \`have_url\` (the URL contains a value), \`gone\` (an element
129
- disappeared — e.g. a spinner, or the item you just deleted). §1 has the full
304
+ disappeared — e.g. a spinner, or the item you just deleted). §2 has the full
130
305
  \`expect_kind\` list and the step shape.
131
306
  - **\`have_text\` is an EXACT full-text match on the selector's element** — asserting
132
307
  \`have_text: "Documentation"\` on \`body\` fails, because \`body\`'s text includes all the
@@ -139,38 +314,45 @@ observable proof the flow worked. Get this right and everything else follows.
139
314
  genuinely broken in the app, that's a finding to report — not something to paper over
140
315
  with a weaker assertion.
141
316
 
142
- ## 2. What "durable" and "healable" mean here
143
-
144
- Beryl's cloud runs your test on a schedule. When the app's markup drifts and a selector
145
- stops matching, a heal-vs-fail agent decides whether to **heal** the test (silently
146
- re-derive the selector/trajectory and keep it green) or **fail** it (surface a real
147
- regression). It decides that against your test's **intent**:
148
-
149
- - **The natural-language intent is the immutable anchor. Beryl never rewrites it.** It's
150
- the description of what the test proves the load-bearing statement the heal agent
151
- judges every future run against.
152
- - **Selectors and the trajectory are the healable "how".** A button moved, a class name
153
- changed, a step needs an extra click those are mechanics Beryl can re-derive on its
154
- own, because your intent tells it what the flow was *for*.
155
- - **A failed outcome assertion is a real regression Beryl will NOT silently heal green.**
156
- If the success signal from §1 stops holding the confirmation never appears, the page
157
- never renders that's the app breaking, and the test fails loudly. That is the point.
158
-
159
- So a test is *healable* exactly when you gave it **a strong intent + a real outcome
160
- assertion**. A test with a vague intent and a chrome-only assertion is brittle: Beryl
161
- can't tell a real regression from cosmetic drift, so it either heals over real breakage or
162
- fails on noise.
317
+ ### Traps when authoring against a real app
318
+
319
+ Every one of these has produced a wrong plan or a red \`tests create\`. Check them.
320
+
321
+ 1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
322
+ profile, so a session can survive from earlier work. Author while signed in and you
323
+ never see the gate — you will mark gated pages as public. **Log out first**, then
324
+ confirm the page you think is gated really does redirect to the login.
325
+ 2. **The sign-UP flow is not the sign-IN flow.** A brand-new address often gets an extra
326
+ "create your account" step that a returning address skips entirely. An account's stored
327
+ login plan must be the **returning** path that is what runs on every future run. Drive
328
+ it twice: once to create the account, once to see signing in again.
329
+ 3. **Read the DOM, not just the accessibility tree, before picking a selector.** Two
330
+ buttons can share a visible label ("Continue" and "Continue with Google"), and Beryl
331
+ relaxes \`text=X\` to a case-insensitive SUBSTRINGso \`text=Continue\` is ambiguous.
332
+ Find something unique (\`button[type=submit]\`, \`input[name=email]\`,
333
+ \`input[autocomplete=one-time-code]\`) and use that.
334
+ 4. **A single-page app can redirect after the first \`goto\`.** If \`/dashboard\` client-side
335
+ redirects to \`/dashboard/<id>\`, a click fired straight after the goto lands on the
336
+ pre-redirect render and its effect is discarded when the app re-renders a dialog that
337
+ opens and instantly vanishes, for instance. Put a \`wait_for\` on something that exists
338
+ only AFTER the redirect, then act.
339
+ 5. **Never bake an id into a URL.** \`goto /projects/8ab46d63-.../settings\` breaks for any
340
+ other account. Navigate to the stable entry point and click through
341
+ (\`a[href$='/settings']\`), so the plan is about the app, not about your row.
342
+ 6. **Assert durable content, not the empty state.** "No tests yet" is true today and false
343
+ the moment anything exists. Prefer what is structural to the page — a section heading, a
344
+ permanent explainer, a control that is always there.
163
345
 
164
346
  ## 3. Writing the natural-language intent
165
347
 
166
348
  Pass the intent as \`--description\` on \`beryl tests create\` (or \`tests set-plan\` when you
167
- re-author). 1–3 sentences. This is the immutable anchor from §2 — write it well.
349
+ re-author). 1–3 sentences. This is the immutable anchor from §6 — write it well.
168
350
 
169
351
  - **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
170
352
  clicks submit" — that's the trajectory, which Beryl already has and which will change.
171
353
  Instead: *what does a green run prove is true about the app?*
172
354
  - **Name the one observable outcome** that is true only if the flow worked — the same
173
- success signal you asserted in §1, in words.
355
+ success signal you asserted in §2, in words.
174
356
  - **Never describe global chrome.** The intent is about the flow's destination and
175
357
  outcome, not "the header is present".
176
358
 
@@ -188,7 +370,7 @@ test's rendered spec and runs it with your local \`@playwright/test\` — no clo
188
370
  for a scheduled run.
189
371
 
190
372
  \`\`\`
191
- npm i -D @playwright/test && npx playwright install # once
373
+ npm i -D @playwright/test && npx playwright install chromium # once
192
374
  beryl runs local <test-id> --no-sync --url-override http://localhost:3000 --dir ./beryl-local
193
375
  beryl runs local # the whole suite, results recorded in Beryl
194
376
  \`\`\`
@@ -203,17 +385,15 @@ beryl runs local # the whole suite, results recorded in Beryl
203
385
  or assertion failed and why, fix the plan, \`beryl tests set-plan\`, run again.
204
386
  - It exits **0** if every test passed, **1** on a failure — so it drops straight into a
205
387
  run-fix-run loop.
206
- - \`await_email\` steps work locally: the CLI mints the run inbox and answers them over
207
- the API, exactly as the cloud runner would.
208
- - **Saved-login tests work locally.** A plan that fills \`{{login_email}}\` /
209
- \`{{login_password}}\` runs fine: the email is baked into the fetched spec and the
210
- password is revealed once over the logged secret-reveal route, then scrubbed from
211
- any uploaded error text or DOM snapshot. If the LOGIN_EMAIL variable or
212
- LOGIN_PASSWORD secret isn't set, the test is skipped with the exact fix-it command.
213
- - **Captured-session tests stay cloud-only.** A test that signs in with a captured
214
- browser session runs only in Beryl's cloud (which holds the encrypted session — it's
215
- never handed to your disk); \`runs local\` skips it with a note. Run those with
216
- \`beryl runs trigger\`.
388
+ - \`await_email\` steps work locally: the CLI answers them over the API against the same
389
+ mailbox the cloud runner would use, exactly as it would.
390
+ - **Authenticated tests work locally.** A plan that signs itself in by filling
391
+ \`{{login_email}}\` / \`{{login_password}}\` runs fine: the email is baked into the fetched
392
+ spec and the password is revealed once over the logged secret-reveal route, then
393
+ scrubbed from any uploaded error text or DOM snapshot. If the test account it names has
394
+ no password stored, the test is skipped with the exact fix-it command.
395
+ - A test that depends on a session Beryl holds server-side, rather than signing itself in,
396
+ is skipped locally with a note run those with \`beryl runs trigger\`.
217
397
 
218
398
  Once the test passes locally against a real outcome, it's ready to bank and let Beryl run
219
399
  and heal it.
@@ -221,17 +401,30 @@ and heal it.
221
401
  ## 5. Testing an OTP / signup flow (\`await_email\`)
222
402
 
223
403
  A flow that emails the user — a signup verification code, a magic sign-in link, a receipt
224
- — is testable with the \`await_email\` action. Beryl mints a **run-scoped inbox**
225
- automatically whenever a plan contains an \`await_email\` step (or cites
226
- \`{{inbox_address}}\`): no setup, no environment configuration, no flag to turn on. The
227
- minted address is in scope from step 1 as the reserved \`{{inbox_address}}\` handle.
404
+ — is testable with the \`await_email\` action. No setup, no environment configuration, no
405
+ flag to turn on.
406
+
407
+ **The project has one permanent mailbox and all of its mail arrives there.** Two handles
408
+ put an address on the page, and the one you cite decides which *identity* the test acts as:
409
+
410
+ | Handle | Renders as | Use it for |
411
+ |---|---|---|
412
+ | \`{{mailbox_address}}\` | the mailbox's own address, the same every run | signing in as the project's standing test account (§1) |
413
+ | \`{{inbox_address}}\` | a \`+tag\` alias of it, fresh every run | tests whose subject IS getting a NEW identity — a signup, an invited teammate |
414
+
415
+ An alias is a real address the site has never issued, so a signup is repeatable run after
416
+ run; the mail still lands in the same mailbox, and Beryl reads only the alias's own mail.
417
+ Nothing expires and there is no second inbox to manage.
418
+
419
+ Default to \`{{mailbox_address}}\`. Reach for \`{{inbox_address}}\` only when an existing
420
+ account would be rejected — a signup form, or an invite you must accept as a second person.
228
421
 
229
422
  The wiring is a three-part chain:
230
423
 
231
- 1. **Type the minted address into the app** — a \`fill\` with \`value: "{{inbox_address}}"\`.
232
- Every run gets a fresh address, so a signup flow is repeatable by construction (no
233
- \`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other must-not-collide
234
- values like a username).
424
+ 1. **Type the address into the app** — a \`fill\` with \`value: "{{inbox_address}}"\` (or
425
+ \`{{mailbox_address}}\`). An alias is fresh every run, so a signup flow is repeatable by
426
+ construction (no \`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other
427
+ must-not-collide values like a username).
235
428
  2. **Await the mail and bank the extracted value** — an \`await_email\` step with:
236
429
  - \`extract\` (required): \`code\` (an OTP), \`link\` (the sign-in/verify URL), or
237
430
  \`pattern\` (your own regex in \`extract_pattern\`, exactly one capture group).
@@ -258,48 +451,33 @@ For a magic-link flow, replace the code steps with
258
451
  Two caveats:
259
452
 
260
453
  - \`beryl tests create\` verifies an \`await_email\` plan like any other — its local
261
- replay mints a fresh run inbox and answers each step over the API, so the app's mail
262
- really is received and extracted before the test is accepted. (Note the replay signs
263
- up / sends mail for real; pass \`--no-verify\` only if that side effect is unwanted.)
264
- \`beryl runs local\` serves \`await_email\` the same way, so the whole local loop covers
265
- OTP/signup flows end to end.
266
- - The outcome assertion discipline from §1 still applies: the green signal is the
454
+ replay receives at the project mailbox and answers each step over the API, so the app's
455
+ mail really is received and extracted before the test is accepted. (Note the replay
456
+ signs up / sends mail for real; pass \`--no-verify\` only if that side effect is
457
+ unwanted.) \`beryl runs local\` serves \`await_email\` the same way, so the whole local
458
+ loop covers OTP/signup flows end to end.
459
+ - The outcome assertion discipline from §2 still applies: the green signal is the
267
460
  post-verification state (the welcome screen, the dashboard), not "an email arrived".
268
461
 
269
- ## 6. Accounts: minted or saved-login never captured
270
-
271
- How a test gets an account is a fixed decision, made at the start of every authoring
272
- session from \`projects get\`, which reports \`login_email_set\` / \`login_password_set\`.
273
- Both paths are plain steps in the plan no captured sessions, no saved browser
274
- state, no human-in-the-loop login.
275
-
276
- - **Saved login present (both flags true) use it.** The project's dedicated test
277
- account lives in config: the \`LOGIN_EMAIL\` variable + the \`LOGIN_PASSWORD\`
278
- secret. Author the login as ordinary opening steps citing the reserved handles —
279
- fill \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in
280
- shell. NEVER paste the real values into a plan: the handles resolve at run time
281
- (the password never lands in the rendered spec and is scrubbed from artifacts).
282
- To drive the real login live while authoring, read the values with
283
- \`config vars get LOGIN_EMAIL\` and \`config secrets get LOGIN_PASSWORD --reveal\`.
284
- \`tests create\` still replays the whole flow (locally, on your machine) before
285
- banking, so a login that doesn't work is rejected with evidence — nothing is
286
- banked on faith.
287
- - **Not set mint.** \`{{inbox_address}}\` is a fresh real mailbox, minted per test,
288
- per run. Type it into the site's own signup form, \`await_email\` the code or link
289
- (§5). Fresh every run means nothing expires, nothing rots, no state leaks between
290
- tests. Do not ask the user for credentials — mint is the default path.
291
- - **Handed credentials in chat? Bank them first** (\`config vars set LOGIN_EMAIL\`,
292
- \`config secrets set LOGIN_PASSWORD\`), then author with the handles as above.
293
- A DEDICATED test account only — never a real user's.
294
- - **Never hard-code an email address or password in a plan.** Minted inboxes expire
295
- and pasted values rot on rotation — the handles are the only durable references.
296
- - **One identity per banked test.** A flow involving a second account — invite a
297
- teammate and accept as them, share and open as the viewer — is not bankable as one
298
- test yet: a run has one inbox, and \`await_email\` reads only that inbox. Bank the
299
- half the app shows to account A ("the invitation is listed as pending", "the share
300
- link is issued") — a real, strong outcome. While AUTHORING you can still verify the
301
- full handshake live: mint a second inbox (\`inbox create\`), read its mail
302
- (\`inbox read\`), drive both sides in the browser — then bank the single-identity
303
- halves.
304
- - SSO-only sites (no email+password form at all) remain webapp territory.
462
+ ## 6. Why this shape: durable and healable
463
+
464
+ Beryl's cloud runs your test on a schedule. When the app's markup drifts and a selector
465
+ stops matching, a heal-vs-fail agent decides whether to **heal** the test (silently
466
+ re-derive the selector/trajectory and keep it green) or **fail** it (surface a real
467
+ regression). It decides that against your test's **intent**:
468
+
469
+ - **The natural-language intent is the immutable anchor. Beryl never rewrites it.** It's
470
+ the description of what the test proves the load-bearing statement the heal agent
471
+ judges every future run against.
472
+ - **Selectors and the trajectory are the healable "how".** A button moved, a class name
473
+ changed, a step needs an extra click those are mechanics Beryl can re-derive on its
474
+ own, because your intent tells it what the flow was *for*.
475
+ - **A failed outcome assertion is a real regression Beryl will NOT silently heal green.**
476
+ If the success signal from §2 stops holding the confirmation never appears, the page
477
+ never renders that's the app breaking, and the test fails loudly. That is the point.
478
+
479
+ So a test is *healable* exactly when you gave it **a strong intent + a real outcome
480
+ assertion**. A test with a vague intent and a chrome-only assertion is brittle: Beryl
481
+ can't tell a real regression from cosmetic drift, so it either heals over real breakage or
482
+ fails on noise.
305
483
  `;