@beryl-so/cli 0.17.0 → 0.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -11,7 +11,7 @@ export const BERYL_TEST_SKILL_DIR = "beryl-test";
11
11
  // lintPlan — the skill's documented shape must pass `beryl tests lint` on the first try.
12
12
  export const BERYL_TEST_SKILL_EXAMPLE_PLAN = {
13
13
  steps: [
14
- { action: "goto", url: "https://app.example.com/pricing" },
14
+ { action: "goto", url: "/pricing" },
15
15
  {
16
16
  action: "expect",
17
17
  expect_kind: "have_text",
@@ -26,7 +26,7 @@ export const BERYL_TEST_SKILL_EXAMPLE_PLAN = {
26
26
  // pass `beryl tests lint` on the first try.
27
27
  export const BERYL_TEST_SKILL_OTP_EXAMPLE_PLAN = {
28
28
  steps: [
29
- { action: "goto", url: "https://app.example.com/signup" },
29
+ { action: "goto", url: "/signup" },
30
30
  { action: "fill", selector: "input[name=email]", value: "{{inbox_address}}" },
31
31
  { action: "click", selector: "button[type=submit]" },
32
32
  {
@@ -57,7 +57,183 @@ because Beryl can **heal** them — but only when you give it what it needs to.
57
57
  Read this before authoring. The three ideas that make a test durable: a real **outcome
58
58
  assertion**, a strong **natural-language intent**, and the **local run-fix loop**.
59
59
 
60
- ## 1. Author locally over the Playwright MCP
60
+ ## 0. Start here
61
+
62
+ **Everything happens on your machine first.** You drive the flow in a real browser here,
63
+ and \`tests create\` replays the plan here — over MCP that means the machine running the
64
+ MCP server, never Beryl's. A red replay banks nothing. Two exceptions to "proven
65
+ locally": a plan that depends on a session only Beryl's cloud holds is verified
66
+ server-side instead (create tells you when), and \`--no-verify\` banks unproven — avoid
67
+ it. So a broken local browser is not a detail you can skip past; it is the whole loop.
68
+
69
+ Before the first plan:
70
+
71
+ \`\`\`
72
+ npm i -D @playwright/test && npx playwright install chromium # once, per project
73
+ beryl envs list # the root URL gotos resolve against (§2)
74
+ beryl envs update <id> --url https://app.example.com # set it if root_url is empty — required
75
+ beryl accounts list # who do authenticated tests sign in as? (§1)
76
+ beryl accounts set-login <id> --file … --probe … # store its sign-in — required (§1)
77
+ beryl accounts check <id> # does that stored sign-in still work? (§1)
78
+ beryl mailbox get # the address they receive mail at (§5)
79
+ \`\`\`
80
+
81
+ \`accounts list\` comes FIRST, before you author anything. It decides whether a flow signs
82
+ in as a standing account or acquires a new identity, and that choice changes the plan you
83
+ write — discovering it afterwards means rewriting. If a flow needs signing in, the
84
+ account's login plan has to be stored before you can create the test at all.
85
+
86
+ Then, per test:
87
+
88
+ 1. **Drive the flow** in a real browser over the Playwright MCP — never author from
89
+ imagination (§2).
90
+ 2. **Write the ActionPlan**, with one real outcome assertion (§2).
91
+ 3. \`beryl tests lint --file plan.json\` — schema check, offline, no network.
92
+ 4. \`beryl tests create --title … --file … --description "<the intent>"\` — replays the plan
93
+ locally and banks it only if it goes green. A red replay banks nothing and hands back
94
+ the failure; fix the file and re-run (§3 for the intent, §4 for the loop).
95
+ 5. \`beryl runs local\` — re-run banked tests on your machine while iterating.
96
+ 6. \`beryl runs trigger\` — hand it to Beryl's cloud, on demand or on a schedule.
97
+
98
+ ## 1. First: who do your tests sign in as?
99
+
100
+ An authenticated test **signs in as an account that already exists** — it does not sign
101
+ one up. Start every authoring session with \`beryl accounts list\`:
102
+
103
+ \`\`\`
104
+ label email type login status
105
+ default * qa@acme.test user_provided password ready
106
+ admin qa+admin@x7k2p9.email.beryl.so beryl otp ready
107
+ \`\`\`
108
+
109
+ Author the sign-in as ordinary opening steps citing the reserved handles — fill
110
+ \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell. Set
111
+ \`requires_auth: true\` AND \`auth_mode\` — both, always. \`auth_mode\` has NO default and
112
+ a \`requires_auth\` plan without it is rejected at create: \`"inline"\` when the plan
113
+ signs itself in like this, \`"session"\` when it carries no sign-in steps and rides its
114
+ account's once-per-run session (§ Session mode). Add \`auth_label\` only to pick a
115
+ non-default identity (the row marked \`*\` is what a plan gets otherwise). The handles
116
+ resolve at run time to whichever account the plan named, so one plan stays correct
117
+ across environments.
118
+
119
+ **The account needs a stored login plan before any authenticated test can be created.**
120
+ That plan is what a run replays to produce the session its tests ride, and replays again
121
+ when the session expires — so \`tests create\` rejects a \`requires_auth\` plan whose
122
+ account has none, and names the command. Store it once per account with
123
+ \`beryl accounts set-login\` (§ Session mode); everything after that is just authoring.
124
+
125
+ **NEVER paste a real email or password into a plan.** The handles resolve at run time —
126
+ the password never lands in the rendered spec and is scrubbed from artifacts. A pasted
127
+ value is baked into the test forever and rots on the next rotation.
128
+
129
+ ### Why this matters for coverage
130
+
131
+ The account persists between runs, so it accumulates real data — and that is the point. A
132
+ standing account reaches what a fresh signup never could: a populated list, filters with
133
+ something to filter, run history, a dashboard with numbers in it. **When a flow needs
134
+ pre-existing data, that is the signal to use the test account** rather than building the
135
+ data inside the test.
136
+
137
+ The trade is drift — run 200 has 200 of everything run 1 created. So:
138
+
139
+ - **Assert relatively, never absolutely.** "The row I just created is present"
140
+ (\`expect.persisted\` on a \`{{unique}}\`-named row), not "there are 3 rows". A count
141
+ assertion is true in week one and false in week four with nobody having touched it.
142
+ - **Clean up what you create** — put the delete in the plan's \`after\` section, which runs
143
+ on pass AND on fail, unlike a trailing step inside \`steps\`.
144
+
145
+ ### If there is no account yet
146
+
147
+ An empty \`accounts list\` means nothing is set up. In order of preference:
148
+
149
+ 1. **The user has a dedicated test account** → bank it:
150
+ \`beryl accounts create --email <email> --password <password>\`. A DEDICATED test
151
+ account only — never a real user's.
152
+ 2. **No credentials, but the app has a signup form** → let Beryl make one.
153
+ \`beryl accounts create --type beryl\`, then prove it with
154
+ \`beryl accounts provision <id> --file <plan.json>\`. The plan is any ActionPlan that
155
+ ends LOGGED IN — a signup filling \`{{mailbox_address}}\` and \`{{login_password}}\`
156
+ when the account is new, or a **sign-in** when it already exists (you made it by hand,
157
+ or a previous provision succeeded and the record was lost). Both prove the same thing.
158
+ If the app has no password sign-in, pass \`--login-method otp\` (or \`magic_link\`)
159
+ and let the plan \`await_email\` its way through — later sign-ins read the same mailbox.
160
+ 3. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
161
+
162
+ A project with no accounts still falls back to the \`LOGIN_EMAIL\` variable +
163
+ \`LOGIN_PASSWORD\` secret and the handles resolve the same way, so existing tests are
164
+ unaffected — but new work should create an account.
165
+
166
+ ### Session mode: sign in once per run, not once per test
167
+
168
+ Every \`requires_auth\` plan declares its \`auth_mode\` — there is no default. With
169
+ \`"inline"\` the test carries its own sign-in steps and re-types them on every run; that
170
+ is fine and always available.
171
+
172
+ \`auth_mode: "session"\` moves the sign-in out of the test. The account signs in ONCE at
173
+ the start of the run, the resulting browser session is proved live, and every
174
+ \`requires_auth\` test in that run rides it — session-mode tests carry no sign-in steps
175
+ at all, and inline ones simply start already signed in. The test then starts where the
176
+ flow it actually tests begins — no login preamble in the plan, in the replay, or in the
177
+ failure evidence. Nothing has to be turned on for it: every run signs in its accounts and
178
+ hands out the sessions. What a session-mode test DOES need is a stored login plan on its
179
+ account — with none, there is nothing to sign in with, and the test fails at setup with
180
+ \`SESSION_NO_LOGIN_PLAN\` rather than falling back, because it has no sign-in steps of its
181
+ own to fall back to.
182
+
183
+ Set it up once per account:
184
+
185
+ \`\`\`
186
+ beryl accounts set-login <id> --file signin.json --probe probe.json
187
+ beryl accounts check <id> # signs in NOW and proves it — do not skip this
188
+ \`\`\`
189
+
190
+ - **\`--file\`** is the SIGN-IN plan (not the signup): opens the login page, fills
191
+ \`{{login_email}}\` / \`{{login_password}}\`, \`await_email\`s a code if the account
192
+ needs one, and asserts the logged-in shell.
193
+ - **\`--probe\`** is two steps: goto a gated page, then a **positive** assertion that only
194
+ holds when signed in — \`expect.visible\` on the account menu or a "Sign out" control.
195
+ It is required. \`hidden\`, \`count 0\` and a URL match on a redirect ALL pass against a
196
+ logged-out page, so without a positive signal a dead session runs every test logged-out
197
+ and the run still reports green.
198
+
199
+ Then write the tests with \`requires_auth: true\` and \`auth_mode: "session"\`, and NO
200
+ sign-in steps.
201
+
202
+ **Creates share the sign-in.** \`tests create\` replays a session-mode plan against the
203
+ account's established session — reused from the last proven sign-in when it is still
204
+ live — so a batch of creates costs at most one sign-in, same as a run. Still create one
205
+ at a time: two racing creates would race the same mailbox for mail.
206
+
207
+ Reading before writing matters here: the login plan self-heals, so
208
+ \`beryl accounts get-login <id>\` first and pass its \`login_plan_hash\` back as
209
+ \`--base-hash\` — a blind overwrite would clobber a repair you never saw.
210
+
211
+ An expired session is never your problem to notice. Beryl proves the stored session at
212
+ the start of every run and, if it no longer works, treats that as a cache miss and signs
213
+ in again from the login plan. Nothing asks a human to reconnect — that message only
214
+ appears for a project with no stored login plan to refresh from.
215
+
216
+ What to expect when it does not work: \`accounts check\` returning
217
+ \`SESSION_LOGIN_FAILED\` means the sign-in plan itself is wrong — fix it.
218
+ \`SESSION_PROOF_FAILED\` means signing in worked but the session could not be carried into
219
+ a fresh browser, because this app keeps its credential somewhere unextractable. That is not
220
+ your bug, but it IS your move: the account is marked unsupported, and its session-mode
221
+ tests fail at setup with \`SESSION_UNSUPPORTED\` on every run until you re-author them
222
+ with \`auth_mode: "inline"\` and their own sign-in steps. Inline tests are unaffected.
223
+
224
+ \`beryl runs local\` works on session-mode tests too — same shape as the cloud: one
225
+ sign-in per invocation, shared by every session-mode test.
226
+
227
+ ### Two identities in one test
228
+
229
+ Still not bankable as one test: invite-a-teammate-and-accept-as-them needs two identities
230
+ mid-flow. Bank the half the app shows to account A ("the invitation is listed as pending",
231
+ "the share link is issued") — a real, strong outcome. While AUTHORING you can drive the
232
+ full handshake live: create a second account (\`accounts create --label member\`), or add a
233
+ second mailbox (\`mailbox create --label invitee\`) and read it (\`mailbox read\`).
234
+
235
+ SSO-only sites (no email+password form at all) remain webapp territory.
236
+ ## 2. Author locally over the Playwright MCP
61
237
 
62
238
  1. **Drive the flow in a real browser first.** Use the Playwright MCP to open the app and
63
239
  walk the flow by hand — log in, fill the form, submit, whatever the flow is. You act on
@@ -67,6 +243,11 @@ assertion**, a strong **natural-language intent**, and the **local run-fix loop*
67
243
  \`{action, selector, url, value, ...}\`. Two structural rules the plan must satisfy:
68
244
  - the **first executed step is a \`goto\`** (the flow has to start by navigating somewhere), and
69
245
  - **at least one step is an \`expect\`** (a test that asserts nothing is not a test).
246
+ **A \`goto\` at your own app is a PATH, never a full URL** — \`/pricing\`, not
247
+ \`https://app.example.com/pricing\`. The origin comes from the environment's root URL, so
248
+ one plan runs against prod, staging and a preview. Bake the origin in and \`--env\` is
249
+ silently ignored: the test keeps hitting whatever host you typed. Absolute URLs stay
250
+ legal for OTHER origins (an OAuth handoff, a magic link on another domain).
70
251
  Optional \`before\` / \`after\` arrays hold setup and teardown; \`after\` runs even when a
71
252
  main step fails, so a create/update/delete flow can clean up the record it made.
72
253
 
@@ -126,7 +307,7 @@ observable proof the flow worked. Get this right and everything else follows.
126
307
  \`/thank-you\`) and no distinctive destination content is available.
127
308
  - The usual outcome kinds: \`visible\` (the success element showed up), \`have_text\` (an
128
309
  element's text matches), \`have_url\` (the URL contains a value), \`gone\` (an element
129
- disappeared — e.g. a spinner, or the item you just deleted). §1 has the full
310
+ disappeared — e.g. a spinner, or the item you just deleted). §2 has the full
130
311
  \`expect_kind\` list and the step shape.
131
312
  - **\`have_text\` is an EXACT full-text match on the selector's element** — asserting
132
313
  \`have_text: "Documentation"\` on \`body\` fails, because \`body\`'s text includes all the
@@ -139,38 +320,45 @@ observable proof the flow worked. Get this right and everything else follows.
139
320
  genuinely broken in the app, that's a finding to report — not something to paper over
140
321
  with a weaker assertion.
141
322
 
142
- ## 2. What "durable" and "healable" mean here
143
-
144
- Beryl's cloud runs your test on a schedule. When the app's markup drifts and a selector
145
- stops matching, a heal-vs-fail agent decides whether to **heal** the test (silently
146
- re-derive the selector/trajectory and keep it green) or **fail** it (surface a real
147
- regression). It decides that against your test's **intent**:
148
-
149
- - **The natural-language intent is the immutable anchor. Beryl never rewrites it.** It's
150
- the description of what the test proves the load-bearing statement the heal agent
151
- judges every future run against.
152
- - **Selectors and the trajectory are the healable "how".** A button moved, a class name
153
- changed, a step needs an extra click those are mechanics Beryl can re-derive on its
154
- own, because your intent tells it what the flow was *for*.
155
- - **A failed outcome assertion is a real regression Beryl will NOT silently heal green.**
156
- If the success signal from §1 stops holding the confirmation never appears, the page
157
- never renders that's the app breaking, and the test fails loudly. That is the point.
158
-
159
- So a test is *healable* exactly when you gave it **a strong intent + a real outcome
160
- assertion**. A test with a vague intent and a chrome-only assertion is brittle: Beryl
161
- can't tell a real regression from cosmetic drift, so it either heals over real breakage or
162
- fails on noise.
323
+ ### Traps when authoring against a real app
324
+
325
+ Every one of these has produced a wrong plan or a red \`tests create\`. Check them.
326
+
327
+ 1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
328
+ profile, so a session can survive from earlier work. Author while signed in and you
329
+ never see the gate — you will mark gated pages as public. **Log out first**, then
330
+ confirm the page you think is gated really does redirect to the login.
331
+ 2. **The sign-UP flow is not the sign-IN flow.** A brand-new address often gets an extra
332
+ "create your account" step that a returning address skips entirely. An account's stored
333
+ login plan must be the **returning** path that is what runs on every future run. Drive
334
+ it twice: once to create the account, once to see signing in again.
335
+ 3. **Read the DOM, not just the accessibility tree, before picking a selector.** Two
336
+ buttons can share a visible label ("Continue" and "Continue with Google"), and Beryl
337
+ relaxes \`text=X\` to a case-insensitive SUBSTRINGso \`text=Continue\` is ambiguous.
338
+ Find something unique (\`button[type=submit]\`, \`input[name=email]\`,
339
+ \`input[autocomplete=one-time-code]\`) and use that.
340
+ 4. **A single-page app can redirect after the first \`goto\`.** If \`/dashboard\` client-side
341
+ redirects to \`/dashboard/<id>\`, a click fired straight after the goto lands on the
342
+ pre-redirect render and its effect is discarded when the app re-renders a dialog that
343
+ opens and instantly vanishes, for instance. Put a \`wait_for\` on something that exists
344
+ only AFTER the redirect, then act.
345
+ 5. **Never bake an id into a URL.** \`goto /projects/8ab46d63-.../settings\` breaks for any
346
+ other account. Navigate to the stable entry point and click through
347
+ (\`a[href$='/settings']\`), so the plan is about the app, not about your row.
348
+ 6. **Assert durable content, not the empty state.** "No tests yet" is true today and false
349
+ the moment anything exists. Prefer what is structural to the page — a section heading, a
350
+ permanent explainer, a control that is always there.
163
351
 
164
352
  ## 3. Writing the natural-language intent
165
353
 
166
354
  Pass the intent as \`--description\` on \`beryl tests create\` (or \`tests set-plan\` when you
167
- re-author). 1–3 sentences. This is the immutable anchor from §2 — write it well.
355
+ re-author). 1–3 sentences. This is the immutable anchor from §6 — write it well.
168
356
 
169
357
  - **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
170
358
  clicks submit" — that's the trajectory, which Beryl already has and which will change.
171
359
  Instead: *what does a green run prove is true about the app?*
172
360
  - **Name the one observable outcome** that is true only if the flow worked — the same
173
- success signal you asserted in §1, in words.
361
+ success signal you asserted in §2, in words.
174
362
  - **Never describe global chrome.** The intent is about the flow's destination and
175
363
  outcome, not "the header is present".
176
364
 
@@ -188,7 +376,7 @@ test's rendered spec and runs it with your local \`@playwright/test\` — no clo
188
376
  for a scheduled run.
189
377
 
190
378
  \`\`\`
191
- npm i -D @playwright/test && npx playwright install # once
379
+ npm i -D @playwright/test && npx playwright install chromium # once
192
380
  beryl runs local <test-id> --no-sync --url-override http://localhost:3000 --dir ./beryl-local
193
381
  beryl runs local # the whole suite, results recorded in Beryl
194
382
  \`\`\`
@@ -197,23 +385,22 @@ beryl runs local # the whole suite, results recorded in Beryl
197
385
  - **Results sync to Beryl by default** — the finished run is imported as a first-class
198
386
  run (history, replay, report; trigger source \`local\`). While ITERATING on a draft,
199
387
  pass \`--no-sync\` so every fix-loop attempt doesn't land in the project's run history.
200
- - \`--url-override\` points the run at a local dev server or a preview deploy.
388
+ - \`--url-override\` swaps the root URL for THIS run for a throwaway host (a dev server, a
389
+ per-PR preview). A standing environment is \`--env <id>\` instead, not an override.
201
390
  - \`--dir\` keeps the **spec, artifacts, and a JSON \`report.json\`** on disk so you (or your
202
391
  coding agent) can read exactly what happened and iterate: read the report, see which step
203
392
  or assertion failed and why, fix the plan, \`beryl tests set-plan\`, run again.
204
393
  - It exits **0** if every test passed, **1** on a failure — so it drops straight into a
205
394
  run-fix-run loop.
206
- - \`await_email\` steps work locally: the CLI mints the run inbox and answers them over
207
- the API, exactly as the cloud runner would.
208
- - **Saved-login tests work locally.** A plan that fills \`{{login_email}}\` /
209
- \`{{login_password}}\` runs fine: the email is baked into the fetched spec and the
210
- password is revealed once over the logged secret-reveal route, then scrubbed from
211
- any uploaded error text or DOM snapshot. If the LOGIN_EMAIL variable or
212
- LOGIN_PASSWORD secret isn't set, the test is skipped with the exact fix-it command.
213
- - **Captured-session tests stay cloud-only.** A test that signs in with a captured
214
- browser session runs only in Beryl's cloud (which holds the encrypted session — it's
215
- never handed to your disk); \`runs local\` skips it with a note. Run those with
216
- \`beryl runs trigger\`.
395
+ - \`await_email\` steps work locally: the CLI answers them over the API against the same
396
+ mailbox the cloud runner would use, exactly as it would.
397
+ - **Authenticated tests work locally.** A plan that signs itself in by filling
398
+ \`{{login_email}}\` / \`{{login_password}}\` runs fine: the email is baked into the fetched
399
+ spec and the password is revealed once over the logged secret-reveal route, then
400
+ scrubbed from any uploaded error text or DOM snapshot. If the test account it names has
401
+ no password stored, the test is skipped with the exact fix-it command.
402
+ - A test that depends on a session Beryl holds server-side, rather than signing itself in,
403
+ is skipped locally with a note run those with \`beryl runs trigger\`.
217
404
 
218
405
  Once the test passes locally against a real outcome, it's ready to bank and let Beryl run
219
406
  and heal it.
@@ -221,17 +408,30 @@ and heal it.
221
408
  ## 5. Testing an OTP / signup flow (\`await_email\`)
222
409
 
223
410
  A flow that emails the user — a signup verification code, a magic sign-in link, a receipt
224
- — is testable with the \`await_email\` action. Beryl mints a **run-scoped inbox**
225
- automatically whenever a plan contains an \`await_email\` step (or cites
226
- \`{{inbox_address}}\`): no setup, no environment configuration, no flag to turn on. The
227
- minted address is in scope from step 1 as the reserved \`{{inbox_address}}\` handle.
411
+ — is testable with the \`await_email\` action. No setup, no environment configuration, no
412
+ flag to turn on.
413
+
414
+ **The project has one permanent mailbox and all of its mail arrives there.** Two handles
415
+ put an address on the page, and the one you cite decides which *identity* the test acts as:
416
+
417
+ | Handle | Renders as | Use it for |
418
+ |---|---|---|
419
+ | \`{{mailbox_address}}\` | the mailbox's own address, the same every run | signing in as the project's standing test account (§1) |
420
+ | \`{{inbox_address}}\` | a \`+tag\` alias of it, fresh every run | tests whose subject IS getting a NEW identity — a signup, an invited teammate |
421
+
422
+ An alias is a real address the site has never issued, so a signup is repeatable run after
423
+ run; the mail still lands in the same mailbox, and Beryl reads only the alias's own mail.
424
+ Nothing expires and there is no second inbox to manage.
425
+
426
+ Default to \`{{mailbox_address}}\`. Reach for \`{{inbox_address}}\` only when an existing
427
+ account would be rejected — a signup form, or an invite you must accept as a second person.
228
428
 
229
429
  The wiring is a three-part chain:
230
430
 
231
- 1. **Type the minted address into the app** — a \`fill\` with \`value: "{{inbox_address}}"\`.
232
- Every run gets a fresh address, so a signup flow is repeatable by construction (no
233
- \`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other must-not-collide
234
- values like a username).
431
+ 1. **Type the address into the app** — a \`fill\` with \`value: "{{inbox_address}}"\` (or
432
+ \`{{mailbox_address}}\`). An alias is fresh every run, so a signup flow is repeatable by
433
+ construction (no \`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other
434
+ must-not-collide values like a username).
235
435
  2. **Await the mail and bank the extracted value** — an \`await_email\` step with:
236
436
  - \`extract\` (required): \`code\` (an OTP), \`link\` (the sign-in/verify URL), or
237
437
  \`pattern\` (your own regex in \`extract_pattern\`, exactly one capture group).
@@ -258,48 +458,33 @@ For a magic-link flow, replace the code steps with
258
458
  Two caveats:
259
459
 
260
460
  - \`beryl tests create\` verifies an \`await_email\` plan like any other — its local
261
- replay mints a fresh run inbox and answers each step over the API, so the app's mail
262
- really is received and extracted before the test is accepted. (Note the replay signs
263
- up / sends mail for real; pass \`--no-verify\` only if that side effect is unwanted.)
264
- \`beryl runs local\` serves \`await_email\` the same way, so the whole local loop covers
265
- OTP/signup flows end to end.
266
- - The outcome assertion discipline from §1 still applies: the green signal is the
461
+ replay receives at the project mailbox and answers each step over the API, so the app's
462
+ mail really is received and extracted before the test is accepted. (Note the replay
463
+ signs up / sends mail for real; pass \`--no-verify\` only if that side effect is
464
+ unwanted.) \`beryl runs local\` serves \`await_email\` the same way, so the whole local
465
+ loop covers OTP/signup flows end to end.
466
+ - The outcome assertion discipline from §2 still applies: the green signal is the
267
467
  post-verification state (the welcome screen, the dashboard), not "an email arrived".
268
468
 
269
- ## 6. Accounts: minted or saved-login never captured
270
-
271
- How a test gets an account is a fixed decision, made at the start of every authoring
272
- session from \`projects get\`, which reports \`login_email_set\` / \`login_password_set\`.
273
- Both paths are plain steps in the plan no captured sessions, no saved browser
274
- state, no human-in-the-loop login.
275
-
276
- - **Saved login present (both flags true) use it.** The project's dedicated test
277
- account lives in config: the \`LOGIN_EMAIL\` variable + the \`LOGIN_PASSWORD\`
278
- secret. Author the login as ordinary opening steps citing the reserved handles —
279
- fill \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in
280
- shell. NEVER paste the real values into a plan: the handles resolve at run time
281
- (the password never lands in the rendered spec and is scrubbed from artifacts).
282
- To drive the real login live while authoring, read the values with
283
- \`config vars get LOGIN_EMAIL\` and \`config secrets get LOGIN_PASSWORD --reveal\`.
284
- \`tests create\` still replays the whole flow (locally, on your machine) before
285
- banking, so a login that doesn't work is rejected with evidence — nothing is
286
- banked on faith.
287
- - **Not set mint.** \`{{inbox_address}}\` is a fresh real mailbox, minted per test,
288
- per run. Type it into the site's own signup form, \`await_email\` the code or link
289
- (§5). Fresh every run means nothing expires, nothing rots, no state leaks between
290
- tests. Do not ask the user for credentials — mint is the default path.
291
- - **Handed credentials in chat? Bank them first** (\`config vars set LOGIN_EMAIL\`,
292
- \`config secrets set LOGIN_PASSWORD\`), then author with the handles as above.
293
- A DEDICATED test account only — never a real user's.
294
- - **Never hard-code an email address or password in a plan.** Minted inboxes expire
295
- and pasted values rot on rotation — the handles are the only durable references.
296
- - **One identity per banked test.** A flow involving a second account — invite a
297
- teammate and accept as them, share and open as the viewer — is not bankable as one
298
- test yet: a run has one inbox, and \`await_email\` reads only that inbox. Bank the
299
- half the app shows to account A ("the invitation is listed as pending", "the share
300
- link is issued") — a real, strong outcome. While AUTHORING you can still verify the
301
- full handshake live: mint a second inbox (\`inbox create\`), read its mail
302
- (\`inbox read\`), drive both sides in the browser — then bank the single-identity
303
- halves.
304
- - SSO-only sites (no email+password form at all) remain webapp territory.
469
+ ## 6. Why this shape: durable and healable
470
+
471
+ Beryl's cloud runs your test on a schedule. When the app's markup drifts and a selector
472
+ stops matching, a heal-vs-fail agent decides whether to **heal** the test (silently
473
+ re-derive the selector/trajectory and keep it green) or **fail** it (surface a real
474
+ regression). It decides that against your test's **intent**:
475
+
476
+ - **The natural-language intent is the immutable anchor. Beryl never rewrites it.** It's
477
+ the description of what the test proves the load-bearing statement the heal agent
478
+ judges every future run against.
479
+ - **Selectors and the trajectory are the healable "how".** A button moved, a class name
480
+ changed, a step needs an extra click those are mechanics Beryl can re-derive on its
481
+ own, because your intent tells it what the flow was *for*.
482
+ - **A failed outcome assertion is a real regression Beryl will NOT silently heal green.**
483
+ If the success signal from §2 stops holding the confirmation never appears, the page
484
+ never renders that's the app breaking, and the test fails loudly. That is the point.
485
+
486
+ So a test is *healable* exactly when you gave it **a strong intent + a real outcome
487
+ assertion**. A test with a vague intent and a chrome-only assertion is brittle: Beryl
488
+ can't tell a real regression from cosmetic drift, so it either heals over real breakage or
489
+ fails on noise.
305
490
  `;