@beryl-so/cli 0.17.0 → 0.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -7
- package/dist/adapters/cli.js +15 -1
- package/dist/adapters/mcp.js +44 -5
- package/dist/beryl-test-skill.js +275 -90
- package/dist/commands/accounts.js +331 -0
- package/dist/commands/environments.js +0 -2
- package/dist/commands/init.js +43 -27
- package/dist/commands/mailboxes.js +159 -0
- package/dist/commands/mcp.js +25 -0
- package/dist/commands/projects.js +0 -6
- package/dist/commands/runs.js +159 -46
- package/dist/commands/tests.js +29 -6
- package/dist/email-pump.js +5 -2
- package/dist/http.js +4 -1
- package/dist/local-exec.js +86 -27
- package/dist/local-run.js +21 -3
- package/dist/playwright-install.js +118 -7
- package/dist/registry/index.js +4 -2
- package/dist/schema.generated.js +32 -0
- package/package.json +2 -2
- package/dist/commands/inboxes.js +0 -145
package/dist/beryl-test-skill.js
CHANGED
|
@@ -11,7 +11,7 @@ export const BERYL_TEST_SKILL_DIR = "beryl-test";
|
|
|
11
11
|
// lintPlan — the skill's documented shape must pass `beryl tests lint` on the first try.
|
|
12
12
|
export const BERYL_TEST_SKILL_EXAMPLE_PLAN = {
|
|
13
13
|
steps: [
|
|
14
|
-
{ action: "goto", url: "
|
|
14
|
+
{ action: "goto", url: "/pricing" },
|
|
15
15
|
{
|
|
16
16
|
action: "expect",
|
|
17
17
|
expect_kind: "have_text",
|
|
@@ -26,7 +26,7 @@ export const BERYL_TEST_SKILL_EXAMPLE_PLAN = {
|
|
|
26
26
|
// pass `beryl tests lint` on the first try.
|
|
27
27
|
export const BERYL_TEST_SKILL_OTP_EXAMPLE_PLAN = {
|
|
28
28
|
steps: [
|
|
29
|
-
{ action: "goto", url: "
|
|
29
|
+
{ action: "goto", url: "/signup" },
|
|
30
30
|
{ action: "fill", selector: "input[name=email]", value: "{{inbox_address}}" },
|
|
31
31
|
{ action: "click", selector: "button[type=submit]" },
|
|
32
32
|
{
|
|
@@ -57,7 +57,183 @@ because Beryl can **heal** them — but only when you give it what it needs to.
|
|
|
57
57
|
Read this before authoring. The three ideas that make a test durable: a real **outcome
|
|
58
58
|
assertion**, a strong **natural-language intent**, and the **local run-fix loop**.
|
|
59
59
|
|
|
60
|
-
##
|
|
60
|
+
## 0. Start here
|
|
61
|
+
|
|
62
|
+
**Everything happens on your machine first.** You drive the flow in a real browser here,
|
|
63
|
+
and \`tests create\` replays the plan here — over MCP that means the machine running the
|
|
64
|
+
MCP server, never Beryl's. A red replay banks nothing. Two exceptions to "proven
|
|
65
|
+
locally": a plan that depends on a session only Beryl's cloud holds is verified
|
|
66
|
+
server-side instead (create tells you when), and \`--no-verify\` banks unproven — avoid
|
|
67
|
+
it. So a broken local browser is not a detail you can skip past; it is the whole loop.
|
|
68
|
+
|
|
69
|
+
Before the first plan:
|
|
70
|
+
|
|
71
|
+
\`\`\`
|
|
72
|
+
npm i -D @playwright/test && npx playwright install chromium # once, per project
|
|
73
|
+
beryl envs list # the root URL gotos resolve against (§2)
|
|
74
|
+
beryl envs update <id> --url https://app.example.com # set it if root_url is empty — required
|
|
75
|
+
beryl accounts list # who do authenticated tests sign in as? (§1)
|
|
76
|
+
beryl accounts set-login <id> --file … --probe … # store its sign-in — required (§1)
|
|
77
|
+
beryl accounts check <id> # does that stored sign-in still work? (§1)
|
|
78
|
+
beryl mailbox get # the address they receive mail at (§5)
|
|
79
|
+
\`\`\`
|
|
80
|
+
|
|
81
|
+
\`accounts list\` comes FIRST, before you author anything. It decides whether a flow signs
|
|
82
|
+
in as a standing account or acquires a new identity, and that choice changes the plan you
|
|
83
|
+
write — discovering it afterwards means rewriting. If a flow needs signing in, the
|
|
84
|
+
account's login plan has to be stored before you can create the test at all.
|
|
85
|
+
|
|
86
|
+
Then, per test:
|
|
87
|
+
|
|
88
|
+
1. **Drive the flow** in a real browser over the Playwright MCP — never author from
|
|
89
|
+
imagination (§2).
|
|
90
|
+
2. **Write the ActionPlan**, with one real outcome assertion (§2).
|
|
91
|
+
3. \`beryl tests lint --file plan.json\` — schema check, offline, no network.
|
|
92
|
+
4. \`beryl tests create --title … --file … --description "<the intent>"\` — replays the plan
|
|
93
|
+
locally and banks it only if it goes green. A red replay banks nothing and hands back
|
|
94
|
+
the failure; fix the file and re-run (§3 for the intent, §4 for the loop).
|
|
95
|
+
5. \`beryl runs local\` — re-run banked tests on your machine while iterating.
|
|
96
|
+
6. \`beryl runs trigger\` — hand it to Beryl's cloud, on demand or on a schedule.
|
|
97
|
+
|
|
98
|
+
## 1. First: who do your tests sign in as?
|
|
99
|
+
|
|
100
|
+
An authenticated test **signs in as an account that already exists** — it does not sign
|
|
101
|
+
one up. Start every authoring session with \`beryl accounts list\`:
|
|
102
|
+
|
|
103
|
+
\`\`\`
|
|
104
|
+
label email type login status
|
|
105
|
+
default * qa@acme.test user_provided password ready
|
|
106
|
+
admin qa+admin@x7k2p9.email.beryl.so beryl otp ready
|
|
107
|
+
\`\`\`
|
|
108
|
+
|
|
109
|
+
Author the sign-in as ordinary opening steps citing the reserved handles — fill
|
|
110
|
+
\`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell. Set
|
|
111
|
+
\`requires_auth: true\` AND \`auth_mode\` — both, always. \`auth_mode\` has NO default and
|
|
112
|
+
a \`requires_auth\` plan without it is rejected at create: \`"inline"\` when the plan
|
|
113
|
+
signs itself in like this, \`"session"\` when it carries no sign-in steps and rides its
|
|
114
|
+
account's once-per-run session (§ Session mode). Add \`auth_label\` only to pick a
|
|
115
|
+
non-default identity (the row marked \`*\` is what a plan gets otherwise). The handles
|
|
116
|
+
resolve at run time to whichever account the plan named, so one plan stays correct
|
|
117
|
+
across environments.
|
|
118
|
+
|
|
119
|
+
**The account needs a stored login plan before any authenticated test can be created.**
|
|
120
|
+
That plan is what a run replays to produce the session its tests ride, and replays again
|
|
121
|
+
when the session expires — so \`tests create\` rejects a \`requires_auth\` plan whose
|
|
122
|
+
account has none, and names the command. Store it once per account with
|
|
123
|
+
\`beryl accounts set-login\` (§ Session mode); everything after that is just authoring.
|
|
124
|
+
|
|
125
|
+
**NEVER paste a real email or password into a plan.** The handles resolve at run time —
|
|
126
|
+
the password never lands in the rendered spec and is scrubbed from artifacts. A pasted
|
|
127
|
+
value is baked into the test forever and rots on the next rotation.
|
|
128
|
+
|
|
129
|
+
### Why this matters for coverage
|
|
130
|
+
|
|
131
|
+
The account persists between runs, so it accumulates real data — and that is the point. A
|
|
132
|
+
standing account reaches what a fresh signup never could: a populated list, filters with
|
|
133
|
+
something to filter, run history, a dashboard with numbers in it. **When a flow needs
|
|
134
|
+
pre-existing data, that is the signal to use the test account** rather than building the
|
|
135
|
+
data inside the test.
|
|
136
|
+
|
|
137
|
+
The trade is drift — run 200 has 200 of everything run 1 created. So:
|
|
138
|
+
|
|
139
|
+
- **Assert relatively, never absolutely.** "The row I just created is present"
|
|
140
|
+
(\`expect.persisted\` on a \`{{unique}}\`-named row), not "there are 3 rows". A count
|
|
141
|
+
assertion is true in week one and false in week four with nobody having touched it.
|
|
142
|
+
- **Clean up what you create** — put the delete in the plan's \`after\` section, which runs
|
|
143
|
+
on pass AND on fail, unlike a trailing step inside \`steps\`.
|
|
144
|
+
|
|
145
|
+
### If there is no account yet
|
|
146
|
+
|
|
147
|
+
An empty \`accounts list\` means nothing is set up. In order of preference:
|
|
148
|
+
|
|
149
|
+
1. **The user has a dedicated test account** → bank it:
|
|
150
|
+
\`beryl accounts create --email <email> --password <password>\`. A DEDICATED test
|
|
151
|
+
account only — never a real user's.
|
|
152
|
+
2. **No credentials, but the app has a signup form** → let Beryl make one.
|
|
153
|
+
\`beryl accounts create --type beryl\`, then prove it with
|
|
154
|
+
\`beryl accounts provision <id> --file <plan.json>\`. The plan is any ActionPlan that
|
|
155
|
+
ends LOGGED IN — a signup filling \`{{mailbox_address}}\` and \`{{login_password}}\`
|
|
156
|
+
when the account is new, or a **sign-in** when it already exists (you made it by hand,
|
|
157
|
+
or a previous provision succeeded and the record was lost). Both prove the same thing.
|
|
158
|
+
If the app has no password sign-in, pass \`--login-method otp\` (or \`magic_link\`)
|
|
159
|
+
and let the plan \`await_email\` its way through — later sign-ins read the same mailbox.
|
|
160
|
+
3. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
|
|
161
|
+
|
|
162
|
+
A project with no accounts still falls back to the \`LOGIN_EMAIL\` variable +
|
|
163
|
+
\`LOGIN_PASSWORD\` secret and the handles resolve the same way, so existing tests are
|
|
164
|
+
unaffected — but new work should create an account.
|
|
165
|
+
|
|
166
|
+
### Session mode: sign in once per run, not once per test
|
|
167
|
+
|
|
168
|
+
Every \`requires_auth\` plan declares its \`auth_mode\` — there is no default. With
|
|
169
|
+
\`"inline"\` the test carries its own sign-in steps and re-types them on every run; that
|
|
170
|
+
is fine and always available.
|
|
171
|
+
|
|
172
|
+
\`auth_mode: "session"\` moves the sign-in out of the test. The account signs in ONCE at
|
|
173
|
+
the start of the run, the resulting browser session is proved live, and every
|
|
174
|
+
\`requires_auth\` test in that run rides it — session-mode tests carry no sign-in steps
|
|
175
|
+
at all, and inline ones simply start already signed in. The test then starts where the
|
|
176
|
+
flow it actually tests begins — no login preamble in the plan, in the replay, or in the
|
|
177
|
+
failure evidence. Nothing has to be turned on for it: every run signs in its accounts and
|
|
178
|
+
hands out the sessions. What a session-mode test DOES need is a stored login plan on its
|
|
179
|
+
account — with none, there is nothing to sign in with, and the test fails at setup with
|
|
180
|
+
\`SESSION_NO_LOGIN_PLAN\` rather than falling back, because it has no sign-in steps of its
|
|
181
|
+
own to fall back to.
|
|
182
|
+
|
|
183
|
+
Set it up once per account:
|
|
184
|
+
|
|
185
|
+
\`\`\`
|
|
186
|
+
beryl accounts set-login <id> --file signin.json --probe probe.json
|
|
187
|
+
beryl accounts check <id> # signs in NOW and proves it — do not skip this
|
|
188
|
+
\`\`\`
|
|
189
|
+
|
|
190
|
+
- **\`--file\`** is the SIGN-IN plan (not the signup): opens the login page, fills
|
|
191
|
+
\`{{login_email}}\` / \`{{login_password}}\`, \`await_email\`s a code if the account
|
|
192
|
+
needs one, and asserts the logged-in shell.
|
|
193
|
+
- **\`--probe\`** is two steps: goto a gated page, then a **positive** assertion that only
|
|
194
|
+
holds when signed in — \`expect.visible\` on the account menu or a "Sign out" control.
|
|
195
|
+
It is required. \`hidden\`, \`count 0\` and a URL match on a redirect ALL pass against a
|
|
196
|
+
logged-out page, so without a positive signal a dead session runs every test logged-out
|
|
197
|
+
and the run still reports green.
|
|
198
|
+
|
|
199
|
+
Then write the tests with \`requires_auth: true\` and \`auth_mode: "session"\`, and NO
|
|
200
|
+
sign-in steps.
|
|
201
|
+
|
|
202
|
+
**Creates share the sign-in.** \`tests create\` replays a session-mode plan against the
|
|
203
|
+
account's established session — reused from the last proven sign-in when it is still
|
|
204
|
+
live — so a batch of creates costs at most one sign-in, same as a run. Still create one
|
|
205
|
+
at a time: two racing creates would race the same mailbox for mail.
|
|
206
|
+
|
|
207
|
+
Reading before writing matters here: the login plan self-heals, so
|
|
208
|
+
\`beryl accounts get-login <id>\` first and pass its \`login_plan_hash\` back as
|
|
209
|
+
\`--base-hash\` — a blind overwrite would clobber a repair you never saw.
|
|
210
|
+
|
|
211
|
+
An expired session is never your problem to notice. Beryl proves the stored session at
|
|
212
|
+
the start of every run and, if it no longer works, treats that as a cache miss and signs
|
|
213
|
+
in again from the login plan. Nothing asks a human to reconnect — that message only
|
|
214
|
+
appears for a project with no stored login plan to refresh from.
|
|
215
|
+
|
|
216
|
+
What to expect when it does not work: \`accounts check\` returning
|
|
217
|
+
\`SESSION_LOGIN_FAILED\` means the sign-in plan itself is wrong — fix it.
|
|
218
|
+
\`SESSION_PROOF_FAILED\` means signing in worked but the session could not be carried into
|
|
219
|
+
a fresh browser, because this app keeps its credential somewhere unextractable. That is not
|
|
220
|
+
your bug, but it IS your move: the account is marked unsupported, and its session-mode
|
|
221
|
+
tests fail at setup with \`SESSION_UNSUPPORTED\` on every run until you re-author them
|
|
222
|
+
with \`auth_mode: "inline"\` and their own sign-in steps. Inline tests are unaffected.
|
|
223
|
+
|
|
224
|
+
\`beryl runs local\` works on session-mode tests too — same shape as the cloud: one
|
|
225
|
+
sign-in per invocation, shared by every session-mode test.
|
|
226
|
+
|
|
227
|
+
### Two identities in one test
|
|
228
|
+
|
|
229
|
+
Still not bankable as one test: invite-a-teammate-and-accept-as-them needs two identities
|
|
230
|
+
mid-flow. Bank the half the app shows to account A ("the invitation is listed as pending",
|
|
231
|
+
"the share link is issued") — a real, strong outcome. While AUTHORING you can drive the
|
|
232
|
+
full handshake live: create a second account (\`accounts create --label member\`), or add a
|
|
233
|
+
second mailbox (\`mailbox create --label invitee\`) and read it (\`mailbox read\`).
|
|
234
|
+
|
|
235
|
+
SSO-only sites (no email+password form at all) remain webapp territory.
|
|
236
|
+
## 2. Author locally over the Playwright MCP
|
|
61
237
|
|
|
62
238
|
1. **Drive the flow in a real browser first.** Use the Playwright MCP to open the app and
|
|
63
239
|
walk the flow by hand — log in, fill the form, submit, whatever the flow is. You act on
|
|
@@ -67,6 +243,11 @@ assertion**, a strong **natural-language intent**, and the **local run-fix loop*
|
|
|
67
243
|
\`{action, selector, url, value, ...}\`. Two structural rules the plan must satisfy:
|
|
68
244
|
- the **first executed step is a \`goto\`** (the flow has to start by navigating somewhere), and
|
|
69
245
|
- **at least one step is an \`expect\`** (a test that asserts nothing is not a test).
|
|
246
|
+
**A \`goto\` at your own app is a PATH, never a full URL** — \`/pricing\`, not
|
|
247
|
+
\`https://app.example.com/pricing\`. The origin comes from the environment's root URL, so
|
|
248
|
+
one plan runs against prod, staging and a preview. Bake the origin in and \`--env\` is
|
|
249
|
+
silently ignored: the test keeps hitting whatever host you typed. Absolute URLs stay
|
|
250
|
+
legal for OTHER origins (an OAuth handoff, a magic link on another domain).
|
|
70
251
|
Optional \`before\` / \`after\` arrays hold setup and teardown; \`after\` runs even when a
|
|
71
252
|
main step fails, so a create/update/delete flow can clean up the record it made.
|
|
72
253
|
|
|
@@ -126,7 +307,7 @@ observable proof the flow worked. Get this right and everything else follows.
|
|
|
126
307
|
\`/thank-you\`) and no distinctive destination content is available.
|
|
127
308
|
- The usual outcome kinds: \`visible\` (the success element showed up), \`have_text\` (an
|
|
128
309
|
element's text matches), \`have_url\` (the URL contains a value), \`gone\` (an element
|
|
129
|
-
disappeared — e.g. a spinner, or the item you just deleted). §
|
|
310
|
+
disappeared — e.g. a spinner, or the item you just deleted). §2 has the full
|
|
130
311
|
\`expect_kind\` list and the step shape.
|
|
131
312
|
- **\`have_text\` is an EXACT full-text match on the selector's element** — asserting
|
|
132
313
|
\`have_text: "Documentation"\` on \`body\` fails, because \`body\`'s text includes all the
|
|
@@ -139,38 +320,45 @@ observable proof the flow worked. Get this right and everything else follows.
|
|
|
139
320
|
genuinely broken in the app, that's a finding to report — not something to paper over
|
|
140
321
|
with a weaker assertion.
|
|
141
322
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
323
|
+
### Traps when authoring against a real app
|
|
324
|
+
|
|
325
|
+
Every one of these has produced a wrong plan or a red \`tests create\`. Check them.
|
|
326
|
+
|
|
327
|
+
1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
|
|
328
|
+
profile, so a session can survive from earlier work. Author while signed in and you
|
|
329
|
+
never see the gate — you will mark gated pages as public. **Log out first**, then
|
|
330
|
+
confirm the page you think is gated really does redirect to the login.
|
|
331
|
+
2. **The sign-UP flow is not the sign-IN flow.** A brand-new address often gets an extra
|
|
332
|
+
"create your account" step that a returning address skips entirely. An account's stored
|
|
333
|
+
login plan must be the **returning** path — that is what runs on every future run. Drive
|
|
334
|
+
it twice: once to create the account, once to see signing in again.
|
|
335
|
+
3. **Read the DOM, not just the accessibility tree, before picking a selector.** Two
|
|
336
|
+
buttons can share a visible label ("Continue" and "Continue with Google"), and Beryl
|
|
337
|
+
relaxes \`text=X\` to a case-insensitive SUBSTRING — so \`text=Continue\` is ambiguous.
|
|
338
|
+
Find something unique (\`button[type=submit]\`, \`input[name=email]\`,
|
|
339
|
+
\`input[autocomplete=one-time-code]\`) and use that.
|
|
340
|
+
4. **A single-page app can redirect after the first \`goto\`.** If \`/dashboard\` client-side
|
|
341
|
+
redirects to \`/dashboard/<id>\`, a click fired straight after the goto lands on the
|
|
342
|
+
pre-redirect render and its effect is discarded when the app re-renders — a dialog that
|
|
343
|
+
opens and instantly vanishes, for instance. Put a \`wait_for\` on something that exists
|
|
344
|
+
only AFTER the redirect, then act.
|
|
345
|
+
5. **Never bake an id into a URL.** \`goto /projects/8ab46d63-.../settings\` breaks for any
|
|
346
|
+
other account. Navigate to the stable entry point and click through
|
|
347
|
+
(\`a[href$='/settings']\`), so the plan is about the app, not about your row.
|
|
348
|
+
6. **Assert durable content, not the empty state.** "No tests yet" is true today and false
|
|
349
|
+
the moment anything exists. Prefer what is structural to the page — a section heading, a
|
|
350
|
+
permanent explainer, a control that is always there.
|
|
163
351
|
|
|
164
352
|
## 3. Writing the natural-language intent
|
|
165
353
|
|
|
166
354
|
Pass the intent as \`--description\` on \`beryl tests create\` (or \`tests set-plan\` when you
|
|
167
|
-
re-author). 1–3 sentences. This is the immutable anchor from §
|
|
355
|
+
re-author). 1–3 sentences. This is the immutable anchor from §6 — write it well.
|
|
168
356
|
|
|
169
357
|
- **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
|
|
170
358
|
clicks submit" — that's the trajectory, which Beryl already has and which will change.
|
|
171
359
|
Instead: *what does a green run prove is true about the app?*
|
|
172
360
|
- **Name the one observable outcome** that is true only if the flow worked — the same
|
|
173
|
-
success signal you asserted in §
|
|
361
|
+
success signal you asserted in §2, in words.
|
|
174
362
|
- **Never describe global chrome.** The intent is about the flow's destination and
|
|
175
363
|
outcome, not "the header is present".
|
|
176
364
|
|
|
@@ -188,7 +376,7 @@ test's rendered spec and runs it with your local \`@playwright/test\` — no clo
|
|
|
188
376
|
for a scheduled run.
|
|
189
377
|
|
|
190
378
|
\`\`\`
|
|
191
|
-
npm i -D @playwright/test && npx playwright install # once
|
|
379
|
+
npm i -D @playwright/test && npx playwright install chromium # once
|
|
192
380
|
beryl runs local <test-id> --no-sync --url-override http://localhost:3000 --dir ./beryl-local
|
|
193
381
|
beryl runs local # the whole suite, results recorded in Beryl
|
|
194
382
|
\`\`\`
|
|
@@ -197,23 +385,22 @@ beryl runs local # the whole suite, results recorded in Beryl
|
|
|
197
385
|
- **Results sync to Beryl by default** — the finished run is imported as a first-class
|
|
198
386
|
run (history, replay, report; trigger source \`local\`). While ITERATING on a draft,
|
|
199
387
|
pass \`--no-sync\` so every fix-loop attempt doesn't land in the project's run history.
|
|
200
|
-
- \`--url-override\`
|
|
388
|
+
- \`--url-override\` swaps the root URL for THIS run — for a throwaway host (a dev server, a
|
|
389
|
+
per-PR preview). A standing environment is \`--env <id>\` instead, not an override.
|
|
201
390
|
- \`--dir\` keeps the **spec, artifacts, and a JSON \`report.json\`** on disk so you (or your
|
|
202
391
|
coding agent) can read exactly what happened and iterate: read the report, see which step
|
|
203
392
|
or assertion failed and why, fix the plan, \`beryl tests set-plan\`, run again.
|
|
204
393
|
- It exits **0** if every test passed, **1** on a failure — so it drops straight into a
|
|
205
394
|
run-fix-run loop.
|
|
206
|
-
- \`await_email\` steps work locally: the CLI
|
|
207
|
-
the
|
|
208
|
-
- **
|
|
209
|
-
\`{{login_password}}\` runs fine: the email is baked into the fetched
|
|
210
|
-
password is revealed once over the logged secret-reveal route, then
|
|
211
|
-
any uploaded error text or DOM snapshot. If the
|
|
212
|
-
|
|
213
|
-
-
|
|
214
|
-
|
|
215
|
-
never handed to your disk); \`runs local\` skips it with a note. Run those with
|
|
216
|
-
\`beryl runs trigger\`.
|
|
395
|
+
- \`await_email\` steps work locally: the CLI answers them over the API against the same
|
|
396
|
+
mailbox the cloud runner would use, exactly as it would.
|
|
397
|
+
- **Authenticated tests work locally.** A plan that signs itself in by filling
|
|
398
|
+
\`{{login_email}}\` / \`{{login_password}}\` runs fine: the email is baked into the fetched
|
|
399
|
+
spec and the password is revealed once over the logged secret-reveal route, then
|
|
400
|
+
scrubbed from any uploaded error text or DOM snapshot. If the test account it names has
|
|
401
|
+
no password stored, the test is skipped with the exact fix-it command.
|
|
402
|
+
- A test that depends on a session Beryl holds server-side, rather than signing itself in,
|
|
403
|
+
is skipped locally with a note — run those with \`beryl runs trigger\`.
|
|
217
404
|
|
|
218
405
|
Once the test passes locally against a real outcome, it's ready to bank and let Beryl run
|
|
219
406
|
and heal it.
|
|
@@ -221,17 +408,30 @@ and heal it.
|
|
|
221
408
|
## 5. Testing an OTP / signup flow (\`await_email\`)
|
|
222
409
|
|
|
223
410
|
A flow that emails the user — a signup verification code, a magic sign-in link, a receipt
|
|
224
|
-
— is testable with the \`await_email\` action.
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
411
|
+
— is testable with the \`await_email\` action. No setup, no environment configuration, no
|
|
412
|
+
flag to turn on.
|
|
413
|
+
|
|
414
|
+
**The project has one permanent mailbox and all of its mail arrives there.** Two handles
|
|
415
|
+
put an address on the page, and the one you cite decides which *identity* the test acts as:
|
|
416
|
+
|
|
417
|
+
| Handle | Renders as | Use it for |
|
|
418
|
+
|---|---|---|
|
|
419
|
+
| \`{{mailbox_address}}\` | the mailbox's own address, the same every run | signing in as the project's standing test account (§1) |
|
|
420
|
+
| \`{{inbox_address}}\` | a \`+tag\` alias of it, fresh every run | tests whose subject IS getting a NEW identity — a signup, an invited teammate |
|
|
421
|
+
|
|
422
|
+
An alias is a real address the site has never issued, so a signup is repeatable run after
|
|
423
|
+
run; the mail still lands in the same mailbox, and Beryl reads only the alias's own mail.
|
|
424
|
+
Nothing expires and there is no second inbox to manage.
|
|
425
|
+
|
|
426
|
+
Default to \`{{mailbox_address}}\`. Reach for \`{{inbox_address}}\` only when an existing
|
|
427
|
+
account would be rejected — a signup form, or an invite you must accept as a second person.
|
|
228
428
|
|
|
229
429
|
The wiring is a three-part chain:
|
|
230
430
|
|
|
231
|
-
1. **Type the
|
|
232
|
-
|
|
233
|
-
\`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other
|
|
234
|
-
values like a username).
|
|
431
|
+
1. **Type the address into the app** — a \`fill\` with \`value: "{{inbox_address}}"\` (or
|
|
432
|
+
\`{{mailbox_address}}\`). An alias is fresh every run, so a signup flow is repeatable by
|
|
433
|
+
construction (no \`{{unique}}\` needed for the email itself; use \`{{unique}}\` for other
|
|
434
|
+
must-not-collide values like a username).
|
|
235
435
|
2. **Await the mail and bank the extracted value** — an \`await_email\` step with:
|
|
236
436
|
- \`extract\` (required): \`code\` (an OTP), \`link\` (the sign-in/verify URL), or
|
|
237
437
|
\`pattern\` (your own regex in \`extract_pattern\`, exactly one capture group).
|
|
@@ -258,48 +458,33 @@ For a magic-link flow, replace the code steps with
|
|
|
258
458
|
Two caveats:
|
|
259
459
|
|
|
260
460
|
- \`beryl tests create\` verifies an \`await_email\` plan like any other — its local
|
|
261
|
-
replay
|
|
262
|
-
really is received and extracted before the test is accepted. (Note the replay
|
|
263
|
-
up / sends mail for real; pass \`--no-verify\` only if that side effect is
|
|
264
|
-
\`beryl runs local\` serves \`await_email\` the same way, so the whole local
|
|
265
|
-
OTP/signup flows end to end.
|
|
266
|
-
- The outcome assertion discipline from §
|
|
461
|
+
replay receives at the project mailbox and answers each step over the API, so the app's
|
|
462
|
+
mail really is received and extracted before the test is accepted. (Note the replay
|
|
463
|
+
signs up / sends mail for real; pass \`--no-verify\` only if that side effect is
|
|
464
|
+
unwanted.) \`beryl runs local\` serves \`await_email\` the same way, so the whole local
|
|
465
|
+
loop covers OTP/signup flows end to end.
|
|
466
|
+
- The outcome assertion discipline from §2 still applies: the green signal is the
|
|
267
467
|
post-verification state (the welcome screen, the dashboard), not "an email arrived".
|
|
268
468
|
|
|
269
|
-
## 6.
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
- **
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
tests. Do not ask the user for credentials — mint is the default path.
|
|
291
|
-
- **Handed credentials in chat? Bank them first** (\`config vars set LOGIN_EMAIL\`,
|
|
292
|
-
\`config secrets set LOGIN_PASSWORD\`), then author with the handles as above.
|
|
293
|
-
A DEDICATED test account only — never a real user's.
|
|
294
|
-
- **Never hard-code an email address or password in a plan.** Minted inboxes expire
|
|
295
|
-
and pasted values rot on rotation — the handles are the only durable references.
|
|
296
|
-
- **One identity per banked test.** A flow involving a second account — invite a
|
|
297
|
-
teammate and accept as them, share and open as the viewer — is not bankable as one
|
|
298
|
-
test yet: a run has one inbox, and \`await_email\` reads only that inbox. Bank the
|
|
299
|
-
half the app shows to account A ("the invitation is listed as pending", "the share
|
|
300
|
-
link is issued") — a real, strong outcome. While AUTHORING you can still verify the
|
|
301
|
-
full handshake live: mint a second inbox (\`inbox create\`), read its mail
|
|
302
|
-
(\`inbox read\`), drive both sides in the browser — then bank the single-identity
|
|
303
|
-
halves.
|
|
304
|
-
- SSO-only sites (no email+password form at all) remain webapp territory.
|
|
469
|
+
## 6. Why this shape: durable and healable
|
|
470
|
+
|
|
471
|
+
Beryl's cloud runs your test on a schedule. When the app's markup drifts and a selector
|
|
472
|
+
stops matching, a heal-vs-fail agent decides whether to **heal** the test (silently
|
|
473
|
+
re-derive the selector/trajectory and keep it green) or **fail** it (surface a real
|
|
474
|
+
regression). It decides that against your test's **intent**:
|
|
475
|
+
|
|
476
|
+
- **The natural-language intent is the immutable anchor. Beryl never rewrites it.** It's
|
|
477
|
+
the description of what the test proves — the load-bearing statement the heal agent
|
|
478
|
+
judges every future run against.
|
|
479
|
+
- **Selectors and the trajectory are the healable "how".** A button moved, a class name
|
|
480
|
+
changed, a step needs an extra click — those are mechanics Beryl can re-derive on its
|
|
481
|
+
own, because your intent tells it what the flow was *for*.
|
|
482
|
+
- **A failed outcome assertion is a real regression Beryl will NOT silently heal green.**
|
|
483
|
+
If the success signal from §2 stops holding — the confirmation never appears, the page
|
|
484
|
+
never renders — that's the app breaking, and the test fails loudly. That is the point.
|
|
485
|
+
|
|
486
|
+
So a test is *healable* exactly when you gave it **a strong intent + a real outcome
|
|
487
|
+
assertion**. A test with a vague intent and a chrome-only assertion is brittle: Beryl
|
|
488
|
+
can't tell a real regression from cosmetic drift, so it either heals over real breakage or
|
|
489
|
+
fails on noise.
|
|
305
490
|
`;
|