@beryl-so/cli 0.64.0 → 0.73.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -8
- package/dist/adapters/mcp.js +6 -4
- package/dist/beryl-test-skill.js +84 -91
- package/dist/commands/accounts.js +96 -19
- package/dist/commands/config-vars.js +1 -1
- package/dist/commands/environments.js +5 -0
- package/dist/commands/health.js +3 -1
- package/dist/commands/init.js +18 -18
- package/dist/commands/jobs.js +125 -0
- package/dist/commands/projects.js +74 -21
- package/dist/commands/runs.js +3 -2
- package/dist/commands/tests.js +60 -317
- package/dist/commands/tracker.js +4 -1
- package/dist/commands/watch.js +36 -0
- package/dist/context.js +1 -2
- package/dist/lint.js +0 -6
- package/dist/local-exec.js +4 -4
- package/dist/registry/index.js +2 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# beryl CLI
|
|
4
4
|
|
|
5
|
-
Beryl on the command line: create projects,
|
|
5
|
+
Beryl on the command line: create projects, author tests,
|
|
6
6
|
trigger runs in CI, manage everything the web app can, plus an MCP server (`beryl mcp`)
|
|
7
7
|
that exposes every command to coding agents.
|
|
8
8
|
|
|
@@ -78,11 +78,11 @@ Set up Beryl in this repo: sign in and wire up your coding agent
|
|
|
78
78
|
|
|
79
79
|
### guide
|
|
80
80
|
|
|
81
|
-
Print the Beryl test
|
|
81
|
+
Print the Beryl test guide
|
|
82
82
|
|
|
83
83
|
| Command | Summary | MCP tool |
|
|
84
84
|
| --- | --- | --- |
|
|
85
|
-
| `beryl guide` | Print the Beryl test
|
|
85
|
+
| `beryl guide` | Print the Beryl test guide | `guide` |
|
|
86
86
|
|
|
87
87
|
### login
|
|
88
88
|
|
|
@@ -166,7 +166,7 @@ Send, list, and revoke workspace invitations, and accept ones sent to you.
|
|
|
166
166
|
|
|
167
167
|
### projects
|
|
168
168
|
|
|
169
|
-
Create and manage projects: a site
|
|
169
|
+
Create and manage projects: a site you author tests for and run.
|
|
170
170
|
|
|
171
171
|
`beryl projects` with no subcommand runs `projects list`.
|
|
172
172
|
|
|
@@ -174,12 +174,14 @@ Create and manage projects: a site Beryl explores, authors tests for, and runs.
|
|
|
174
174
|
| --- | --- | --- |
|
|
175
175
|
| `beryl projects list` | List projects in the workspace | `projects_list` |
|
|
176
176
|
| `beryl projects get` | Show one project, including its current exploration state | `projects_get` |
|
|
177
|
-
| `beryl projects create [url]` | Create a project
|
|
177
|
+
| `beryl projects create [url]` | Create a project for a URL, or an empty one with just --name | `projects_create` |
|
|
178
178
|
| `beryl projects rename <name>` | Rename a project | `projects_rename` |
|
|
179
179
|
| `beryl projects quarantine [state]` | Turn auto quarantine on or off and tune when it mutes and releases a test | `projects_quarantine` |
|
|
180
|
+
| `beryl projects surfaces <surfaces...>` | Set which surfaces are enabled for this project | `projects_surfaces` |
|
|
181
|
+
| `beryl projects findings-gate <mode>` | Turn the findings gate on each run on or off | `projects_findings_gate` |
|
|
180
182
|
| `beryl projects delete` | Delete a project and all its tests and runs | `projects_delete` |
|
|
181
183
|
| `beryl projects re-explore` | Send the agent back in to run/heal existing tests and discover new flows | none |
|
|
182
|
-
| `beryl projects report` | Aggregate quality report across recent runs (pass rates,
|
|
184
|
+
| `beryl projects report` | Aggregate quality report across recent runs (pass rates, per-test history, trend) | `projects_report` |
|
|
183
185
|
| `beryl projects reusable-auth <url>` | Check whether a saved login can be reused for a URL before creating a project | `projects_reusable_auth` |
|
|
184
186
|
|
|
185
187
|
### envs
|
|
@@ -217,9 +219,10 @@ Author, inspect, version, and heal a project's tests: the checks Beryl runs on e
|
|
|
217
219
|
| `beryl tests list` | List the project's tests with their latest result | `tests_list` |
|
|
218
220
|
| `beryl tests get <test-id>` | Show one test | `tests_get` |
|
|
219
221
|
| `beryl tests plan <test-id>` | Print a test's current step plan (JSON) | `tests_plan` |
|
|
220
|
-
| `beryl tests create` |
|
|
222
|
+
| `beryl tests create` | Retired: tests are now created by the cloud test-creation job (`jobs create`) | `tests_create` |
|
|
221
223
|
| `beryl tests set-plan <test-id>` | Replace a test's step plan from a JSON file (creates a new version) | `tests_set_plan` |
|
|
222
224
|
| `beryl tests rename <test-id> <title>` | Rename a test | `tests_rename` |
|
|
225
|
+
| `beryl tests set-intent <test-id> <intent>` | Edit what a test proves, the target heal judges against | `tests_set_intent` |
|
|
223
226
|
| `beryl tests set-groups <test-id>` | Replace the groups a test belongs to | `tests_set_groups` |
|
|
224
227
|
| `beryl tests set-access <test-id> <access>` | Mark a test read or write | `tests_set_access` |
|
|
225
228
|
| `beryl tests add-groups <test-ids...>` | Add groups to one or more tests, keeping the groups they already have | `tests_add_groups` |
|
|
@@ -286,6 +289,18 @@ Site Health, or how your site reads to search engines and visitors: content, spe
|
|
|
286
289
|
| `beryl health tracker teams` | List the teams of the connected issue tracker | `health_tracker_teams` |
|
|
287
290
|
| `beryl health tracker projects` | List a tracker team's projects | `health_tracker_projects` |
|
|
288
291
|
|
|
292
|
+
### jobs
|
|
293
|
+
|
|
294
|
+
Test-creation jobs: the cloud agent reads a document and authors tests from it.
|
|
295
|
+
|
|
296
|
+
| Command | Summary | MCP tool |
|
|
297
|
+
| --- | --- | --- |
|
|
298
|
+
| `beryl jobs create [file]` | Create tests from a document (PRD, release notes) on Beryl's cloud | `jobs_create` |
|
|
299
|
+
| `beryl jobs list` | List the project's test-creation jobs | `jobs_list` |
|
|
300
|
+
| `beryl jobs get <job-id>` | Show a job's status and the tests it created | `jobs_get` |
|
|
301
|
+
| `beryl jobs clone <job-id>` | Run a job again: queue a new job with the same document and environment | `jobs_clone` |
|
|
302
|
+
| `beryl jobs delete <job-id>` | Delete a job; the flows and tests it created stay | `jobs_delete` |
|
|
303
|
+
|
|
289
304
|
### explorations
|
|
290
305
|
|
|
291
306
|
Inspect the agent's exploration runs: how it crawled a site and authored its tests.
|
|
@@ -308,7 +323,7 @@ Manage the variables, secrets, and files the agent can use while exploring and r
|
|
|
308
323
|
| `beryl config vars set <key> <value>` | Create or update a config variable | `config_vars_set` |
|
|
309
324
|
| `beryl config vars get <key>` | Show one config variable | `config_vars_get` |
|
|
310
325
|
| `beryl config vars delete <key>` | Delete a config variable | `config_vars_delete` |
|
|
311
|
-
| `beryl config secrets list` | List the project's secrets (values are never returned) | `config_secrets_list` |
|
|
326
|
+
| `beryl config secrets list` | List the project's secrets (values are never returned); a test account's password is managed on the account, not here | `config_secrets_list` |
|
|
312
327
|
| `beryl config secrets set <key> <value>` | Create a secret (write-only; re-setting a key replaces it) | `config_secrets_set` |
|
|
313
328
|
| `beryl config secrets get <key>` | Show one secret's metadata, or reveal its value with --reveal | `config_secrets_get` |
|
|
314
329
|
| `beryl config secrets delete <key>` | Delete a secret | `config_secrets_delete` |
|
|
@@ -354,7 +369,9 @@ Durable identities on the site under test: what an authenticated test signs in a
|
|
|
354
369
|
| Command | Summary | MCP tool |
|
|
355
370
|
| --- | --- | --- |
|
|
356
371
|
| `beryl accounts list` | List the test accounts an environment's tests sign in as | `accounts_list` |
|
|
372
|
+
| `beryl accounts get <account-id>` | Show one test account: origin, lifetime, sign-in state | `accounts_get` |
|
|
357
373
|
| `beryl accounts create` | Add a test account: the customer's own, or one Beryl signs up | `accounts_create` |
|
|
374
|
+
| `beryl accounts capture` | Store a signed-in session as a captured test account | `accounts_capture` |
|
|
358
375
|
| `beryl accounts signup <account-id>` | Store a test account's signup plan and sign it up on its environment | `accounts_signup` |
|
|
359
376
|
| `beryl accounts set-login <account-id>` | Store the sign-in plan a run replays once, plus the probe that proves it | `accounts_set_login` |
|
|
360
377
|
| `beryl accounts get-login <account-id>` | Read the stored sign-in plan, its probe, and the hash a safe write must cite | `accounts_get_login` |
|
package/dist/adapters/mcp.js
CHANGED
|
@@ -278,13 +278,15 @@ export function mcpInstructions() {
|
|
|
278
278
|
"version, and whether this build is behind npm's latest). " +
|
|
279
279
|
"Beryl authors, runs, and heals end-to-end tests for any web app: tests are JSON " +
|
|
280
280
|
"action plans replayed in real cloud browsers, signing in as a durable test " +
|
|
281
|
-
"account whose mail arrives at the project's own mailbox
|
|
281
|
+
"account whose mail arrives at the project's own mailbox, so signup/OTP/" +
|
|
282
282
|
"magic-link flows are self-contained, with no human login needed. " +
|
|
283
|
-
"
|
|
284
|
-
"
|
|
283
|
+
"New tests are authored only by Beryl's cloud agent: give it a document describing " +
|
|
284
|
+
"the flows with `jobs_create`; there is no local authoring. " +
|
|
285
|
+
"The guide (plan shape, outcome assertions, accounts, email/OTP wiring, the local " +
|
|
286
|
+
"run-fix loop) ships as both the beryl-test skill and the `guide` tool, same " +
|
|
285
287
|
`content. If a beryl-test skill stating v${version} is already loaded, do not ` +
|
|
286
288
|
"call `guide`; if no beryl-test skill is available or it states another version, " +
|
|
287
|
-
"call `guide` before
|
|
289
|
+
"call `guide` before editing a test plan or a sign-in plan. " +
|
|
288
290
|
"An account can hold several workspaces and projects: call `projects_list` first " +
|
|
289
291
|
"and pass `project` (an id is enough) on every project-scoped tool, or the call fails.");
|
|
290
292
|
}
|
package/dist/beryl-test-skill.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// The `beryl-test`
|
|
1
|
+
// The `beryl-test` skill, installed by `beryl init` into
|
|
2
2
|
// `.agents/skills/beryl-test/SKILL.md` (vendor-neutral, editor-agnostic) — and, when
|
|
3
3
|
// claude-code is a selected editor, ALSO into the `.claude/skills/beryl-test/SKILL.md`
|
|
4
4
|
// that Claude Code actually indexes (see `writeSkills` in `commands/init.ts`). Kept as an
|
|
@@ -17,23 +17,24 @@ export const BERYL_TEST_SKILL_DIR = "beryl-test";
|
|
|
17
17
|
// nor `.claude/skills/`, so `init` writes them their own pointer instead. Thin on purpose:
|
|
18
18
|
// it sends the agent to the `guide` MCP tool rather than inlining the 500-line skill, and
|
|
19
19
|
// `guide` ships with the CLI so the real guidance never drifts from the installed version.
|
|
20
|
-
const BERYL_GUIDE_POINTER = `When
|
|
21
|
-
MCP tool first. It returns the full
|
|
22
|
-
assertions, test accounts and the project mailbox, and the
|
|
20
|
+
const BERYL_GUIDE_POINTER = `When creating, running, editing, or debugging Beryl tests in this repo, call the
|
|
21
|
+
\`guide\` MCP tool first. It returns the full guide: where new tests come from, the
|
|
22
|
+
ActionPlan shape, outcome assertions, test accounts and the project mailbox, and the
|
|
23
|
+
local run-fix loop.
|
|
23
24
|
|
|
24
|
-
Then use the Beryl MCP tools: \`projects_list\` to find the project,
|
|
25
|
-
|
|
25
|
+
Then use the Beryl MCP tools: \`projects_list\` to find the project, \`jobs_create\` to
|
|
26
|
+
have Beryl's cloud agent author tests from a document, and \`runs_local\` to run them here.
|
|
26
27
|
|
|
27
28
|
The \`guide\` tool ships with the CLI and always matches the installed version, so treat
|
|
28
29
|
it as the source of truth over anything cached here.`;
|
|
29
30
|
// Cursor reads its own `.cursor/rules/*.mdc`, auto-loaded when alwaysApply is set.
|
|
30
31
|
export const BERYL_CURSOR_RULE_FILENAME = "beryl.mdc";
|
|
31
32
|
export const BERYL_CURSOR_RULE = `---
|
|
32
|
-
description:
|
|
33
|
+
description: Creating, running and editing Beryl end-to-end tests
|
|
33
34
|
alwaysApply: true
|
|
34
35
|
---
|
|
35
36
|
|
|
36
|
-
# Beryl
|
|
37
|
+
# Beryl tests
|
|
37
38
|
|
|
38
39
|
${BERYL_GUIDE_POINTER}
|
|
39
40
|
`;
|
|
@@ -43,7 +44,7 @@ ${BERYL_GUIDE_POINTER}
|
|
|
43
44
|
export const BERYL_COPILOT_MARK_BEGIN = "<!-- BEGIN beryl (managed by beryl init) -->";
|
|
44
45
|
export const BERYL_COPILOT_MARK_END = "<!-- END beryl -->";
|
|
45
46
|
export const BERYL_COPILOT_SECTION = `${BERYL_COPILOT_MARK_BEGIN}
|
|
46
|
-
## Beryl
|
|
47
|
+
## Beryl tests
|
|
47
48
|
|
|
48
49
|
${BERYL_GUIDE_POINTER}
|
|
49
50
|
${BERYL_COPILOT_MARK_END}`;
|
|
@@ -89,78 +90,78 @@ export const BERYL_TEST_SKILL_OTP_EXAMPLE_PLAN = {
|
|
|
89
90
|
};
|
|
90
91
|
export const BERYL_TEST_SKILL = `---
|
|
91
92
|
name: beryl-test
|
|
92
|
-
description:
|
|
93
|
+
description: Create, run, and fix durable, healable end-to-end tests for a web app with Beryl. Use when asking Beryl's cloud agent to create tests from a document (\`jobs_create\`), running tests locally with \`runs_local\`, setting up test accounts, or fixing a banked test's plan with your own coding agent.
|
|
93
94
|
---
|
|
94
95
|
|
|
95
|
-
#
|
|
96
|
+
# Beryl tests
|
|
96
97
|
|
|
97
|
-
beryl-test v${cliVersion()}
|
|
98
|
+
beryl-test v${cliVersion()}, installed by \`beryl init\`; the \`guide\` tool always returns the current version of this guide.
|
|
98
99
|
|
|
99
100
|
A Beryl test drives a real browser through a flow and asserts the flow reached its
|
|
100
|
-
meaningful outcome. \`beryl init\` wired two MCP servers: **beryl** (
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
101
|
+
meaningful outcome. \`beryl init\` wired two MCP servers: **beryl** (\`jobs_create\`,
|
|
102
|
+
\`accounts_list\`, \`runs_local\`, and the rest) and **playwright** (drive a real
|
|
103
|
+
browser). You do everything below by calling those tools. Beryl **heals** tests as the
|
|
104
|
+
app's markup drifts, but only when a test has the three things that make it durable: a
|
|
105
|
+
real **outcome assertion**, a strong **natural-language intent**, and the **local
|
|
106
|
+
run-fix loop**.
|
|
106
107
|
|
|
107
108
|
## 0. Start here
|
|
108
109
|
|
|
109
|
-
**
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
110
|
+
**New tests are authored only by Beryl's cloud agent.** There is no local authoring: you
|
|
111
|
+
do not draft a new plan here. A human writes a document (a PRD, release notes, a list of
|
|
112
|
+
flows) and \`jobs_create\` hands it to the agent, which explores the environment and
|
|
113
|
+
authors one test per scenario and surface. \`jobs_get\` shows the job's status, the tests
|
|
114
|
+
it produced and any scenario it skipped. The document is sent to the LLM that drives the
|
|
115
|
+
agent, so it must not hold anything the user would not share with it.
|
|
116
|
+
|
|
117
|
+
What you do locally: set up the accounts tests sign in as (§1), run tests on this machine
|
|
118
|
+
(§4), and fix a banked test's plan when it needs it (§2).
|
|
113
119
|
|
|
114
120
|
Set up (once): if you cannot call the Beryl tools yet, run \`beryl init\` in a terminal to
|
|
115
|
-
install the browser and wire the tools. Then, before the first
|
|
121
|
+
install the browser and wire the tools. Then, before the first job:
|
|
116
122
|
|
|
117
123
|
- \`envs_list\`: the root URL gotos resolve against (§2). If it is empty, set it with
|
|
118
|
-
\`envs_update
|
|
124
|
+
\`envs_update\` to a URL Beryl's cloud can reach, never \`localhost\`, \`127.0.0.1\`
|
|
125
|
+
or a private IP (refused). For a local dev server, pass \`url-override\` to
|
|
126
|
+
\`runs_local\` instead.
|
|
119
127
|
- \`accounts_list\`: who do authenticated tests sign in as (§1)? Store its sign-in with
|
|
120
128
|
\`accounts_set_login\`, prove it with \`accounts_check\`.
|
|
121
129
|
- \`mailbox_get\`: the address the account receives mail at (§5).
|
|
122
130
|
|
|
123
|
-
\`accounts_list\` comes FIRST:
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
2. **Write the ActionPlan**, with one real outcome assertion (§2).
|
|
131
|
-
3. **Lint it** with \`tests_lint\` (the plan plus its \`read\`/\`write\` access): schema and
|
|
132
|
-
access check, offline.
|
|
133
|
-
4. **Bank it** with \`tests_create\` (the title, the plan, its \`access\`, and the
|
|
134
|
-
\`description\` intent): it replays the plan locally, banks only on green, records no
|
|
135
|
-
run; a red replay hands back the failure to fix and re-run (§3 for the intent, §4 for
|
|
136
|
-
the loop).
|
|
137
|
-
5. When the batch is banked, run the new ids together ONCE with \`runs_local\` (or
|
|
131
|
+
\`accounts_list\` comes FIRST: an authenticated test cannot be saved until its account's
|
|
132
|
+
login plan is stored.
|
|
133
|
+
|
|
134
|
+
Then:
|
|
135
|
+
|
|
136
|
+
1. **Create tests** with \`jobs_create\` (the document, and the environment to explore).
|
|
137
|
+
2. When the job is done, run the new ids together ONCE with \`runs_local\` (or
|
|
138
138
|
\`runs_trigger\` for the cloud) so they land as one run with real first results.
|
|
139
|
+
3. **A test is red or wrong?** Fix its plan with the local run-fix loop (§4).
|
|
139
140
|
|
|
140
141
|
## 1. First: who do your tests sign in as?
|
|
141
142
|
|
|
142
143
|
An authenticated test **signs in as an account that already exists** — it does not sign
|
|
143
|
-
one up. Start every
|
|
144
|
+
one up. Start every session with \`accounts_list\`:
|
|
144
145
|
|
|
145
146
|
\`\`\`
|
|
146
|
-
label email type
|
|
147
|
-
default * qa@acme.test
|
|
148
|
-
admin qa+admin@x7k2p9.email.beryl.so
|
|
147
|
+
label email type login status
|
|
148
|
+
default * qa@acme.test provided password ready
|
|
149
|
+
admin qa+admin@x7k2p9.email.beryl.so provisioned (permanent) otp ready
|
|
149
150
|
\`\`\`
|
|
150
151
|
|
|
151
152
|
Author the sign-in as ordinary opening steps citing the reserved handles — fill
|
|
152
153
|
\`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell.
|
|
153
|
-
Set \`requires_auth: true\` AND \`auth_mode
|
|
154
|
-
default and
|
|
154
|
+
Set \`requires_auth: true\` AND \`auth_mode\`, both, always; \`auth_mode\` has NO
|
|
155
|
+
default and a save rejects a \`requires_auth\` plan without it. \`"inline"\` = the plan
|
|
155
156
|
signs itself in like this; \`"session"\` = no sign-in steps, rides the account's
|
|
156
157
|
once-per-run session (§ Session mode). \`auth_label\` picks a non-default identity
|
|
157
158
|
(the \`*\` row is the default). Handles resolve at run time to the named account, so
|
|
158
159
|
one plan stays correct across environments.
|
|
159
160
|
|
|
160
|
-
**The account needs a stored login plan before any authenticated test can be
|
|
161
|
+
**The account needs a stored login plan before any authenticated test can be saved.**
|
|
161
162
|
A run replays that plan to produce the session its tests ride, and replays it again
|
|
162
|
-
when the session expires, so
|
|
163
|
-
|
|
163
|
+
when the session expires, so Beryl rejects a \`requires_auth\` plan whose account has
|
|
164
|
+
none, naming the fix. Store it once per account with
|
|
164
165
|
\`accounts_set_login\` (§ Session mode).
|
|
165
166
|
|
|
166
167
|
**NEVER paste a real email or password into a plan.** Handles resolve at run time —
|
|
@@ -237,9 +238,9 @@ An empty \`accounts_list\` means nothing is set up. In order of preference:
|
|
|
237
238
|
let the plan \`await_email\` through; later sign-ins read the same mailbox.
|
|
238
239
|
4. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
|
|
239
240
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
241
|
+
Every saved login is a test account: \`{{login_email}}\` and \`{{login_password}}\` resolve
|
|
242
|
+
only from the environment's account, and a login password is never set with
|
|
243
|
+
\`config secrets set\`. The webapp's Test accounts tab edits the same accounts.
|
|
243
244
|
|
|
244
245
|
### Session mode: sign in once per run, not once per test
|
|
245
246
|
|
|
@@ -262,26 +263,21 @@ descriptions.
|
|
|
262
263
|
Then write the tests with \`requires_auth: true\`, \`auth_mode: "session"\`, and NO
|
|
263
264
|
sign-in steps.
|
|
264
265
|
|
|
265
|
-
**Creates share the sign-in.** \`tests_create\` replays session-mode plans against the
|
|
266
|
-
account's established session (reused while it stays live), so a batch of creates costs
|
|
267
|
-
at most one sign-in — but still create one at a time: racing creates race the same
|
|
268
|
-
mailbox for mail.
|
|
269
|
-
|
|
270
266
|
An expired session is never your problem: every run proves the stored session and, on a
|
|
271
267
|
miss, signs in again from the login plan. A "reconnect" message only appears when there
|
|
272
268
|
is no stored login plan to refresh from.
|
|
273
269
|
|
|
274
270
|
When \`accounts_check\` fails, its description explains which half broke and each
|
|
275
|
-
\`SESSION_*\` reason. The one that changes your
|
|
271
|
+
\`SESSION_*\` reason. The one that changes your plans: an account marked
|
|
276
272
|
unsupported (its session cannot survive a fresh browser) is not your bug, but it IS
|
|
277
|
-
your move
|
|
273
|
+
your move: switch its session-mode tests to \`auth_mode: "inline"\` and their own
|
|
278
274
|
sign-in steps.
|
|
279
275
|
|
|
280
276
|
### Two identities in one test
|
|
281
277
|
|
|
282
|
-
Invite-a-teammate-and-accept-as-them needs two identities mid-flow
|
|
283
|
-
one test. Bank the half the app shows to account A ("the invitation is listed as
|
|
284
|
-
pending", "the share link is issued")
|
|
278
|
+
Invite-a-teammate-and-accept-as-them needs two identities mid-flow, so it is not bankable
|
|
279
|
+
as one test. Bank the half the app shows to account A ("the invitation is listed as
|
|
280
|
+
pending", "the share link is issued"): a real, strong outcome. While fixing a plan you can
|
|
285
281
|
drive the full handshake live: a second account (\`accounts_create\`, labelled), or a
|
|
286
282
|
second mailbox (\`mailbox_create\`, read with \`mailbox_read\`).
|
|
287
283
|
|
|
@@ -289,11 +285,14 @@ SSO-only sites are not a dead end: with a dedicated Google test account they fol
|
|
|
289
285
|
item 2 like any password account. The captured session is the FALLBACK — no test account
|
|
290
286
|
can be provided, or Google challenges the scripted sign-in: a human signs in once
|
|
291
287
|
through a live browser view in the Beryl webapp, Beryl stores the session encrypted
|
|
292
|
-
server-side and injects it into cloud runs
|
|
293
|
-
|
|
294
|
-
|
|
288
|
+
server-side and injects it into cloud runs: \`runs_trigger\` runs such tests,
|
|
289
|
+
\`runs_local\` skips them (the session never leaves Beryl's cloud).
|
|
290
|
+
|
|
291
|
+
## 2. Fixing a test's plan over the Playwright MCP
|
|
295
292
|
|
|
296
|
-
|
|
293
|
+
New tests come from \`jobs_create\` (§0). You write plan JSON only to fix a banked test
|
|
294
|
+
(\`tests_plan\` returns its current plan) or to store an account's sign-in or signup
|
|
295
|
+
plan (§1). Either way, the rules below hold.
|
|
297
296
|
|
|
298
297
|
1. **Drive the flow in a real browser first.** Walk it by hand over the Playwright MCP
|
|
299
298
|
— log in, fill, submit — acting on elements by their accessibility ref from the
|
|
@@ -355,17 +354,13 @@ ${JSON.stringify(BERYL_TEST_SKILL_EXAMPLE_PLAN, null, 2)
|
|
|
355
354
|
.map((line) => ` ${line}`)
|
|
356
355
|
.join("\n")}
|
|
357
356
|
\`\`\`
|
|
358
|
-
3. **Validate offline, then
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
own description. Inspect the exact spec with \`tests_script\`.
|
|
357
|
+
3. **Validate offline, then save.** Lint with \`tests_lint\` (schema check, no network),
|
|
358
|
+
inspect the exact spec with \`tests_script\`, then save with \`tests_recompile\`
|
|
359
|
+
(replays the edit against the live site first and rejects it on a red replay) or
|
|
360
|
+
\`tests_set_plan\` (saves with no replay; the next run proves it). Pass a new
|
|
361
|
+
\`description\` only when the edit changes what the test proves (§3).
|
|
364
362
|
Full ActionPlan JSON Schema:
|
|
365
363
|
https://api.beryl.so/api/v1/schemas/action-plan.schema.json.
|
|
366
|
-
Groups are optional and never block: after the test is banked, if \`groups_list\`
|
|
367
|
-
shows any, offer them once (\`tests_set_groups\`); no answer = leave it ungrouped and
|
|
368
|
-
move on. Never create a group yourself, that is the user's call.
|
|
369
364
|
|
|
370
365
|
### Testing a mobile or tablet viewport
|
|
371
366
|
A test runs at a desktop viewport unless you say otherwise. To cover a responsive layout,
|
|
@@ -374,7 +369,7 @@ full Chromium device emulation, not a resize: the viewport, a mobile Chrome user
|
|
|
374
369
|
touch and the device pixel ratio all change together, so a hamburger or a nav that only
|
|
375
370
|
exists on small screens actually renders and is testable. \`mobile\` emulates a Pixel-class
|
|
376
371
|
phone, \`tablet\` a Galaxy-class tablet; both keep a Chrome user-agent so the emulated UA
|
|
377
|
-
matches the engine.
|
|
372
|
+
matches the engine. Drive the flow at that size over the Playwright MCP (resize the MCP
|
|
378
373
|
browser to match) so the refs you act on are the ones the small-screen layout renders. The
|
|
379
374
|
account, session and teardown rules are unchanged; \`device\` only sets the frame it runs in.
|
|
380
375
|
|
|
@@ -406,9 +401,9 @@ observable proof the flow worked.
|
|
|
406
401
|
- **Don't work around a real app failure to make a test go green.** A genuinely broken
|
|
407
402
|
flow is a finding to report, not something to paper over with a weaker assertion.
|
|
408
403
|
|
|
409
|
-
### Traps when
|
|
404
|
+
### Traps when writing a plan against a real app
|
|
410
405
|
|
|
411
|
-
Every one of these has produced a wrong plan or a red
|
|
406
|
+
Every one of these has produced a wrong plan or a red run. Check them.
|
|
412
407
|
|
|
413
408
|
1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
|
|
414
409
|
profile. Author while signed in and you never see the gate — you will mark gated
|
|
@@ -440,8 +435,9 @@ Every one of these has produced a wrong plan or a red \`tests_create\`. Check th
|
|
|
440
435
|
|
|
441
436
|
## 3. Writing the natural-language intent
|
|
442
437
|
|
|
443
|
-
|
|
444
|
-
|
|
438
|
+
The cloud agent drafts each new test's intent. When a fix changes what a test proves,
|
|
439
|
+
pass the new intent as the \`description\` of \`tests_set_plan\`. 1-3 sentences. This is
|
|
440
|
+
the immutable anchor from §6.
|
|
445
441
|
|
|
446
442
|
- **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
|
|
447
443
|
clicks submit" — the trajectory will change. Instead: *what does a green run prove is
|
|
@@ -464,10 +460,10 @@ with the Playwright runner bundled in the CLI: no cloud, no waiting for a schedu
|
|
|
464
460
|
Run it un-synced with a report directory against one test id while you iterate, or with
|
|
465
461
|
no id to run the whole suite and record the results in Beryl.
|
|
466
462
|
|
|
467
|
-
The loop:
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
463
|
+
The loop: \`runs_local\` un-synced against the test id → read the run's \`report.json\` to
|
|
464
|
+
see which step or assertion failed and why → fix the plan (§2) → \`tests_lint\` →
|
|
465
|
+
\`tests_set_plan\` → run again. When the batch is done, one synced \`runs_local\` (or
|
|
466
|
+
\`runs_trigger\`) records it as a single run. Its options,
|
|
471
467
|
session-mode behavior, and \`await_email\` / authenticated-test handling are in the
|
|
472
468
|
\`runs_local\` tool's own description.
|
|
473
469
|
|
|
@@ -525,12 +521,9 @@ For a magic-link flow, replace the code steps with
|
|
|
525
521
|
|
|
526
522
|
Two caveats:
|
|
527
523
|
|
|
528
|
-
- \`
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
signs up / sends mail FOR REAL, so turn \`tests_create\`'s verify off only if that side
|
|
532
|
-
effect is unwanted. \`runs_local\` serves \`await_email\` the same way, so the local loop
|
|
533
|
-
covers OTP/signup flows end to end.
|
|
524
|
+
- \`runs_local\` serves \`await_email\` like a cloud run: it receives at the project
|
|
525
|
+
mailbox and answers each step over the API, so the local loop covers OTP/signup flows
|
|
526
|
+
end to end. It signs up / sends mail FOR REAL, like every run.
|
|
534
527
|
- §2's outcome discipline still applies: the green signal is the post-verification
|
|
535
528
|
state (the welcome screen, the dashboard), not "an email arrived".
|
|
536
529
|
|
|
@@ -575,8 +568,8 @@ Rules:
|
|
|
575
568
|
|
|
576
569
|
- Always tear down what the test connected in \`after\`: disconnect the integration in
|
|
577
570
|
the app.
|
|
578
|
-
- These steps run only in Beryl's cloud: \`
|
|
579
|
-
\`
|
|
571
|
+
- These steps run only in Beryl's cloud: \`runs_local\` skips them, so run them with
|
|
572
|
+
\`runs_trigger\`.
|
|
580
573
|
- A setup error saying Beryl's access "is being restored" is on Beryl's side: do not
|
|
581
574
|
edit the test.
|
|
582
575
|
- Never add steps that browse Slack itself; the actions above do all of it.
|