@beryl-so/cli 0.67.0 → 0.77.1

package/README.md

@@ -13,10 +13,9 @@ npx @beryl-so/cli@latest init
 ```
 
 Signs you in (browser approval by default, emailed one-time code as fallback; new
-emails are signed up on the spot), wires the beryl + playwright MCP servers into the
+emails are signed up on the spot), wires the beryl MCP server into the
 agent you pick with --claude, --cursor, --vscode or --opencode (it asks when you pass
-none), installs the beryl-test authoring skill, and installs Playwright for local runs.
-Safe to re-run.
+none), and installs the beryl-test authoring skill. Safe to re-run.
 
 ## Install
 
@@ -78,11 +77,11 @@ Set up Beryl in this repo: sign in and wire up your coding agent
 
 ### guide
 
-Print the Beryl test-authoring guide
+Print the Beryl test guide
 
 | Command | Summary | MCP tool |
 | --- | --- | --- |
-| `beryl guide` | Print the Beryl test-authoring guide | `guide` |
+| `beryl guide` | Print the Beryl test guide | `guide` |
 
 ### login
 
@@ -177,9 +176,11 @@ Create and manage projects: a site you author tests for and run.
 | `beryl projects create [url]` | Create a project for a URL, or an empty one with just --name | `projects_create` |
 | `beryl projects rename <name>` | Rename a project | `projects_rename` |
 | `beryl projects quarantine [state]` | Turn auto quarantine on or off and tune when it mutes and releases a test | `projects_quarantine` |
+| `beryl projects surfaces <surfaces...>` | Set which surfaces are enabled for this project | `projects_surfaces` |
+| `beryl projects findings-gate <mode>` | Turn the findings gate on each run on or off | `projects_findings_gate` |
 | `beryl projects delete` | Delete a project and all its tests and runs | `projects_delete` |
 | `beryl projects re-explore` | Send the agent back in to run/heal existing tests and discover new flows | none |
-| `beryl projects report` | Aggregate quality report across recent runs (pass rates, flaky tests, trend) | `projects_report` |
+| `beryl projects report` | Aggregate quality report across recent runs (pass rates, per-test history, trend) | `projects_report` |
 | `beryl projects reusable-auth <url>` | Check whether a saved login can be reused for a URL before creating a project | `projects_reusable_auth` |
 
 ### envs
@@ -191,7 +192,7 @@ Manage a project's environments: the URLs and auth Beryl runs tests against.
 | `beryl envs list` | List a project's environments | `envs_list` |
 | `beryl envs get <env-id>` | Show one environment | `envs_get` |
 | `beryl envs create <name> <url>` | Add an environment (e.g. staging) to a project | `envs_create` |
-| `beryl envs update <env-id>` | Update an environment's name, URL, auth settings, Site Health screen sizes, schedule, CI and pages, or issue tracker settings | `envs_update` |
+| `beryl envs update <env-id>` | Update an environment's name, URL, app ids, auth settings, Site Health screen sizes, schedule, CI and pages, or issue tracker settings | `envs_update` |
 | `beryl envs delete <env-id>` | Delete an environment | `envs_delete` |
 
 ### schedule
@@ -213,13 +214,12 @@ Author, inspect, version, and heal a project's tests: the checks Beryl runs on e
 
 | Command | Summary | MCP tool |
 | --- | --- | --- |
-| `beryl tests lint` | Validate a plan JSON file offline, before sending it to the server | `tests_lint` |
 | `beryl tests list` | List the project's tests with their latest result | `tests_list` |
 | `beryl tests get <test-id>` | Show one test | `tests_get` |
 | `beryl tests plan <test-id>` | Print a test's current step plan (JSON) | `tests_plan` |
-| `beryl tests create` | Create a test case from a JSON action plan (for tests authored locally, e.g. by your coding agent) | `tests_create` |
 | `beryl tests set-plan <test-id>` | Replace a test's step plan from a JSON file (creates a new version) | `tests_set_plan` |
 | `beryl tests rename <test-id> <title>` | Rename a test | `tests_rename` |
+| `beryl tests set-intent <test-id> <intent>` | Edit what a test proves, the target heal judges against | `tests_set_intent` |
 | `beryl tests set-groups <test-id>` | Replace the groups a test belongs to | `tests_set_groups` |
 | `beryl tests set-access <test-id> <access>` | Mark a test read or write | `tests_set_access` |
 | `beryl tests add-groups <test-ids...>` | Add groups to one or more tests, keeping the groups they already have | `tests_add_groups` |
@@ -234,7 +234,7 @@ Author, inspect, version, and heal a project's tests: the checks Beryl runs on e
 | `beryl tests reset <test-id>` | Discard user edits and return the test to its latest system-authored version | `tests_reset` |
 | `beryl tests heal <test-id> <state>` | Turn self-healing on or off for a test | `tests_heal` |
 | `beryl tests history <test-id>` | Pass/fail history, streak, and stability for a test | `tests_history` |
-| `beryl tests script [test-id]` | Print the rendered Playwright spec for a test (or an unbanked plan file) | `tests_script` |
+| `beryl tests script <test-id>` | Print the rendered Playwright spec for a test | `tests_script` |
 | `beryl tests export <test-ids...>` | Export tests as Playwright .spec.ts files in a ZIP | `tests_export` |
 
 ### groups
@@ -259,7 +259,6 @@ Trigger a run of a project's tests (e.g. in CI), then watch, inspect, and downlo
 | Command | Summary | MCP tool |
 | --- | --- | --- |
 | `beryl runs trigger` | Trigger a test run (whole suite, a subset, or one environment) | `runs_trigger` |
-| `beryl runs local [test-ids...]` | Run tests on your machine with Playwright; results sync to Beryl | `runs_local` |
 | `beryl runs list` | List recent runs | `runs_list` |
 | `beryl runs get <run-id>` | Show one run with its per-test results | `runs_get` |
 | `beryl runs watch <run-id>` | Follow a run until it finishes | `runs_watch` |
@@ -286,6 +285,18 @@ Site Health, or how your site reads to search engines and visitors: content, spe
 | `beryl health tracker teams` | List the teams of the connected issue tracker | `health_tracker_teams` |
 | `beryl health tracker projects` | List a tracker team's projects | `health_tracker_projects` |
 
+### jobs
+
+Test-creation jobs: the cloud agent reads a document and authors tests from it.
+
+| Command | Summary | MCP tool |
+| --- | --- | --- |
+| `beryl jobs create [file]` | Create tests from a document (PRD, release notes) on Beryl's cloud | `jobs_create` |
+| `beryl jobs list` | List the project's test-creation jobs | `jobs_list` |
+| `beryl jobs get <job-id>` | Show a job's status and the tests it created | `jobs_get` |
+| `beryl jobs clone <job-id>` | Run a job again: queue a new job with the same document and environment | `jobs_clone` |
+| `beryl jobs delete <job-id>` | Delete a job; the flows and tests it created stay | `jobs_delete` |
+
 ### explorations
 
 Inspect the agent's exploration runs: how it crawled a site and authored its tests.
@@ -308,13 +319,13 @@ Manage the variables, secrets, and files the agent can use while exploring and r
 | `beryl config vars set <key> <value>` | Create or update a config variable | `config_vars_set` |
 | `beryl config vars get <key>` | Show one config variable | `config_vars_get` |
 | `beryl config vars delete <key>` | Delete a config variable | `config_vars_delete` |
-| `beryl config secrets list` | List the project's secrets (values are never returned) | `config_secrets_list` |
+| `beryl config secrets list` | List the project's secrets (values are never returned); a test account's password is managed on the account, not here | `config_secrets_list` |
 | `beryl config secrets set <key> <value>` | Create a secret (write-only; re-setting a key replaces it) | `config_secrets_set` |
 | `beryl config secrets get <key>` | Show one secret's metadata, or reveal its value with --reveal | `config_secrets_get` |
 | `beryl config secrets delete <key>` | Delete a secret | `config_secrets_delete` |
-| `beryl config files list` | List files uploaded for the agent to use (e.g. CSVs, upload fixtures) | `config_files_list` |
+| `beryl config files list` | List uploaded files: fixtures tests attach and Android and iOS app builds | `config_files_list` |
 | `beryl config files get <file>` | Show one uploaded file's metadata | `config_files_get` |
-| `beryl config files upload <file>` | Upload a file | `config_files_upload` |
+| `beryl config files upload <file>` | Upload a file: a fixture an upload step attaches, or an Android or iOS app build | `config_files_upload` |
 | `beryl config files download <file-id>` | Get a short-lived download URL for a file | `config_files_download` |
 | `beryl config files delete <file-id>` | Delete an uploaded file | `config_files_delete` |
 
@@ -356,6 +367,7 @@ Durable identities on the site under test: what an authenticated test signs in a
 | `beryl accounts list` | List the test accounts an environment's tests sign in as | `accounts_list` |
 | `beryl accounts get <account-id>` | Show one test account: origin, lifetime, sign-in state | `accounts_get` |
 | `beryl accounts create` | Add a test account: the customer's own, or one Beryl signs up | `accounts_create` |
+| `beryl accounts capture` | Store a signed-in session as a captured test account | `accounts_capture` |
 | `beryl accounts signup <account-id>` | Store a test account's signup plan and sign it up on its environment | `accounts_signup` |
 | `beryl accounts set-login <account-id>` | Store the sign-in plan a run replays once, plus the probe that proves it | `accounts_set_login` |
 | `beryl accounts get-login <account-id>` | Read the stored sign-in plan, its probe, and the hash a safe write must cite | `accounts_get_login` |

package/dist/adapters/mcp.js

@@ -278,13 +278,15 @@ export function mcpInstructions() {
         "version, and whether this build is behind npm's latest). " +
         "Beryl authors, runs, and heals end-to-end tests for any web app: tests are JSON " +
         "action plans replayed in real cloud browsers, signing in as a durable test " +
-        "account whose mail arrives at the project's own mailbox — so signup/OTP/" +
+        "account whose mail arrives at the project's own mailbox, so signup/OTP/" +
         "magic-link flows are self-contained, with no human login needed. " +
-        "The full authoring guide (plan shape, outcome assertions, email/OTP wiring, " +
-        "run-fix loop) ships as both the beryl-test skill and the `guide` tool — same " +
+        "New tests are authored only by Beryl's cloud agent: give it a document describing " +
+        "the flows with `jobs_create`; there is no local authoring. " +
+        "The guide (plan shape, outcome assertions, accounts, email/OTP wiring, the " +
+        "run-fix loop) ships as both the beryl-test skill and the `guide` tool, same " +
         `content. If a beryl-test skill stating v${version} is already loaded, do not ` +
         "call `guide`; if no beryl-test skill is available or it states another version, " +
-        "call `guide` before authoring your first test plan. " +
+        "call `guide` before editing a test plan or a sign-in plan. " +
         "An account can hold several workspaces and projects: call `projects_list` first " +
         "and pass `project` (an id is enough) on every project-scoped tool, or the call fails.");
 }

package/dist/beryl-test-skill.js

@@ -1,4 +1,4 @@
-// The `beryl-test` authoring skill, installed by `beryl init` into
+// The `beryl-test` skill, installed by `beryl init` into
 // `.agents/skills/beryl-test/SKILL.md` (vendor-neutral, editor-agnostic) — and, when
 // claude-code is a selected editor, ALSO into the `.claude/skills/beryl-test/SKILL.md`
 // that Claude Code actually indexes (see `writeSkills` in `commands/init.ts`). Kept as an
@@ -17,23 +17,24 @@ export const BERYL_TEST_SKILL_DIR = "beryl-test";
 // nor `.claude/skills/`, so `init` writes them their own pointer instead. Thin on purpose:
 // it sends the agent to the `guide` MCP tool rather than inlining the 500-line skill, and
 // `guide` ships with the CLI so the real guidance never drifts from the installed version.
-const BERYL_GUIDE_POINTER = `When writing, editing, or debugging Beryl tests in this repo, call the \`guide\`
-MCP tool first. It returns the full authoring guide: the ActionPlan shape, outcome
-assertions, test accounts and the project mailbox, and the local run-fix loop.
+const BERYL_GUIDE_POINTER = `When creating, running, editing, or debugging Beryl tests in this repo, call the
+\`guide\` MCP tool first. It returns the full guide: where new tests come from, the
+ActionPlan shape, outcome assertions, test accounts and the project mailbox, and the
+local run-fix loop.
 
-Then use the Beryl MCP tools: \`projects_list\` to find the project, and \`tests_create\`
-to author a plan and replay it locally before it is banked.
+Then use the Beryl MCP tools: \`projects_list\` to find the project, \`jobs_create\` to
+have Beryl's cloud agent author tests from a document, and \`runs_trigger\` to run them.
 
 The \`guide\` tool ships with the CLI and always matches the installed version, so treat
 it as the source of truth over anything cached here.`;
 // Cursor reads its own `.cursor/rules/*.mdc`, auto-loaded when alwaysApply is set.
 export const BERYL_CURSOR_RULE_FILENAME = "beryl.mdc";
 export const BERYL_CURSOR_RULE = `---
-description: Authoring and editing Beryl end-to-end tests
+description: Creating, running and editing Beryl end-to-end tests
 alwaysApply: true
 ---
 
-# Beryl test authoring
+# Beryl tests
 
 ${BERYL_GUIDE_POINTER}
 `;
@@ -43,7 +44,7 @@ ${BERYL_GUIDE_POINTER}
 export const BERYL_COPILOT_MARK_BEGIN = "<!-- BEGIN beryl (managed by beryl init) -->";
 export const BERYL_COPILOT_MARK_END = "<!-- END beryl -->";
 export const BERYL_COPILOT_SECTION = `${BERYL_COPILOT_MARK_BEGIN}
-## Beryl test authoring
+## Beryl tests
 
 ${BERYL_GUIDE_POINTER}
 ${BERYL_COPILOT_MARK_END}`;
@@ -89,60 +90,64 @@ export const BERYL_TEST_SKILL_OTP_EXAMPLE_PLAN = {
 };
 export const BERYL_TEST_SKILL = `---
 name: beryl-test
-description: Author durable, healable end-to-end tests for a web app with Beryl. Use when writing, running, or fixing a Beryl test locally with your own coding agent, drafting the plan over the Playwright MCP, writing the natural-language intent, and running it with \`runs_local\`.
+description: Create, run, and fix durable, healable end-to-end tests for a web app with Beryl. Use when asking Beryl's cloud agent to create tests from a document (\`jobs_create\`), running them in Beryl's cloud with \`runs_trigger\`, setting up test accounts, or fixing a banked test's plan with your own coding agent.
 ---
 
-# Authoring Beryl tests
+# Beryl tests
 
-beryl-test v${cliVersion()} — installed by \`beryl init\`; the \`guide\` tool always returns the current version of this guide.
+beryl-test v${cliVersion()}, installed by \`beryl init\`; the \`guide\` tool always returns the current version of this guide.
 
 A Beryl test drives a real browser through a flow and asserts the flow reached its
-meaningful outcome. \`beryl init\` wired two MCP servers: **beryl** (the tools you author
-with: \`accounts_list\`, \`tests_create\`, \`runs_local\`, and the rest) and **playwright**
-(drive a real browser). You do everything below by calling those tools. Beryl **heals**
-tests as the app's markup drifts, but only when you supply the three things that make a
-test durable: a real **outcome assertion**, a strong **natural-language intent**, and the
-**local run-fix loop**.
+meaningful outcome. \`beryl init\` wired the **beryl** MCP server (\`jobs_create\`,
+\`accounts_list\`, \`runs_trigger\`, and the rest). You do everything below by
+calling those tools; tests are authored and run in Beryl's cloud. Beryl **heals**
+tests as the app's markup drifts, but only when a test has the three things that make
+it durable: a real **outcome assertion**, a strong **natural-language intent**, and a
+**verified plan** (every save is proven by a replay).
 
 ## 0. Start here
 
-**Everything happens on your machine first**: you drive the flow here, and
-\`tests_create\` proves the plan here too, so a working local browser IS the loop.
-Keep \`tests_create\`'s verify on (banking unproven is a trap); replay mechanics and the
-cloud-session exception are in the \`tests_create\` tool's own description.
+**New tests are authored only by Beryl's cloud agent.** There is no local authoring: you
+do not draft a new plan here. A human writes a document (a PRD, release notes, a list of
+flows) and \`jobs_create\` hands it to the agent, which explores the environment and
+authors one test per scenario and surface. \`jobs_get\` shows the job's status, the tests
+it produced and any scenario it skipped. The document is sent to the LLM that drives the
+agent, so it must not hold anything the user would not share with it.
 
-Set up (once): if you cannot call the Beryl tools yet, run \`beryl init\` in a terminal to
-install the browser and wire the tools. Then, before the first plan:
+What you do here: set up the accounts tests sign in as (§1), trigger and read runs
+(§4), and fix a banked test's plan when it needs it (§2).
+
+Set up (once): if you cannot call the Beryl tools yet, run \`beryl init\` in a terminal
+to wire them. Then, before the first job:
 
 - \`envs_list\`: the root URL gotos resolve against (§2). If it is empty, set it with
   \`envs_update\` to a URL Beryl's cloud can reach, never \`localhost\`, \`127.0.0.1\`
-  or a private IP (refused). For a local dev server, pass \`url-override\` to
-  \`tests_create\` and \`runs_local\` instead.
+  or a private IP (refused). A local dev server is not testable until it is reachable
+  from the cloud (deploy a preview, or tunnel it).
 - \`accounts_list\`: who do authenticated tests sign in as (§1)? Store its sign-in with
   \`accounts_set_login\`, prove it with \`accounts_check\`.
 - \`mailbox_get\`: the address the account receives mail at (§5).
 
-\`accounts_list\` comes FIRST: standing account vs new identity changes the plan you
-write (discovering it afterwards means rewriting), and an authenticated test cannot be
-created until its account's login plan is stored.
+\`accounts_list\` comes FIRST: an authenticated test cannot be saved until its account's
+login plan is stored.
 
-Then, per test:
+Then:
 
-1. **Drive the flow** in a real browser over the Playwright MCP, never from imagination (§2).
-2. **Write the ActionPlan**, with one real outcome assertion (§2).
-3. **Lint it** with \`tests_lint\` (the plan plus its \`read\`/\`write\` access): schema and
-   access check, offline.
-4. **Bank it** with \`tests_create\` (the title, the plan, its \`access\`, and the
-   \`description\` intent): it replays the plan locally, banks only on green, records no
-   run; a red replay hands back the failure to fix and re-run (§3 for the intent, §4 for
-   the loop).
-5. When the batch is banked, run the new ids together ONCE with \`runs_local\` (or
-   \`runs_trigger\` for the cloud) so they land as one run with real first results.
+1. **Create tests** with \`jobs_create\` (the document, and the environment to
+   explore). \`jobs_get\` shows progress, the tests it banked (each one already
+   proven by a clean replay), and every scenario it skipped, with why.
+2. A job can pause as \`awaiting_input\` when the app's login wall blocks it (no
+   self-serve signup). Its \`needs\` field says what to do: capture a signed-in
+   session with the Beryl browser extension from the webapp. The job resumes by
+   itself and keeps everything it already banked.
+3. When the job is done, run the new ids together ONCE with \`runs_trigger\` so they
+   land as one run with real first results.
+4. **A test is red or wrong?** Fix its plan with the run-fix loop (§4).
 
 ## 1. First: who do your tests sign in as?
 
 An authenticated test **signs in as an account that already exists** — it does not sign
-one up. Start every authoring session with \`accounts_list\`:
+one up. Start every session with \`accounts_list\`:
 
 \`\`\`
 label      email                          type                     login     status
@@ -152,17 +157,17 @@ admin      qa+admin@x7k2p9.email.beryl.so provisioned (permanent)  otp       rea
 
 Author the sign-in as ordinary opening steps citing the reserved handles — fill
 \`{{login_email}}\`, fill \`{{login_password}}\`, submit, assert the logged-in shell.
-Set \`requires_auth: true\` AND \`auth_mode\` — both, always; \`auth_mode\` has NO
-default and create rejects a \`requires_auth\` plan without it. \`"inline"\` = the plan
+Set \`requires_auth: true\` AND \`auth_mode\`, both, always; \`auth_mode\` has NO
+default and a save rejects a \`requires_auth\` plan without it. \`"inline"\` = the plan
 signs itself in like this; \`"session"\` = no sign-in steps, rides the account's
 once-per-run session (§ Session mode). \`auth_label\` picks a non-default identity
 (the \`*\` row is the default). Handles resolve at run time to the named account, so
 one plan stays correct across environments.
 
-**The account needs a stored login plan before any authenticated test can be created.**
+**The account needs a stored login plan before any authenticated test can be saved.**
 A run replays that plan to produce the session its tests ride, and replays it again
-when the session expires, so \`tests_create\` rejects a \`requires_auth\` plan whose
-account has none, naming the fix. Store it once per account with
+when the session expires, so Beryl rejects a \`requires_auth\` plan whose account has
+none, naming the fix. Store it once per account with
 \`accounts_set_login\` (§ Session mode).
 
 **NEVER paste a real email or password into a plan.** Handles resolve at run time —
@@ -239,9 +244,9 @@ An empty \`accounts_list\` means nothing is set up. In order of preference:
    let the plan \`await_email\` through; later sign-ins read the same mailbox.
 4. **Neither** → the flow is not testable authenticated. Say so rather than guessing.
 
-A project with no accounts falls back to the \`LOGIN_EMAIL\` variable +
-\`LOGIN_PASSWORD\` secret with the same handle resolution, so existing tests are
-unaffected — but new work should create an account.
+Every saved login is a test account: \`{{login_email}}\` and \`{{login_password}}\` resolve
+only from the environment's account, and a login password is never set with
+\`config secrets set\`. The webapp's Test accounts tab edits the same accounts.
 
 ### Session mode: sign in once per run, not once per test
 
@@ -264,26 +269,21 @@ descriptions.
 Then write the tests with \`requires_auth: true\`, \`auth_mode: "session"\`, and NO
 sign-in steps.
 
-**Creates share the sign-in.** \`tests_create\` replays session-mode plans against the
-account's established session (reused while it stays live), so a batch of creates costs
-at most one sign-in — but still create one at a time: racing creates race the same
-mailbox for mail.
-
 An expired session is never your problem: every run proves the stored session and, on a
 miss, signs in again from the login plan. A "reconnect" message only appears when there
 is no stored login plan to refresh from.
 
 When \`accounts_check\` fails, its description explains which half broke and each
-\`SESSION_*\` reason. The one that changes your authoring: an account marked
+\`SESSION_*\` reason. The one that changes your plans: an account marked
 unsupported (its session cannot survive a fresh browser) is not your bug, but it IS
-your move — re-author its session-mode tests with \`auth_mode: "inline"\` and their own
+your move: switch its session-mode tests to \`auth_mode: "inline"\` and their own
 sign-in steps.
 
 ### Two identities in one test
 
-Invite-a-teammate-and-accept-as-them needs two identities mid-flow — not bankable as
-one test. Bank the half the app shows to account A ("the invitation is listed as
-pending", "the share link is issued") — a real, strong outcome. While AUTHORING you can
+Invite-a-teammate-and-accept-as-them needs two identities mid-flow, so it is not bankable
+as one test. Bank the half the app shows to account A ("the invitation is listed as
+pending", "the share link is issued"): a real, strong outcome. While fixing a plan you can
 drive the full handshake live: a second account (\`accounts_create\`, labelled), or a
 second mailbox (\`mailbox_create\`, read with \`mailbox_read\`).
 
@@ -291,15 +291,18 @@ SSO-only sites are not a dead end: with a dedicated Google test account they fol
 item 2 like any password account. The captured session is the FALLBACK — no test account
 can be provided, or Google challenges the scripted sign-in: a human signs in once
 through a live browser view in the Beryl webapp, Beryl stores the session encrypted
-server-side and injects it into cloud runs, and \`tests_create\` verifies such plans
-server-side, \`runs_trigger\` runs them, \`runs_local\` skips them (the session never
-leaves Beryl's cloud).
+server-side and injects it into cloud runs; the session never leaves Beryl's cloud.
+
+## 2. Fixing a test's plan
 
-## 2. Author locally over the Playwright MCP
+New tests come from \`jobs_create\` (§0). You write plan JSON only to fix a banked test
+(\`tests_plan\` returns its current plan) or to store an account's sign-in or signup
+plan (§1). Either way, the rules below hold.
 
-1. **Drive the flow in a real browser first.** Walk it by hand over the Playwright MCP
-   — log in, fill, submit — acting on elements by their accessibility ref from the
-   latest page snapshot, never a guessed selector. Watch what actually happens.
+1. **Start from evidence, not memory.** Pull the failed run first: \`runs_get\` for
+   the failing step and error, \`runs_explain\` for the run's own diagnosis,
+   \`tests_events\` for how the test behaved over time. Fix what the evidence shows,
+   never what you assume the page looks like.
 2. **Write it as an ActionPlan** — a JSON object whose \`steps\` are
    \`{action, selector, url, value, ...}\`. Two structural rules:
    - the **first executed step is a \`goto\`**, and
@@ -357,17 +360,13 @@ ${JSON.stringify(BERYL_TEST_SKILL_EXAMPLE_PLAN, null, 2)
     .map((line) => `   ${line}`)
     .join("\n")}
    \`\`\`
-3. **Validate offline, then create.** Lint with \`tests_lint\` (schema check, no network),
-   then bank with \`tests_create\` (the title, the plan, and the \`description\` intent, see
-   §3). \`tests_create\` banks the plan only on a green local replay; a red one hands back
-   the failure to fix and re-run. It records no run. Replay mechanics (where it runs over
-   MCP, its verify option, the cloud-session exception) are in the \`tests_create\` tool's
-   own description. Inspect the exact spec with \`tests_script\`.
+3. **Save through the replay gate.** \`tests_recompile\` replays the edited plan
+   against the live site first and rejects it on a red replay; prefer it always.
+   \`tests_set_plan\` saves with no replay (the next run proves it); reserve it for
+   when the flow cannot replay right now (for example a third-party step). Pass a new
+   \`description\` only when the edit changes what the test proves (§3).
    Full ActionPlan JSON Schema:
    https://api.beryl.so/api/v1/schemas/action-plan.schema.json.
-   Groups are optional and never block: after the test is banked, if \`groups_list\`
-   shows any, offer them once (\`tests_set_groups\`); no answer = leave it ungrouped and
-   move on. Never create a group yourself, that is the user's call.
 
 ### Testing a mobile or tablet viewport
 A test runs at a desktop viewport unless you say otherwise. To cover a responsive layout,
@@ -376,9 +375,32 @@ full Chromium device emulation, not a resize: the viewport, a mobile Chrome user
 touch and the device pixel ratio all change together, so a hamburger or a nav that only
 exists on small screens actually renders and is testable. \`mobile\` emulates a Pixel-class
 phone, \`tablet\` a Galaxy-class tablet; both keep a Chrome user-agent so the emulated UA
-matches the engine. Author the flow at that size over the Playwright MCP (resize the MCP
-browser to match) so the refs you act on are the ones the small-screen layout renders. The
-account, session and teardown rules are unchanged; \`device\` only sets the frame it runs in.
+matches the engine. A test's surface is fixed when it is created, so when fixing a plan
+keep \`device\` as it is: a small-screen test's selectors must come from what the
+small-screen layout renders (a hamburger, not the desktop nav). The account, session and
+teardown rules are unchanged; \`device\` only sets the frame it runs in.
+
+### App tests (Android and iOS)
+\`device: "android"\` and \`device: "ios"\` are not browsers: the plan drives the environment's
+app (its \`app_ids.android\` from the newest APK, or \`app_ids.ios\` from the newest simulator
+build, uploaded to Files) on an emulator or simulator in Beryl's cloud. App tests are written
+and verified by a test-creation job (\`jobs_create\`), never run locally. When you edit one,
+its rules differ from a web plan's:
+
+- The first step is \`launch\` (open the app), not \`goto\`. A later \`launch\` relaunches
+  with the app's data kept, which is how a plan proves something persisted.
+- Steps: launch, back (Android only; iOS has no back button), fill, click, press, scroll,
+  wait_for, expect, await_email.
+- Selectors are \`id=<resource id>\` (Android) or \`id=<accessibility identifier>\` (iOS), or
+  \`text=<visible text or accessibility label>\`; no CSS. React Native's \`testID\` shows up as
+  the id on both; a Flutter icon button needs a \`Semantics\` label to have text.
+- \`have_text\` needs an \`id=\` selector; to check text is on screen, use \`visible\` with
+  \`text=\`. No URL or title expects: an app has no URL.
+- Keys: Enter and Backspace, plus Tab on Android. Never put \`\${\` in plan text.
+- A signed-in app test sets \`requires_auth\` and \`auth_mode: "session"\` and rides the
+  account's sign-in on that surface. The test-creation job saves that sign-in itself when
+  the app has a login screen; \`accounts_set_login\` with \`surface: "android"\` or
+  \`surface: "ios"\` only replaces it.
 
 ### The outcome assertion is the whole game
 
@@ -408,17 +430,16 @@ observable proof the flow worked.
 - **Don't work around a real app failure to make a test go green.** A genuinely broken
   flow is a finding to report, not something to paper over with a weaker assertion.
 
-### Traps when authoring against a real app
+### Traps when writing a plan against a real app
 
-Every one of these has produced a wrong plan or a red \`tests_create\`. Check them.
+Every one of these has produced a wrong plan or a red run. Check them.
 
-1. **Your browser may already be signed in.** The Playwright MCP keeps a persistent
-   profile. Author while signed in and you never see the gate — you will mark gated
-   pages as public. **Log out first**, then confirm the gated page really redirects to
-   the login.
+1. **A gated page can look public in signed-in evidence.** Run artifacts captured
+   while a session was live never show the login wall, so do not mark a flow public
+   (no \`requires_auth\`) from them; check how the page behaves logged out.
 2. **The sign-UP flow is not the sign-IN flow.** A brand-new address often gets an
    extra "create your account" step a returning address skips. A stored login plan must
-   be the **returning** path — drive it twice: once to create, once to sign in again.
+   be the **returning** path: what the app shows the SECOND time that account arrives.
 3. **Read the DOM, not just the accessibility tree, before picking a selector.** Two
    buttons can share a label ("Continue" / "Continue with Google"), and Beryl relaxes
    \`text=X\` to a case-insensitive SUBSTRING — \`text=Continue\` is ambiguous. Use
@@ -442,8 +463,9 @@ Every one of these has produced a wrong plan or a red \`tests_create\`. Check th
 
 ## 3. Writing the natural-language intent
 
-Give the intent as the \`description\` of \`tests_create\` (or \`tests_set_plan\`
-when you re-author). 1-3 sentences. This is the immutable anchor from §6.
+The cloud agent drafts each new test's intent. When a fix changes what a test proves,
+pass the new intent as the \`description\` of \`tests_set_plan\`. 1-3 sentences. This is
+the immutable anchor from §6.
 
 - **State the purpose, not the steps.** Not "clicks Sign in, types email and password,
   clicks submit" — the trajectory will change. Instead: *what does a green run prove is
@@ -459,19 +481,18 @@ Good:
 Weak (describes steps + asserts nothing meaningful):
 > "Goes to /login, fills the form, and checks the page loaded."
 
-## 4. The local run-fix loop
+## 4. The run-fix loop
 
-Iterate on your machine before you rely on the cloud. \`runs_local\` runs banked tests
-with the Playwright runner bundled in the CLI: no cloud, no waiting for a scheduled run.
-Run it un-synced with a report directory against one test id while you iterate, or with
-no id to run the whole suite and record the results in Beryl.
+Every run happens in Beryl's cloud. \`runs_trigger\` against one test id (or a list)
+starts a run; \`runs_watch\` follows it live; \`runs_get\` returns each result's
+failing step, error and artifacts; \`runs_explain\` turns a red run into a diagnosis.
 
-The loop: draft → \`tests_lint\` → \`tests_create\` (proves, records nothing) → \`runs_local\`
-un-synced while iterating → read the run's \`report.json\` to see which step or assertion
-failed and why → fix the plan → \`tests_set_plan\` → run again. When the batch is done, one
-synced \`runs_local\` (or \`runs_trigger\`) records it as a single run. Its options,
-session-mode behavior, and \`await_email\` / authenticated-test handling are in the
-\`runs_local\` tool's own description.
+The loop: \`runs_trigger\` against the test id, read the failure with \`runs_get\` /
+\`runs_explain\`, fix the plan (§2), save with \`tests_recompile\` (which already
+replays it against the live site, so a green recompile IS the fix proven), then one
+final \`runs_trigger\` so the suite records a clean run. If recompile keeps rejecting
+an edit, the evidence disagrees with your theory: pull more of it rather than trying
+variations.
 
 ## 5. Testing an OTP / signup flow (\`await_email\`)
 
@@ -527,12 +548,8 @@ For a magic-link flow, replace the code steps with
 
 Two caveats:
 
-- \`tests_create\` verifies an \`await_email\` plan like any other: the local
-  replay receives at the project mailbox and answers each step over the API, so the
-  app's mail really is received and extracted before the test is accepted. The replay
-  signs up / sends mail FOR REAL, so turn \`tests_create\`'s verify off only if that side
-  effect is unwanted. \`runs_local\` serves \`await_email\` the same way, so the local loop
-  covers OTP/signup flows end to end.
+- Every run serves \`await_email\` from the project mailbox. It signs up / sends
+  mail FOR REAL, like every run.
 - §2's outcome discipline still applies: the green signal is the post-verification
   state (the welcome screen, the dashboard), not "an email arrived".
 
@@ -571,14 +588,19 @@ Two shapes:
   \`third_party_integrations_target\` names, trigger the app's message, then
   \`{ "action": "await_third_party", "provider": "slack", "expect_text": "..." }\`.
 - **Integration intact**: no declaration. Trigger the app's message, then
-  \`await_third_party\`.
+  \`await_third_party\`. When the app only answers its bot, ping the bot instead:
+  \`{ "action": "send_third_party", "provider": "slack", "value": "hello" }\` mentions it in
+  the project's channel, then \`await_third_party\` waits for its reply (in the channel or
+  in the thread). The bot's app must be recorded for the environment first:
+  \`third_party_integrations_target\` with \`env\` and \`app\`. A reply only the sender can
+  see never reaches Beryl, so such a bot can't be tested this way.
 
 Rules:
 
 - Always tear down what the test connected in \`after\`: disconnect the integration in
   the app.
-- These steps run only in Beryl's cloud: \`tests_create\` verifies server-side and
-  \`runs_local\` skips them, so run them with \`runs_trigger\`.
+- These steps need Beryl's provider accounts, which live in Beryl's cloud; they run
+  like any other cloud test.
 - A setup error saying Beryl's access "is being restored" is on Beryl's side: do not
   edit the test.
 - Never add steps that browse Slack itself; the actions above do all of it.