@beryl-so/cli 0.54.1 → 0.61.0

package/README.md

@@ -191,7 +191,7 @@ Manage a project's environments: the URLs and auth Beryl runs tests against.
 | `beryl envs list` | List a project's environments | `envs_list` |
 | `beryl envs get <env-id>` | Show one environment | `envs_get` |
 | `beryl envs create <name> <url>` | Add an environment (e.g. staging) to a project | `envs_create` |
-| `beryl envs update <env-id>` | Update an environment's name, URL, or auth settings | `envs_update` |
+| `beryl envs update <env-id>` | Update an environment's name, URL, auth settings, Site Health screen sizes, schedule, CI and pages, or issue tracker settings | `envs_update` |
 | `beryl envs delete <env-id>` | Delete an environment | `envs_delete` |
 
 ### schedule
@@ -223,7 +223,8 @@ Author, inspect, version, and heal a project's tests: the checks Beryl runs on e
 | `beryl tests set-groups <test-id>` | Replace the groups a test belongs to | `tests_set_groups` |
 | `beryl tests set-access <test-id> <access>` | Mark a test read or write | `tests_set_access` |
 | `beryl tests add-groups <test-ids...>` | Add groups to one or more tests, keeping the groups they already have | `tests_add_groups` |
-| `beryl tests quarantine <test-id> <state>` | Mute a flaky test: it keeps running, but its failures stop failing the run | `tests_quarantine` |
+| `beryl tests quarantine <state> <test-ids...>` | Mute flaky tests: they keep running, but their failures stop failing the run | `tests_quarantine` |
+| `beryl tests events <test-id>` | Who quarantined, grouped or renamed a test, and why | `tests_events` |
 | `beryl tests delete <test-ids...>` | Delete tests, their version history, and their results | `tests_delete` |
 | `beryl tests recompile <test-id>` | Validate + verify an edited plan against the live site before persisting | `tests_recompile` |
 | `beryl tests versions <test-id>` | List a test's version history | `tests_versions` |
@@ -269,14 +270,21 @@ Trigger a run of a project's tests (e.g. in CI), then watch, inspect, and downlo
 
 ### health
 
-Site Health, or how your site reads to search engines and visitors: content, speed, mobile, links and security, graded from a real check of your live pages.
+Site Health, or how your site reads to search engines and visitors: content, speed, links, security, and a visual check of layout at phone, tablet and desktop sizes, graded from a real check of your live pages.
 
 `beryl health` with no subcommand runs `health get`.
 
 | Command | Summary | MCP tool |
 | --- | --- | --- |
-| `beryl health get` | Show the latest Site Health report for a project environment | `health_get` |
-| `beryl health run` | Run a fresh Site Health check for a project environment | `health_run` |
+| `beryl health get` | Show the latest Site Health report, or one report by id | `health_get` |
+| `beryl health run` | Run a fresh Site Health check, or a visual check of one page | `health_run` |
+| `beryl health intended <key>` | Mark a visual finding as intended, so it stops counting against the grade | `health_intended` |
+| `beryl health tracker status` | Show how Site Health files findings as issue tracker tickets for an environment | `health_tracker_status` |
+| `beryl health tracker tickets` | List the findings Site Health tracks in the issue tracker, with their tickets | `health_tracker_tickets` |
+| `beryl health tracker connect` | Get the link that connects an issue tracker to the workspace | `health_tracker_connect` |
+| `beryl health tracker disconnect` | Remove the workspace's issue tracker connection | `health_tracker_disconnect` |
+| `beryl health tracker teams` | List the teams of the connected issue tracker | `health_tracker_teams` |
+| `beryl health tracker projects` | List a tracker team's projects | `health_tracker_projects` |
 
 ### explorations
 
@@ -326,6 +334,17 @@ The project's standing email addresses, where its tests receive sign-in mail.
 | `beryl mailbox read <mailbox-id>` | Read the latest email in a mailbox (waits for one to arrive) | `mailbox_read` |
 | `beryl mailbox emails <mailbox-id>` | List the emails a mailbox has received | `mailbox_emails` |
 
+### third-party-integrations
+
+Test a site's own integrations (Connect Slack) against Beryl's accounts on those providers.
+
+`beryl third-party-integrations` with no subcommand runs `third-party-integrations list`.
+
+| Command | Summary | MCP tool |
+| --- | --- | --- |
+| `beryl third-party-integrations list` | The third party providers this project's tests can connect to | `third_party_integrations_list` |
+| `beryl third-party-integrations target` | The channel this project's tests pick when the app asks where to post | `third_party_integrations_target` |
+
 ### accounts
 
 Durable identities on the site under test: what an authenticated test signs in as.

package/dist/beryl-test-skill.js

@@ -551,4 +551,31 @@ against your test's **intent**:
 So a test is *healable* exactly when it has **a strong intent + a real outcome
 assertion**. With a vague intent and a chrome-only assertion Beryl can't tell
 regression from cosmetic drift — it either heals over real breakage or fails on noise.
+
+## 7. Third party integration tests
+
+An app's own integrations (its "Connect Slack" button, the message it then posts) are
+testable against Beryl's account on that provider. Call
+\`third_party_integrations_list\` first; if the provider is not listed, do not write the
+test.
+
+Two shapes:
+
+- **Connect from scratch**: declare \`"third_party_integrations": ["slack"]\`, click the
+  app's Connect button, then \`{ "action": "third_party_authorize", "provider": "slack" }\`
+  approves the consent page and returns to the app. Pick the channel
+  \`third_party_integrations_target\` names, trigger the app's message, then
+  \`{ "action": "await_third_party", "provider": "slack", "expect_text": "..." }\`.
+- **Integration intact**: no declaration. Trigger the app's message, then
+  \`await_third_party\`.
+
+Rules:
+
+- Always tear down what the test connected in \`after\`: disconnect the integration in
+  the app.
+- These steps run only in Beryl's cloud: \`tests_create\` verifies server-side and
+  \`runs_local\` skips them, so run them with \`runs_trigger\`.
+- A setup error saying Beryl's access "is being restored" is on Beryl's side: do not
+  edit the test.
+- Never add steps that browse Slack itself; the actions above do all of it.
 `;

package/dist/commands/environments.js

@@ -1,5 +1,7 @@
 import { UsageError } from "../errors.js";
-import { arg, findGroup, flagBool, flagNum, flagStr, projectPath } from "./util.js";
+import { HEALTH_RUN_SETTING_FLAGS, VIEWPORTS, runSettingsFromFlags } from "./health.js";
+import { TRACKER_SETTING_FLAGS, trackerSettingsFromFlags } from "./tracker.js";
+import { arg, findGroup, flagBool, flagNum, flagStr, flagStrings, projectPath } from "./util.js";
 const SCHEDULE_GROUPS_FLAG = "Comma-separated group names or ids the schedule targets (the run covers the union of their active tests at fire time); omit to run every active test";
 const SCHEDULE_CADENCE_FLAGS = [
     {
@@ -65,13 +67,30 @@ export const environmentCommands = [
     },
     {
         name: "envs update",
-        summary: "Update an environment's name, URL, or auth settings",
+        summary: "Update an environment's name, URL, auth settings, Site Health screen sizes, schedule, CI and pages, or issue tracker settings",
         scope: "project",
         args: [{ name: "env-id", description: "Environment id", required: true }],
         flags: [
             { name: "name", type: "string", description: "New name" },
             { name: "url", type: "string", description: "New root URL" },
             { name: "auth", type: "string", enum: ["public", "gated"], description: "New auth choice" },
+            {
+                name: "viewport",
+                type: "strings",
+                enum: ["all", ...VIEWPORTS],
+                description: "Screen size the Site Health visual checks run at, repeatable; replaces the set. " +
+                    "Pass `--viewport all` to reset to all five.",
+            },
+            ...HEALTH_RUN_SETTING_FLAGS,
+            ...TRACKER_SETTING_FLAGS,
+        ],
+        examples: [
+            "beryl envs update <env-id> --tracker linear --tracker-team <team-id>",
+            "beryl envs update <env-id> --tracker-threshold medium --tracker-max-tickets 5",
+            "beryl envs update <env-id> --tracker-priority high=urgent --tracker-pause on",
+            "beryl envs update <env-id> --health-schedule daily --health-time 06:00 --health-timezone Europe/Berlin",
+            "beryl envs update <env-id> --health-pages crawl --health-max-pages 25 --health-exclude '/blog/*'",
+            "beryl envs update <env-id> --health-ci main_only --health-ci-fail-on medium",
         ],
         async run(ctx, input) {
             const { workspaceId, projectId } = await ctx.requireProject(input);
@@ -82,6 +101,20 @@ export const environmentCommands = [
                 body.root_url = flagStr(input, "url");
             if (flagStr(input, "auth"))
                 body.requires_auth_choice = flagStr(input, "auth");
+            const viewports = flagStrings(input, "viewport");
+            if (viewports) {
+                const bad = viewports.find((v) => v !== "all" && !VIEWPORTS.includes(v));
+                if (bad) {
+                    throw new UsageError(`--viewport must be one of: all, ${VIEWPORTS.join(", ")}`);
+                }
+                body.site_health_viewports = viewports.includes("all") ? [] : viewports;
+            }
+            const tracker = trackerSettingsFromFlags(input);
+            if (tracker)
+                body.site_health_tracker = tracker;
+            const runs = runSettingsFromFlags(input);
+            if (runs)
+                body.site_health_runs = runs;
             if (Object.keys(body).length === 0)
                 throw new UsageError("Nothing to update");
             return {

package/dist/commands/health.js

@@ -1,43 +1,404 @@
-import { flagStr, projectPath } from "./util.js";
+import { UsageError } from "../errors.js";
+import { arg, flagBool, flagNum, flagStr, flagStrings, projectPath } from "./util.js";
 const ENV_FLAG = {
     name: "env",
     type: "string",
     description: "Environment id (defaults to the project's default environment)",
 };
+export const VIEWPORTS = [
+    "phone",
+    "tablet_portrait",
+    "tablet_landscape",
+    "desktop",
+    "desktop_wide",
+];
+const SCREENSHOTS_FLAG = {
+    name: "screenshots",
+    type: "boolean",
+    description: "Include the annotated screenshots (as images over MCP, data URIs in --json)",
+};
+// Annotated evidence is spent from the agent's context window; the top few carry it.
+const MAX_MCP_IMAGES = 3;
+const WAIT_POLL_MS = 5000;
+const WAIT_LIMIT_MS = 6 * 60 * 1000;
+function toImage(dataUri) {
+    const [, mimeType, data] = /^data:([^;]+);base64,(.*)$/.exec(dataUri) ?? [];
+    return mimeType && data ? { data, mimeType } : undefined;
+}
+// Over MCP the screenshots go out as image blocks and never as inline base64 text.
+function present(ctx, report, withShots) {
+    if (!withShots || !ctx.mcp)
+        return { data: report };
+    const images = [];
+    for (const cat of report.categories ?? []) {
+        for (const f of cat.findings ?? []) {
+            if (f.screenshot && !f.intended && f.status !== "pass" && images.length < MAX_MCP_IMAGES) {
+                const img = toImage(f.screenshot);
+                if (img)
+                    images.push(img);
+            }
+            delete f.screenshot;
+        }
+    }
+    const stripped = { ...report, screenshots: null };
+    return { data: stripped, images: images.length ? images : undefined };
+}
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+// A run fans out to up to 100 pages, each on its own worker.
+const CI_WAIT_LIMIT_MS = 30 * 60 * 1000;
+const DAYS = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"];
+const ON_OFF = ["on", "off"];
+/** `envs update` flags for when Site Health runs and which pages it checks. */
+export const HEALTH_RUN_SETTING_FLAGS = [
+    {
+        name: "health-schedule",
+        type: "string",
+        enum: ["off", "daily", "weekly"],
+        description: "Run Site Health on a schedule (default off)",
+    },
+    {
+        name: "health-time",
+        type: "string",
+        placeholder: "HH:MM",
+        description: "Time of day scheduled runs start, 24h, in --health-timezone (default 09:00)",
+    },
+    {
+        name: "health-day",
+        type: "string",
+        enum: DAYS,
+        description: "Day weekly runs start (default mon)",
+    },
+    {
+        name: "health-timezone",
+        type: "string",
+        description: "IANA timezone for the schedule, e.g. Europe/Berlin (default UTC)",
+    },
+    {
+        name: "health-ci",
+        type: "string",
+        enum: ["off", "every_build", "main_only", "deploy_only"],
+        description: "Which CI builds `health run --ci` checks; other builds skip and pass (default off)",
+    },
+    {
+        name: "health-ci-fail-on",
+        type: "string",
+        enum: ["high", "medium", "never"],
+        description: "Lowest finding severity that fails the CI build; never reports only (default high)",
+    },
+    {
+        name: "health-ci-tickets",
+        type: "string",
+        enum: ON_OFF,
+        description: "CI runs file tracker tickets; off leaves the findings to the PR (default off)",
+    },
+    {
+        name: "health-pages",
+        type: "string",
+        enum: ["homepage", "paths", "sitemap", "crawl"],
+        description: "Pages a run checks: the homepage, --health-path list, sitemap.xml, or a crawl from " +
+            "the homepage (default homepage)",
+    },
+    {
+        name: "health-path",
+        type: "strings",
+        description: "Page path for the paths mode, repeatable; replaces the list. `none` clears it",
+    },
+    {
+        name: "health-crawl-depth",
+        type: "number",
+        description: "Link hops from the homepage in crawl mode, 1-5 (default 2)",
+    },
+    {
+        name: "health-max-pages",
+        type: "number",
+        description: "Most pages one run checks, homepage included, 1-100 (default 25)",
+    },
+    {
+        name: "health-include",
+        type: "strings",
+        description: "Path pattern a page must match, e.g. '/blog/*', repeatable; replaces the list. `none` clears it",
+    },
+    {
+        name: "health-exclude",
+        type: "strings",
+        description: "Path pattern to skip, repeatable; replaces the list. `none` clears it",
+    },
+    {
+        name: "health-logged-in",
+        type: "strings",
+        description: "Path pattern checked signed in with the environment's saved test sign-in, repeatable; " +
+            "the rest run logged out. `none` clears it",
+    },
+    {
+        name: "health-retries",
+        type: "number",
+        description: "Extra attempts when a page check fails or times out, 0-3 (default 1)",
+    },
+];
+function list(input, name) {
+    const values = flagStrings(input, name);
+    if (!values)
+        return undefined;
+    return values.includes("none") ? [] : values;
+}
+/** The `site_health_runs` PATCH body from `envs update` flags; undefined when none were given. */
+export function runSettingsFromFlags(input) {
+    const body = {};
+    const strings = [
+        ["health-schedule", "schedule"],
+        ["health-timezone", "schedule_timezone"],
+        ["health-ci", "ci_mode"],
+        ["health-ci-fail-on", "ci_fail_on"],
+        ["health-pages", "pages"],
+    ];
+    for (const [flag, field] of strings) {
+        const v = flagStr(input, flag);
+        if (v !== undefined)
+            body[field] = v;
+    }
+    const time = flagStr(input, "health-time");
+    if (time !== undefined) {
+        const m = /^(\d{1,2}):(\d{2})$/.exec(time);
+        if (!m || Number(m[1]) > 23 || Number(m[2]) > 59) {
+            throw new UsageError("--health-time must be HH:MM, 24h, e.g. 09:30");
+        }
+        body.schedule_hour = Number(m[1]);
+        body.schedule_minute = Number(m[2]);
+    }
+    const day = flagStr(input, "health-day");
+    if (day !== undefined) {
+        const idx = DAYS.indexOf(day.toLowerCase());
+        if (idx < 0)
+            throw new UsageError(`--health-day must be one of: ${DAYS.join(", ")}`);
+        body.schedule_day_of_week = idx;
+    }
+    const tickets = flagStr(input, "health-ci-tickets");
+    if (tickets !== undefined) {
+        if (!ON_OFF.includes(tickets))
+            throw new UsageError("--health-ci-tickets must be on or off");
+        body.ci_files_tickets = tickets === "on";
+    }
+    const lists = [
+        ["health-path", "paths"],
+        ["health-include", "include"],
+        ["health-exclude", "exclude"],
+        ["health-logged-in", "logged_in"],
+    ];
+    for (const [flag, field] of lists) {
+        const v = list(input, flag);
+        if (v !== undefined)
+            body[field] = v;
+    }
+    const numbers = [
+        ["health-crawl-depth", "crawl_depth"],
+        ["health-max-pages", "max_pages"],
+        ["health-retries", "retries"],
+    ];
+    for (const [flag, field] of numbers) {
+        const v = flagNum(input, flag);
+        if (v !== undefined)
+            body[field] = v;
+    }
+    return Object.keys(body).length ? body : undefined;
+}
+function ciBranch() {
+    const env = process.env;
+    return (env.GITHUB_HEAD_REF ||
+        env.GITHUB_REF_NAME ||
+        env.CI_COMMIT_REF_NAME ||
+        env.CIRCLE_BRANCH ||
+        env.BITBUCKET_BRANCH ||
+        env.BRANCH_NAME ||
+        undefined);
+}
+// Markdown, so a CI step can post it as the PR comment as is.
+export function ciMarkdown(run) {
+    if (run.skipped)
+        return `Beryl Site Health: skipped (${run.reason ?? "CI mode"}).`;
+    const f = run.findings ?? {};
+    const verdict = run.status === "failed"
+        ? "failed to complete"
+        : run.gate?.passed
+            ? "passed"
+            : `failed (findings at or above ${run.gate?.fail_on})`;
+    const lines = [
+        `**Beryl Site Health ${verdict}**`,
+        "",
+        `${run.pages?.length ?? 0} page(s) checked. Open findings: ${f.high ?? 0} high, ` +
+            `${f.medium ?? 0} medium, ${f.low ?? 0} low.`,
+    ];
+    if (run.failing?.length) {
+        lines.push("", "| Page | Severity | Finding |", "| --- | --- | --- |");
+        for (const x of run.failing)
+            lines.push(`| ${x.page_path} | ${x.severity} | ${x.title} |`);
+    }
+    const broken = (run.pages ?? []).filter((p) => p.status === "failed" || p.problem);
+    if (broken.length) {
+        lines.push("", "Pages that could not be checked:");
+        for (const p of broken) {
+            lines.push(`- ${p.page_path}: ${p.problem ?? p.error ?? "check failed"}`);
+        }
+    }
+    return lines.join("\n");
+}
+async function runCi(ctx, input, base) {
+    let run = (await ctx.client.post(`${base}/runs`, {
+        trigger: "ci",
+        branch: flagStr(input, "branch") ?? ciBranch(),
+        deploy: flagBool(input, "deploy"),
+    }, { environment_id: flagStr(input, "env") }));
+    if (run.skipped)
+        return { data: run, human: ciMarkdown(run) };
+    const deadline = Date.now() + CI_WAIT_LIMIT_MS;
+    while (run.status === "queued" || run.status === "running") {
+        if (Date.now() > deadline) {
+            return { data: run, human: "Beryl Site Health: timed out waiting for the run.", exitCode: 1 };
+        }
+        await sleep(WAIT_POLL_MS);
+        run = (await ctx.client.get(`${base}/runs/${run.run_id}`));
+    }
+    return { data: run, human: ciMarkdown(run), exitCode: run.gate?.passed ? 0 : 1 };
+}
 export const healthCommands = [
     {
         name: "health get",
         groupDefault: true,
         groupSummary: "Site Health, or how your site reads to search engines and visitors: content, " +
-            "speed, mobile, links and security, graded from a real check of your live pages.",
-        summary: "Show the latest Site Health report for a project environment",
+            "speed, links, security, and a visual check of layout at phone, tablet and desktop " +
+            "sizes, graded from a real check of your live pages.",
+        summary: "Show the latest Site Health report, or one report by id",
         description: "A check runs automatically when a project or environment gets its URL. While one " +
-            "is in flight the report comes back with status queued/running and no grades yet; " +
-            "call again to pick up the finished result.",
+            "is in flight the report comes back with status queued/running and no grades yet, " +
+            "plus the `stage` a running check is on; call again to pick up the finished result. Visual findings sit in the `layout` " +
+            "category, which also grades them per screen size (`screens`: phone, tablet, " +
+            "desktop) and as `mobile` (phone and tablet only). Each carries a `key`, the " +
+            "`selector`, measured values, the `viewports` it shows at and a fix hint in " +
+            "`recommendation`. The `performance` category grades Google PageSpeed on mobile " +
+            "and on desktop separately (`speeds`, each with its own Core Web Vitals) and takes " +
+            "the worse of the two; one that couldn't be measured is left out. Pass --report for a single-page check started with " +
+            "`health run --path`.",
         scope: "project",
-        flags: [ENV_FLAG],
+        flags: [
+            ENV_FLAG,
+            { name: "report", type: "string", description: "Report id (from `health run`)" },
+            {
+                name: "run",
+                type: "string",
+                description: "Run id: every page of one run, finding counts and the CI gate verdict",
+            },
+            SCREENSHOTS_FLAG,
+        ],
         async run(ctx, input) {
             const { workspaceId, projectId } = await ctx.requireProject(input);
-            return {
-                data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/seo-report`, {
-                    environment_id: flagStr(input, "env"),
-                }),
-            };
+            const runId = flagStr(input, "run");
+            if (runId) {
+                return {
+                    data: await ctx.client.get(`${projectPath(workspaceId, projectId)}/seo-report/runs/${runId}`),
+                };
+            }
+            const withShots = flagBool(input, "screenshots");
+            const reportId = flagStr(input, "report");
+            const base = `${projectPath(workspaceId, projectId)}/seo-report`;
+            const report = (await ctx.client.get(reportId ? `${base}/${reportId}` : base, {
+                environment_id: reportId ? undefined : flagStr(input, "env"),
+                screenshots: withShots ? "true" : "false",
+            }));
+            return present(ctx, report, withShots);
         },
     },
     {
         name: "health run",
-        summary: "Run a fresh Site Health check for a project environment",
-        description: "Queues a new check and returns immediately. Returns the in-flight report instead " +
-            "of stacking a second one when a check is already running.",
+        summary: "Run a fresh Site Health check, or a visual check of one page",
+        description: "Queues a check and returns immediately (or, with --wait, once it finishes; a check " +
+            "takes one to three minutes). Without --path it is the full Site Health run of the " +
+            "environment URL. With --path it runs only the visual checks (clipping, overlap, tap " +
+            "targets, scroll traps, content stability, scroll-through) on that page, logged out " +
+            "unless it matches the environment's --health-logged-in patterns, " +
+            "at the chosen --viewport sizes (default: the environment's configured set). Returns " +
+            "the in-flight report instead of stacking a second one of the same scope. A full run " +
+            "also checks the environment's configured pages (see `envs update --health-pages`); " +
+            "each page gets its own report, grouped under the homepage report's `run_id`.\n\n" +
+            "With --ci it starts a CI run, waits for every page, prints a Markdown summary you can " +
+            "post as the PR comment, and exits 1 when findings at or above the environment's " +
+            "fail-on severity exist, a page couldn't be checked, or the run failed. Builds that the " +
+            "environment's CI mode doesn't cover skip and exit 0. CI runs file tracker tickets only when --health-ci-tickets is on.",
+        scope: "project",
+        flags: [
+            ENV_FLAG,
+            { name: "path", type: "string", description: "Page path to check, e.g. /pricing" },
+            {
+                name: "viewport",
+                type: "strings",
+                enum: VIEWPORTS,
+                description: "Screen size, repeatable: phone (390x844), tablet_portrait (768x1024), " +
+                    "tablet_landscape (1024x768), desktop (1440x900), desktop_wide (1920x1080)",
+            },
+            { name: "wait", type: "boolean", description: "Wait for the check to finish" },
+            SCREENSHOTS_FLAG,
+            {
+                name: "ci",
+                type: "boolean",
+                description: "CI run: wait, print a PR-ready summary, exit 1 at or above the fail-on severity",
+            },
+            {
+                name: "branch",
+                type: "string",
+                description: "Branch of the CI build (default: read from the CI environment)",
+            },
+            { name: "deploy", type: "boolean", description: "The CI build is a deploy (for --health-ci deploy_only)" },
+        ],
+        examples: [
+            "beryl health run",
+            "beryl health run --path /pricing --viewport phone --wait",
+            "beryl health run --ci",
+            "beryl health run --ci --deploy --env <env-id>",
+        ],
+        async run(ctx, input) {
+            const { workspaceId, projectId } = await ctx.requireProject(input);
+            const base = `${projectPath(workspaceId, projectId)}/seo-report`;
+            const path = flagStr(input, "path");
+            const viewports = flagStrings(input, "viewport");
+            if (flagBool(input, "ci")) {
+                if (path || viewports)
+                    throw new UsageError("--ci runs the configured pages; drop --path and --viewport");
+                return runCi(ctx, input, base);
+            }
+            const bad = viewports?.find((v) => !VIEWPORTS.includes(v));
+            if (bad)
+                throw new UsageError(`--viewport must be one of: ${VIEWPORTS.join(", ")}`);
+            const body = path || viewports ? { path, viewports } : undefined;
+            let report = (await ctx.client.post(base, body, {
+                environment_id: flagStr(input, "env"),
+            }));
+            const withShots = flagBool(input, "screenshots");
+            if (!flagBool(input, "wait"))
+                return { data: report };
+            const deadline = Date.now() + WAIT_LIMIT_MS;
+            while (report.status === "queued" || report.status === "running") {
+                if (Date.now() > deadline)
+                    return { data: report, exitCode: 1 };
+                await sleep(WAIT_POLL_MS);
+                report = (await ctx.client.get(`${base}/${report.id}`, {
+                    screenshots: withShots ? "true" : "false",
+                }));
+            }
+            const result = present(ctx, report, withShots);
+            return report.status === "failed" ? { ...result, exitCode: 1 } : result;
+        },
+    },
+    {
+        name: "health intended",
+        summary: "Mark a visual finding as intended, so it stops counting against the grade",
+        description: "Takes the finding's `key` from `health get`. The finding stays in the report, flagged " +
+            "intended, and is left out of the grade on this report and every later run of the " +
+            "environment. --clear removes the mark.",
         scope: "project",
-        flags: [ENV_FLAG],
-        examples: ["beryl health run", "beryl health run --env 4f…"],
+        args: [{ name: "key", description: "Finding key", required: true }],
+        flags: [ENV_FLAG, { name: "clear", type: "boolean", description: "Remove the mark" }],
         async run(ctx, input) {
             const { workspaceId, projectId } = await ctx.requireProject(input);
-            return {
-                data: await ctx.client.post(`${projectPath(workspaceId, projectId)}/seo-report`, undefined, { environment_id: flagStr(input, "env") }),
-            };
+            const report = (await ctx.client.put(`${projectPath(workspaceId, projectId)}/seo-report/intended`, { key: arg(input, "key"), intended: !flagBool(input, "clear") }, { environment_id: flagStr(input, "env") }));
+            return present(ctx, report, false);
         },
     },
 ];

package/dist/commands/runs.js

@@ -2,7 +2,7 @@ import fs from "node:fs";
 import path from "node:path";
 import { downloadRunArtifacts, failureImages, isFailing, resultsOf, } from "../artifacts.js";
 import { CliError, UsageError } from "../errors.js";
-import { buildImportForm, establishAccountSession, executeLocalSpec, IMPORT_MAX_ERROR_LEN, toRunEntry, } from "../local-exec.js";
+import { buildImportForm, establishAccountSession, executeLocalSpec, IMPORT_MAX_ERROR_LEN, specUsesThirdParty, THIRD_PARTY_USE_TRIGGER, toRunEntry, } from "../local-exec.js";
 import { countPlannedFrames, countWrittenFrames, PlaywrightMissingError, } from "../local-run.js";
 import { dim, green, red, yellow } from "../output.js";
 import { anyGap, confirmInstall, describeGaps, installCommandsFor, installHint, installPlaywright, playwrightGaps, } from "../playwright-install.js";
@@ -152,7 +152,8 @@ export const runCommands = [
             "in the cloud: their account signs in once per invocation and every session-mode " +
             "test rides that session; a failed sign-in fails those tests with the same " +
             "SESSION_* reason a cloud run reports. Only a test that depends on a session Beryl " +
-            "captured server-side is skipped, with a note. Run those with `runs trigger`. Point " +
+            "captured server-side, or that has third party integration steps (cloud only), is " +
+            "skipped, with a note. Run those with `runs trigger`. Point " +
             "--url-override at a local dev server or preview, and --dir to keep specs, artifacts, " +
             "and reports on disk. Exits 0 only if every executed test passed.",
         scope: "project",
@@ -255,6 +256,7 @@ export const runCommands = [
                 requiresAuth: Boolean(script.requires_auth),
                 usesEmail: /__vmInbox\(|__vmAwaitEmail\(/.test(script.content),
                 usesLoginPassword: Boolean(script.uses_login_password),
+                usesThirdParty: specUsesThirdParty(script.content),
                 inlinedLoginSteps: script.inlined_login_steps ?? 0,
                 ...(script.session_mode && script.account_id
                     ? { sessionMode: true, accountId: script.account_id }
@@ -288,13 +290,17 @@ export const runCommands = [
             // a cloud run's setup phase does. A test whose account is missing its password
             // would type a literal placeholder into the page: skip, with the server's fix-it
             // message.
-            const skipped = specs.filter((s) => (s.requiresAuth && !(s.sessionMode && s.accountId)) || s.loginConfigError);
+            const skipped = specs.filter((s) => (s.requiresAuth && !(s.sessionMode && s.accountId)) ||
+                s.loginConfigError ||
+                s.usesThirdParty);
             const runnable = specs.filter((s) => !skipped.includes(s));
             for (const s of skipped) {
                 ctx.err(yellow(s.loginConfigError
                     ? `! ${s.title}: skipped — ${s.loginConfigError}`
-                    : `! ${s.title}: needs a session only Beryl's cloud holds — skipped ` +
-                        `(use \`beryl runs trigger\`).`));
+                    : s.usesThirdParty
+                        ? `! ${s.title}: skipped, ${THIRD_PARTY_USE_TRIGGER}.`
+                        : `! ${s.title}: needs a session only Beryl's cloud holds, skipped ` +
+                            `(use \`beryl runs trigger\`).`));
             }
             if (runnable.length === 0)
                 throw new CliError("None of the selected tests can run locally.");
@@ -585,8 +591,8 @@ export const runCommands = [
             {
                 name: "status",
                 type: "string",
-                description: "With --page, show only this bucket: passed, failed, errored, cancelled, or " +
-                    "quarantined",
+                description: "With --page, show only this bucket: passed, failed, errored, cancelled, " +
+                    "quarantined, or unstable (currently unstable in the run's environment)",
             },
             {
                 name: "group",