@1aboveio/skills 0.10.0

package/skills/engineering/smoke/references/manifest.md

@@ -0,0 +1,192 @@
+# The smoke manifest (curated E2E golden-path format)
+
+The manifest is the curated set of flows whose failure means **do not ship**. It is the gate's input — keep it small, keep it ruthless.
+
+## You author this per app — there is no shared default set
+
+**The manifest is repo-specific, and the skill ships none.** One app's golden paths are wrong for another: a payments app's are checkout → pay → receipt; a docs app's are create → edit → share; an auth product's are sign-in / SSO / signup. A generic "login + dashboard" set gives a false **ALIVE** on an app whose real do-not-ship flow is checkout — which is why the gate takes a manifest instead of hardcoding paths. Whoever knows the product authors it once, with the same judgment that picks `e2e-test`'s Tier-3 journeys, just filtered harder (see [What earns a slot](#what-earns-a-slot)). The [example manifest](../assets/smoke.manifest.example.json) is for one consumer auth product and exists to be **adapted and deleted, not kept** — copying its paths into a different app is the mistake this warns against.
+
+## There is no fixed number of paths
+
+Don't anchor on a count. The right size is **"every flow whose failure alone would block a release, and no more"** — 2 paths or 8. The bar is the filter, not a number. If the set creeps toward dozens, you've stopped curating and started re-deriving `e2e-test`'s journey tier; pull the non-do-not-ship flows back out and let e2e-test cover them.
+
+## Where candidates come from (discover, then curate)
+
+Curation splits in two: **discovering candidates** (automatable) and **deciding the final set** (a product/release call that stays human — a *guessed* manifest gives a false ALIVE). These sources feed discovery, strongest first; an agent drafts, a human promotes.
+
+1. **The `@smoke`-taggable Tier-3 journeys (`e2e-test`).** The best pool, because it's *already curated* — Tier-3 journeys are critical business flows someone judged worth an end-to-end test. Selecting the do-not-ship subset is filtering an existing set, not authoring a new one. For browser paths this is the source; the smoke path just references the journey.
+
+2. **State transitions of core entities (`domain-modeling`).** The most *principled* generator. A core entity's state machine enumerates the transitions the product exists to perform — `Order: cart → placed → paid → fulfilled → refunded`, `User: anonymous → registered → verified → active`, `Subscription: trialing → active → past_due → canceled`. Each **user-driven** transition is a candidate, and the transition hands you the observable for free: *"the entity is now in the target state, and the user can see it."* The transitions that touch money or irreversible state are almost exactly [what earns a slot](#what-earns-a-slot). Walk the two or three entities the product is *about*; skip transitions that are admin-only, automatic/background, or rare-recovery.
+
+3. **ADRs — a ranking signal, not a generator.** ADRs mostly record *build* decisions (a DB, a framework), which aren't journeys, so they don't *enumerate* golden paths. But an ADR pinning a load-bearing invariant ("all payments go through the idempotency ledger," "auth is Feishu SSO") is evidence *that* flow is do-not-ship. Use ADRs to **justify and rank** candidates the first two sources surfaced.
+
+4. **The generated surface inventory (`suggest`) — the crude fallback.** When there's no domain model or journey suite yet, scan routes/APIs mechanically. It's keyword-and-shape scoring (auth/payment-looking routes, mutation routes, health endpoints), blind to product meaning:
+   ```bash
+   node <smoke>/scripts/smoke.mjs suggest \
+     --inventory docs/tests/_generated/surface-inventory.json \
+     --out smoke.candidates.json
+   ```
+   The output uses a `candidates` array, not `paths`, so it is deliberately not runnable — promote by hand.
+
+**Whatever the source, the promotion step is the same:** keep only candidates whose failure *alone* would block a release, write a synthetic `tenant` for any that mutate, and — for browser candidates — make sure the `@smoke` Tier-3 journey they reference exists (author it in `e2e-test` if not). None of these sources decide the set; they stop you starting from memory.
+
+## What earns a slot
+
+A path belongs only if **its failure alone would block a release** — a much higher bar than "has a test." A few **shapes** recur; use them as prompts to find *your* paths, not a set to copy:
+
+- **Get in** — the primary auth a user hits first (sign in; sign up + verify).
+- **The core action** — the one or two things the product exists to do (its revenue / trust journey).
+- **It's up** — a health / liveness baseline.
+- anything touching money, irreversible state, or the thing that pages you at 3am.
+
+The archetype is the question ("what's our *get in*?"); the answer is your app's own routes and observables. Everything else is `e2e-test`'s job (per-route tiers + the presentation sweep over the whole manifest). The smoke set is deliberately *not* a route census — a route that only needs "does it render" is covered by the presentation sweep, not here.
+
+## Format
+
+JSON (`smoke.manifest.json`). A repo may author in YAML and project to JSON — the gate reads JSON, the same way `e2e-test`'s `route-manifest.json` is a generated projection.
+
+```json
+{
+  "profiles": {
+    "pr-local":     { "gate": "pull-request",      "environment": "local", "endpoint": "local",   "prerequisites": ["boot", "seed"], "evidencePolicy": "per-PR screens",             "mutationPolicy": "isolated-writes" },
+    "prod-preview": { "gate": "pre-promotion",     "environment": "prod",  "endpoint": "preview", "prerequisites": ["candidate-revision"], "evidencePolicy": "attested identity tuple", "mutationPolicy": "read-only" }
+  },
+  "paths": [
+    {
+      "id": "login-to-dashboard",
+      "surface": "browser",
+      "journey": "@smoke login-to-dashboard",
+      "mutates": false,
+      "profiles": ["pr-local", "prod-preview"]
+    },
+    {
+      "id": "health",
+      "surface": "api",
+      "start": "GET /healthz",
+      "observable": "200 with body { status: \"ok\" }",
+      "mutates": false,
+      "profiles": ["pr-local", "prod-preview"]
+    }
+  ]
+}
+```
+
+### Fields
+
+| Field | Required | Meaning |
+|---|---|---|
+| `id` | yes | unique, stable slug — the drive step reports results keyed by it |
+| `surface` | yes | `browser` \| `api` \| `cli` — picks the driver (see below) |
+| `journey` | **browser only** | the `@smoke`-tagged e2e Tier-3 journey (by its `@covers` id) that **drives** this path. A browser path *references* the journey — it does not re-specify `start`/`observable`; the spec owns the steps and asserted observable. The flow is authored once, in e2e-test. |
+| `start` | api/cli | where the path begins: a request (`api`, e.g. `GET /healthz`). `cli` uses `command` instead. Not used by `browser` (the journey owns it). |
+| `command` | cli | the command to run, for `cli` paths (alternative to `start`) |
+| `observable` | api/cli | the **concrete thing that proves the path worked** — the 200 body, the exit code. Required for `api`/`cli` (e2e-test doesn't cover them); for `browser` the referenced journey's assertions *are* the observable. |
+| `intent` | no | optional one-line human summary of what the browser journey proves — documentation only, not enforced. |
+| `mutates` | no | `true` if the path writes durable state. Forces a `tenant` and gates it out of read-only profiles. Under a **read-only profile** (any production profile), a tenant grants nothing: every `mutates:true` path is excluded regardless (`resolve-release` principle 4). |
+| `tenant` | when `mutates` | a **synthetic/test tenant** id so the mutating path can run under a `synthetic-tenant` profile (e.g. `dev-active`) without touching real data. A `mutates:true` path with no `tenant` is excluded from a `synthetic-tenant` profile. This is **not** a production permit: the production database is real whether or not the revision serves traffic, so the read-only production profiles exclude the path either way. Name it with the **`e2e-` prefix** (`e2e-synthetic`) per the e2e-* account convention (`e2e-test` → `references/authoring/auth-flows.md`) — `validate` warns on an unprefixed tenant, because that's the one mistaken for live data. |
+| `profiles` | **when a `profiles` registry exists** | the [Validation Profiles](#validation-profiles-the-profiles-registry) this path is required in — a non-empty array of names from the registry. A path is only ever selected in a profile it declares; a path that declares none is a silent non-runner, so `validate` **fails closed** on it once a registry is present. (A per-path array, rather than the inverse profile→paths mapping, so a path's selection facts — `mutates`, `tenant`, `profiles` — all sit together on the path they govern.) |
+
+For `api`/`cli`, `observable` is the load-bearing field — what separates this gate from a vacuous one. "Renders" is too weak (the #237 button *rendered* — it was just invisible); write the perceivable, capturable fact: *"200 with body `{status:ok}`"*, *"exit 0 and the migration count printed."* For `browser`, that discipline lives in the journey spec's assertions — smoke selects the journey, doesn't restate it.
+
+## Validation Profiles (the `profiles` registry)
+
+**Selection is by a named profile, not a caller-assembled target.** A run picks one Validation Profile (`--profile pr-local`), and that name decides both *which* paths run and *whether* writes are allowed — so the run and the verdict, driven by the same name, cannot disagree about the selected set. This replaces the free-form `--target`/`--no-mutations` pair, whose two independent assemblies could diverge and false-DEAD a healthy release (`resolve-release` #440). `--target` remains, but only as the **address** to drive against (a candidate/service URL); it no longer participates in selection.
+
+A profile binds **three axes plus policy**:
+
+| Key | Meaning | Values |
+|---|---|---|
+| `gate` | **why** evidence is required — the lifecycle moment | `pull-request` \| `merge-queue` \| `trunk-integration` \| `pre-promotion` \| `post-promotion` |
+| `environment` | **where** the app runs | `local` \| `dev` \| `prod` |
+| `endpoint` | **which** revision (a split that only exists once deployed) | `local` \| `preview` \| `active` |
+| `prerequisites` | the set the profile requires in place before it can run | array of strings |
+| `evidencePolicy` | what evidence the profile demands | string |
+| `mutationPolicy` | whether (and how) writes are allowed — a property of the **profile**, not the invocation | `isolated-writes` \| `synthetic-tenant` \| `read-only` |
+
+**`validate` fails closed on an invalid binding.** The gate pins exactly one valid `(environment, endpoint)` per `gate` — the PR/queue gates run `local`/`local`; `trunk-integration` runs `dev`/`active` (dev auto-deploys at 100% traffic); `pre-promotion` is the production candidate at 0% traffic, `prod`/`preview`; `post-promotion` is the live production service, `prod`/`active`. Any other pairing (e.g. `pull-request` in `prod`, or `pre-promotion` against `active`) is rejected. And **production is mechanically read-only**: a `prod` profile with any `mutationPolicy` other than `read-only` fails — that is principle 4 enforced by the profile, not by a `--no-mutations` flag the caller must remember. `isolated-writes` is local-only; `synthetic-tenant` is dev-only.
+
+**How `mutationPolicy` selects a `mutates:true` path:**
+
+- `read-only` (the production profiles) — **excluded, tenant or not.** The candidate/service is wired to the real production database; a synthetic tenant grants nothing, and a revision at 0% traffic is not a reason a write is safe.
+- `synthetic-tenant` (`dev-active`) — runs **only if the path declares a synthetic `tenant`**; a tenant-less mutating path is excluded.
+- `isolated-writes` (the local profiles) — runs (a disposable local store).
+
+Read-only paths always run in any profile they declare.
+
+### The five initial profiles
+
+| Name | gate | environment | endpoint | mutationPolicy |
+|---|---|---|---|---|
+| `pr-local` | `pull-request` | `local` | `local` | `isolated-writes` |
+| `queue-local` | `merge-queue` | `local` | `local` | `isolated-writes` |
+| `dev-active` | `trunk-integration` | `dev` | `active` | `synthetic-tenant` |
+| `prod-preview` | `pre-promotion` | `prod` | `preview` | `read-only` |
+| `prod-active` | `post-promotion` | `prod` | `active` | `read-only` |
+
+`resolve-release` consumes `prod-preview` at step 4a (validate the candidate at 0% traffic) and `prod-active` at step 4b (re-validate the live service after the shift). Each smoke **path** declares the profiles it is required in (`"profiles": ["pr-local", "prod-preview"]`); a path selected by zero eligible paths under a profile yields the existing **EMPTY** verdict — a gate that proves nothing must not look like a pass.
+
+**Recommended, not mandatory (for now).** A manifest with no `profiles` registry still `validate`s (legacy manifests don't break) but `validate` warns, and `plan`/`verdict --profile` cannot run against it. The moment a registry is present it is fully checked, and every path must declare its profiles.
+
+**Reserved names are pinned to their gate.** The five shipped names are consumed by `resolve-release` (and the other lanes) by **hardcoded string** — step 4a drives `--profile prod-preview`, 4b drives `prod-active` — so the name is itself a safety key, not just a label. `validate` therefore **pins each reserved name to its gate**: `pr-local`→`pull-request`, `queue-local`→`merge-queue`, `dev-active`→`trunk-integration`, `prod-preview`→`pre-promotion`, `prod-active`→`post-promotion`. From the gate, the `(environment, endpoint)` table and the prod-read-only rule force the rest — so an entry *named* `prod-preview` but *bound* to a writable dev gate **fails closed**, instead of letting `plan --profile prod-preview` inherit a `synthetic-tenant` write policy against the production candidate.
+
+| Hazard | Failure mode | What to prove |
+|---|---|---|
+| **prod-profile-name-drift** | A safety property keyed to a NAME (profile, tenant, env) consumed by another skill/lane, where the name→binding map is repo-authored data no validator pins — rename or rebind and the consumer silently inherits the wrong policy. | Validate the reserved names' bindings, or select by the semantic key (gate), never the label. |
+
+## The `setup` block (prerequisites — co-located with the gate)
+
+A smoke run needs things in place *before* it can drive a path: how to boot, env/secrets, an auth-bypass, seed data, synthetic test tenants. These belong **in the manifest, next to the paths** — not in `AGENTS.md` or scattered prose. Co-locating means a run loads one file and sees the whole picture, and `validate` can *check* the prerequisites are declared instead of trusting prose that drifts.
+
+Add an optional `setup` object above `paths`:
+
+```json
+{
+  "setup": {
+    "boot": "use the run skill (else: pnpm --filter app dev)",
+    "env": ["DATABASE_URL", "AUTH_SECRET"],
+    "authBypass": "E2E_AUTH_BYPASS=1 (see e2e-test global-setup)",
+    "seed": "pnpm seed:smoke",
+    "fixtures": "reuse e2e-test's Playwright fixtures — do not recreate auth here",
+    "tenants": ["e2e-synthetic"]
+  },
+  "paths": [ ... ]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `boot` | how to start the app for a `local` run — a command, or `"use the run skill"`. The one field `validate` insists on when a `setup` block is present (a run that can't boot can't gate). |
+| `env` | env vars / secrets that must be set to boot (names, not values). |
+| `authBypass` | the mechanism for logging in without a real credential (e.g. an `E2E_AUTH_BYPASS` flag) — usually the same one `e2e-test` uses. |
+| `seed` | a command that loads the data the golden paths assume. |
+| `fixtures` | a **pointer** to where the browser fixtures live (the `e2e-test` Playwright harness). Point, don't recopy — duplicated auth setup is the thing that drifts. |
+| `tenants` | the synthetic/test tenants the mutating paths use. Must cover every `tenant` a path declares, or `validate` warns the registry is incomplete. Entries carry the **`e2e-` prefix** (`validate` warns otherwise). |
+
+**Reuse, don't restate.** Most of the auth/seed prerequisite already lives in `e2e-test`'s harness (global-setup, fixtures, `E2E_AUTH_BYPASS`) and the boot path in the `run` skill. The `setup` block should *reference* those and only spell out what's genuinely smoke-specific — the env, the tenants, the seed.
+
+**Recommended, not mandatory.** A manifest with no `setup` block still validates (legacy manifests don't break), but `validate` warns, because then the prerequisites live nowhere checkable. A present block is shape-checked, and any path tenant the `tenants` registry omits is flagged. Keep a one-line pointer in `AGENTS.md` ("smoke prerequisites live in the manifest's `setup` block"); keep the details here.
+
+## Surface → driver
+
+The manifest declares **what**; the drive step picks **how**, by surface:
+
+- **`browser`** → **don't re-author the flow here — run the referenced `@smoke` Tier-3 journey** through `e2e-test`'s Playwright runner. The `journey` field points at that spec (by its `@covers` id); the spec owns the steps, assertions, and render-health checks (no `pageerror` / `console.error` / error-boundary / 5xx). Smoke adds only *selection* (this journey is do-not-ship), the *binary verdict*, and *target-safety* (`mutates`/`tenant`). The spec holds the real-render discipline: no first-party interception — stub only true external boundaries (a CDN SDK, a third-party provider endpoint), never the app's own routes or the component under test; reuse the repo's auth fixtures / `E2E_AUTH_BYPASS`. If the do-not-ship journey doesn't exist, author it in `e2e-test` and tag it `@smoke` — smoke selects, it doesn't create browser specs.
+
+  This reference is *checked*, not just asserted: `validate --journeys <specs-dir>` reconciles both directions — a browser path pointing at a journey that isn't `@smoke`-tagged fails (the gate couldn't drive it), and a spec tagged `@smoke` that no path references *also* fails (it would silently never run — a false ALIVE). Without `--journeys` the link is unenforced, so wire it into CI wherever the specs are available.
+- **`api`** → send the request, capture the response, assert the `observable` (status + body shape). Read-only by default; a writing endpoint needs `mutates`/`tenant`.
+- **`cli`** → run the `command`, capture the pane, assert the `observable` (output + exit code). Isolate shared state (`tmux -L`, a `mktemp -d` workdir).
+
+## Results contract
+
+The drive step emits a JSON array the runner reduces into the verdict:
+
+```json
+[
+  { "id": "login-to-dashboard", "status": "pass", "evidence": "screenshots/login.png" },
+  { "id": "feishu-qr-mount", "status": "fail", "note": "NotFoundError: removeChild on unmount when switching to password" }
+]
+```
+
+`status` ∈ `pass` | `fail` | `blocked` | `cannot-run`. A path with no reported result is `missing`. **Only `pass` is alive** — `fail` means the candidate broke the path and makes the build **DEAD**; `blocked`, `cannot-run`, and `missing` mean the gate itself could not run and make the verdict **CANNOT-RUN** (issue #823), so a misconfigured secret/auth does not discard a healthy candidate. Carry `evidence` (a capturable artifact path) on pass and a `note` (what broke, or why the gate could not run) on the rest.
+
+A consumer may also report the overall verdict as `CANNOT-RUN` or `BLOCKED` when the gate as a whole could not produce a result; `smoke.mjs` reduces path-level `cannot-run`/`blocked`/`missing` into that top-level verdict.