@zappie/verify 0.2.8 → 0.3.0

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 # @zappie/verify
 
+## 0.3.0
+
+### Minor Changes
+
+- d1c8b1d: V14's `menuJsKbGz` now measures the same build the same way on every run. It counts the JavaScript the menu needs to start: the scripts and module preloads the server's document names, read when the document is parsed. Chunks the running page imports later, when it hydrates, in its effects, or on idle or intent, no longer count toward it. Before, they counted only when they happened to load inside the measurement window, so one build of a site measured anywhere from 163.5 to 194.6 KiB. The 200 KiB budget applies to the new figure. `menuJsLoadedKbGz` counts every script loaded once the network goes quiet, later imports included. V14 fails it past 350 KiB with `loaded_js_too_large`, and `findRegressions` compares it with the baseline, allowing 10% or 10 KiB of growth, whichever is larger. A site's `menuJsKbGz` drops on this release, so record a new baseline from its first run.
+- 5a2bce9: Verify can start somewhere other than `/`, for a site whose root is a directory of its menus. Set `verify.startPath` in `zappie.config.ts`, or pass `startPath` to `verify()` or `verifyProject()`. Every check that opened `/` as the menu opens that route instead. Set `verify.cataloguePath`, such as `/menu?menu={menu}`, when each menu has its own page: V03 then needs 95% of each menu's available items on that menu's page, instead of 95% of the branch's items on one page, and the checks that use an item open the page of a menu that holds it. A route must start with one `/` and contain no backslash. A site that sets neither runs as before. The report's `environment` records `startPath`, and `cataloguePath` when it is set. When `--store` is passed and `zappie.config.ts` exists but can't be read, the report's `warnings` says its verify settings were ignored. The `@zappie/sdk` site config and `zappie check` accept the `verify` settings, so a site that adds them needs `@zappie/sdk` 0.1.14 and `@zappie/check` 0.2.4 in the same change. This is a minor release, so a regression comparison doesn't compare these reports with reports from 0.2.
+
+### Patch Changes
+
+- Updated dependencies [5a2bce9]
+  - @zappie/sdk@0.1.14
+
+## 0.2.9
+
+### Patch Changes
+
+- 9b25eb7: The packages now live in zappie/storefront-packages, and each package's repository, homepage and issues links point there. This release is the first one published from that repository.
+- Updated dependencies [9b25eb7]
+  - @zappie/sdk@0.1.13
+
 ## 0.2.8
 
 ### Patch Changes

package/README.md

@@ -24,6 +24,7 @@ With no URL, it builds your project, starts a local preview, runs the checks and
 | -------------------------------- | ------------------------------------------------------------------------------------- |
 | `--suite standard` or `extended` | The standard checks, or those plus the extended ones                                  |
 | `--only <checks>`                | Run only these checks: IDs such as `V01,V04`, or a set such as `quick`                |
+| `--start <path>`                 | The route verify opens as the menu. Default: `verify.startPath`, then `/`             |
 | `--report <file>`                | Save the full report as JSON                                                          |
 | `--json`                         | Print the report as JSON                                                              |
 | `--no-build`                     | Serve the project's existing build instead of building it again. Local mode only      |
@@ -33,6 +34,23 @@ With no URL, it builds your project, starts a local preview, runs the checks and
 
 The `quick` set is V01 to V07, V09, V10 and V13: the fast journey checks without V14 to V18. V18 stays outside build-blocking quick runs while security headers are report-only. The set is exported as `CHECK_SETS.quick`.
 
+### Where the storefront starts
+
+Verify opens `/` as the menu. A site whose root is something else, such as a directory of its menus, tells verify where to start in `zappie.config.ts`:
+
+```ts
+export default {
+  store: "your-store",
+  theme: "./theme.json",
+  verify: { startPath: "/menu", cataloguePath: "/menu?menu={menu}" },
+} satisfies ZappieSiteConfig;
+```
+
+- `startPath` is the route verify opens wherever it opened `/`. It must list items. `--start` wins over it.
+- `cataloguePath` is the page of each menu: `{menu}` stands for the menu id and `{branch}`, when you use it, for the branch id. With it, V03 opens every menu's page and needs 95% of that menu's available items on it, instead of 95% of the branch's items on one page. V04, V05, V13, V16 and V17 open the page of a menu that holds the item they use. An item that sits in no category of any menu belongs to no menu's page, so V03 doesn't judge it.
+
+Without them, verify runs as before. The report's `environment` records `startPath`, and `cataloguePath` when it is set.
+
 ### Chromium
 
 Verify launches Playwright's own Chromium. To use another one, such as a system Chromium on a build machine, set `ZAPPIE_VERIFY_CHROMIUM` to its executable, or pass `browserExecutablePath` from code, which wins over the variable. The report's `environment.browser` then names the executable after the version.
@@ -63,7 +81,7 @@ const report = await verifyProject({
 console.log(printSummary(report));
 ```
 
-`verifyProject` also takes `build: false`, `port`, `only`, `runner`, `artifactsDir` and `browserExecutablePath`, the options behind the flags above. To verify a site you serve yourself, call `verify({ url, key, store, suite })` instead.
+`verifyProject` also takes `build: false`, `port`, `only`, `startPath`, `cataloguePath`, `runner`, `artifactsDir` and `browserExecutablePath`, the options behind the flags above. It reads `startPath` and `cataloguePath` from `zappie.config.ts` when you don't pass them. To verify a site you serve yourself, call `verify({ url, key, store, suite })` instead, with `startPath` and `cataloguePath` when the site sets them.
 
 The report's shape is the JSON Schema at `@zappie/verify/schema/verify-report.v1.json`.
 
@@ -92,6 +110,10 @@ V13 scans each page and overlay once its entry animations have finished, so a fa
 | `data-zappie-action="add-to-cart"` | Checks that the keyboard reaches an Add control. On a menu without quick add, it tabs to an item without required options, opens it with Enter and tabs to its Add. |
 | `data-zappie-checkout`             | Checks that the keyboard reaches the enabled checkout control on `/checkout`.                                                                                       |
 
+## Menu JavaScript
+
+V14 opens the menu on a phone profile and fails when `menuJsKbGz` passes 200 KiB gzipped. It counts the JavaScript the menu needs to start: the scripts and module preloads the server's document names. Chunks the running page imports later, when it hydrates, in its effects, or on idle or intent, don't count, so one build measures the same on every run. `menuJsLoadedKbGz` counts every script loaded once the network goes quiet, those later imports included. It fails V14 past 350 KiB, and `findRegressions` compares it with the baseline, so code moved out of the document into late imports still counts.
+
 ## Overlays
 
 Mark each control that opens an overlay with `data-zappie-overlay-trigger="bag"`, `"branch"`, `"search"` or `"item"`, and the overlay's `role="dialog"` element with `data-zappie-overlay` and the same name. V16 waits until the menu has hydrated and its idle work, such as overlay preloads, and the downloads that work starts are done, as they would be for a shopper who has read the menu. Then it taps each marked trigger on a phone profile with the CPU slowed four times, and fails when the overlay takes more than 100 ms to paint. Anything the tap itself loads counts toward that time. Each tap is a shopper's first open on a first visit: V16 opens every overlay in a new browser context, so the checks before it can't warm it. It opens each overlay up to three times, each in its own context, and judges the median, so one slow sample on a busy machine doesn't fail it. It stops after two samples on the same side of the budget, because the third can't change the median. The CPU slowdown is calibrated to the machine: before the checks run, verify scores the browser's CPU with a short fixed benchmark and sets the throttling rate so that V14, V15 and V16 see the same phone everywhere. A developer laptop gets about four times, a slower CI runner less, between one and eight times. The report's `environment` records `cpuBenchmark` and the `cpuThrottlingRate` the checks used. The report's `overlayOpenMs` holds the medians and `overlayOpenSamplesMs` holds every sample, in the order they ran. V17 opens each one from the keyboard, in a new browser context too, and checks that focus moves in, Tab stays in, the page behind doesn't scroll, Escape closes it and focus returns to the trigger. Both are skipped while the menu has no markers.
@@ -109,6 +131,8 @@ const regressions = findRegressions(baselineReport, report);
 
 A metric regresses when it grows by more than its relative threshold or its absolute floor, whichever is larger. The defaults are in `DEFAULT_REGRESSION_THRESHOLDS`. A metric the baseline records and the new report doesn't also counts as a regression.
 
+Compare only reports with the same `environment.startPath`, and `cataloguePath`. A report without `startPath` started at `/`. Metrics measured from another page don't compare.
+
 ## Docs
 
 - [CLI: zappie verify](https://docs.zappie.ai/cli/verify)

package/dist/index.d.ts

@@ -44,6 +44,13 @@ type VerifyReport = {
         cpuBenchmark?: number;
         /** The CPU throttling rate the timed checks ran at, calibrated from `cpuBenchmark`. */
         cpuThrottlingRate?: number;
+        /**
+         * The route the checks opened as the menu. Reports from before verify 0.3.0 have none, and
+         * started at `/`. Metrics measured from different start paths don't compare.
+         */
+        startPath?: string;
+        /** The catalogue path V03 and the item checks used, when the run had one. */
+        cataloguePath?: string;
     };
     startedAt: string;
     durationMs: number;
@@ -54,8 +61,22 @@ type VerifyReport = {
         skipped: number;
     };
     checks: CheckResult[];
+    /** Findings about the run itself, such as settings verify could not read. */
+    warnings?: {
+        code: string;
+        message: string;
+    }[];
     metrics: {
+        /**
+         * Gzipped JavaScript the menu needs to start: the scripts and module preloads its server
+         * document names. Budgeted.
+         */
         menuJsKbGz?: number;
+        /**
+         * Every script loaded once the network goes quiet, idle and intent preloads included. Held to
+         * a loose 350 KiB budget and compared with a baseline, so code moved out of the document counts.
+         */
+        menuJsLoadedKbGz?: number;
         lcpMs?: number;
         cls?: number;
         productCardTappableMs?: number;
@@ -79,6 +100,23 @@ type VerifyOptions = {
     deploymentId?: string;
     /** Check IDs to run, such as `V01`, or a named set such as `quick`. All checks when unset. */
     only?: string[];
+    /**
+     * The route verify opens as the storefront's menu, such as `/menu`, for a site whose root is
+     * a directory rather than a menu. It must list items. Defaults to `/`.
+     */
+    startPath?: string;
+    /**
+     * The page of each menu, with `{menu}` where the menu id goes and, optionally, `{branch}` for
+     * the branch id, such as `/menu?menu={menu}`. With it, V03 judges each menu on its own page,
+     * and a check that needs an item opens the page of a menu that holds it. Without it, V03
+     * judges every item of the branch on the start page.
+     */
+    cataloguePath?: string;
+    /** Findings about the run to record in the report's `warnings`. */
+    warnings?: {
+        code: string;
+        message: string;
+    }[];
     viewports?: Viewport[];
     artifactsDir?: string;
     timeoutMs?: number;
@@ -147,7 +185,7 @@ type RegressionThreshold = {
     /** Growth always allowed, in the metric's own unit, so small values don't trip on noise. */
     absolute: number;
 };
-type RegressionMetric = "menuJsKbGz" | "menuCssKbGz" | "lcpMs" | "cls" | "productCardTappableMs" | "productCardOpenMs" | "overlayOpenMs";
+type RegressionMetric = "menuJsKbGz" | "menuJsLoadedKbGz" | "menuCssKbGz" | "lcpMs" | "cls" | "productCardTappableMs" | "productCardOpenMs" | "overlayOpenMs";
 /**
  * A metric regresses when `head - base > max(absolute, relative × base)`. Sizes are
  * deterministic, so their allowance is small; timings carry runner noise, so theirs is wider.