@crvouga/atlas 1.13.0 → 1.15.0

package/docs/reference/schema.md

@@ -498,6 +498,39 @@ export const RunDetailSchema = …
 
 `GET /api/reports/runs/detail`: one run and its last result for every test, environment and producer it recorded.
 
+#### `UiProfileSchema`
+
+```ts
+export const UiProfileSchema = …
+```
+
+A state's profile: the client and the choices it was captured under.
+
+#### `UiImageSchema`
+
+```ts
+export const UiImageSchema = …
+```
+
+The picture of a capture.
+
+#### `UiCellSchema`
+
+```ts
+export const UiCellSchema = …
+```
+
+One state in one profile.
+
+#### `UiCatalogSchema`
+
+```ts
+export const UiCatalogSchema = …
+```
+
+`GET /api/reports/ui-catalog`: the UI catalog, computed from the winning results of the `ui-state` and `ui-plan` parts
+of every layer. States are in id order and cells in profile order, so the same layers always give the same document.
+
 ### Command results (`--format json`)
 
 Source: `packages/schema/src/cli-results.ts`

package/docs/report-contract.md

@@ -84,7 +84,7 @@ Zod schema as `LayerDocumentSchema` from `@crvouga/atlas/schema`.
 | `freshness` | `modified`: recorded from a checkout with uncommitted changes. `unknown`: origin not captured. Advisory only; see [Statuses](#statuses-and-the-gate). |
 | `run` | Optional `{ id, title?, startedAtMs?, endedAtMs? }`: the execution a local or manual producer names. Layers with the same `id` in one project are one run in the run list, however many shards or batches uploaded them; a later invocation at the same branch and commit gets another `id`. Without it a run is its `runUrl`, else `branch@sha`, as before. `title` is for people. Grouping and display only: it is **never part of the slot**, never decides a winner or a status, and never hides `sha`, `branch` or `freshness`. A `run` a reader cannot read is treated as absent. |
 | `specs` | The specs the layer was recorded against, with a `fingerprint` of their text, so a reader can warn when a spec changed since. Optional. |
-| `parts[].kind` | An open category (`behavior`, `performance`, `accessibility`, …). A new kind needs no change to any reader. Part of the slot. |
+| `parts[].kind` | An open category (`behavior`, `performance`, `accessibility`, …). A new kind needs no change to any reader. Part of the slot. `ui-state` and `ui-plan` are the [UI catalog](ui-catalog.md)'s: a captured UI state, and the states that are expected. |
 | `parts[].checks[]` | `id` (required, stable), `title`, `status`, `detail`, `specPath`, `measurements`, `durationMs`. |
 | `tags`, `labels`, `links`, `attachments`, `annotations`, `extensions` | Evidence, on the layer, a part or a check: what a result carries beyond its outcome. All optional and open. See [evidence](#evidence) and [the guide](extending.md). |
 | `checks[].steps`, `checks[].attempts` | A check's timeline, and every try when it was retried. Optional. |
@@ -227,7 +227,7 @@ are part of the contract:
 Tools map to these four. CTRF `skipped`, `pending` and `other`, and JUnit `<skipped>`, are `not-run`.
 An Atlas path that passed on retry (`flaky`) is `passed`, with the detail "Passed on retry (flaky)".
 
-Timings that only qualify a check (parts of kind `performance`) are not counted as checks when
+Timings that only qualify a check (parts of kind `performance`), and what a UI catalog expects (`ui-plan`), are not counted as checks when
 anything else is recorded. A measurement over its allowance is reported and never fails a check.
 
 Opinions (parts of kind `critique`, the verdicts of [automated UI reviewers](critics.md)) are never counted as checks,

package/docs/rich-evidence.md

@@ -191,6 +191,11 @@ time. Every field after `unit` is optional and none of them decides a status, a
 - **Reading it from the API.** `GET /api/reports/tests/history` entries carry `latency` (their product-latency
   measurements) next to `durationMs`. The pure functions are in `@crvouga/atlas/reports/latency`.
 
+## A catalog of UI states
+
+A screenshot per screen, per viewport and color scheme, is a *UI catalog*: check ids are the states, the viewport and scheme are
+`parameters`, and the viewer's `/catalog` shows them with what is still to be captured. See [the UI catalog](ui-catalog.md).
+
 ## Finding runs and artifacts
 
 The viewer's **Runs** and **Artifacts** tabs and the API answer one query, so a link means the same thing everywhere.

package/docs/server.md

@@ -522,6 +522,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
 | `GET /api/reports/artifacts` | read | Recorded results that carry evidence, across the whole history. |
 | `GET /api/projects/{project}/reports/artifacts` | read | Recorded results that carry evidence, across the whole history. |
+| `GET /api/reports/ui-catalog` | read | The UI catalog: captured UI states per profile, and what is still expected. |
+| `GET /api/projects/{project}/reports/ui-catalog` | read | The UI catalog: captured UI states per profile, and what is still expected. |
 | `GET /api/reports/runs/detail` | read | Every test one run recorded. |
 | `GET /api/projects/{project}/reports/runs/detail` | read | Every test one run recorded. |
 | `GET /api/reports/outline` | read | The specs a report is laid over. |

package/docs/ui-catalog.md

@@ -0,0 +1,173 @@
+# The UI catalog
+
+A browsable inventory of a product's user interface: each **state** (a screen, or one state of a screen) captured once per
+**profile** (a viewport, a color scheme, a device), with the facts needed to find and trust it, and what should exist but has not
+been captured yet. Any program that can write a layer can feed it, and the catalog fills in as results stream in, so a run of
+thousands of captures can be watched while it happens and split across as many machines as needed.
+
+Nothing about it is new to the report contract. A state is an ordinary check with a screenshot, the expected set is an ordinary
+layer, and the stream is the one every viewer already holds. This page is the convention that makes those usable as a catalog, the
+decisions behind it, and what is not built yet.
+
+- [How a state is written](#how-a-state-is-written)
+- [What is expected](#what-is-expected)
+- [Publishing from any collector](#publishing-from-any-collector)
+- [Sharding](#sharding)
+- [Streaming and loading](#streaming-and-loading)
+- [Exploring it](#exploring-it)
+- [Decisions](#decisions)
+- [What stays in your project, and what is not built yet](#what-stays-in-your-project-and-what-is-not-built-yet)
+
+## How a state is written
+
+A state is one **check** in a layer whose part has `kind` **`ui-state`**.
+
+| Part of the idea | Where it goes |
+| --- | --- |
+| The state's identity | The check `id`. Make it stable (`checkout/payment`, `home?cohort=new`): a new capture replaces an old one only if the id is the same. |
+| How the capture went | `status`: `passed` is **captured**, `failed` is attempted and failed (say why in `detail`), `running` is being taken, `not-run` is skipped. |
+| The picture | An attachment with `role: "screenshot"` (else the first image), stored in the layer by content or given as an `https` `uri`. `width` and `height` help the viewer lay it out. |
+| The profile | The layer's `environment` (`desktop-web`, `mobile-web`, `ios`, `android`) and `parameters` (`{ "viewport": "390x844", "scheme": "dark" }`). Parameters split a slot, so the same state at another viewport is **another result, side by side**, never a replacement. |
+| Facts the producer declares | The check's `extensions["atlas.ui-state"]`: `route`, `fixture`, `cohort`, `area`, `flags` (a list), `covers` (the product surfaces the state shows). All optional. |
+| Facts that describe without splitting | The check's `labels` (`shard`, `browser`) and `tags`. Labels never make a different slot. |
+| Where it came from | The layer's `producer`, `sha`, `branch`, `runUrl`, `freshness`, `recordedAtMs`: shown with every capture, with the layer id and the backend that holds it. |
+
+```json
+{
+  "producer": "device-farm", "kind": "ui-state", "environment": "mobile-web",
+  "parameters": { "viewport": "390x844", "scheme": "dark" }, "recordedAtMs": 1760000000000,
+  "checks": [{
+    "id": "checkout/payment", "title": "Payment, card declined", "status": "passed",
+    "labels": { "shard": "2/8" },
+    "extensions": { "atlas.ui-state": { "route": "/checkout", "fixture": "declined-card", "cohort": "new-customer", "flags": ["new-nav"], "covers": ["payment-form"] } },
+    "attachments": [{ "name": "payment.png", "role": "screenshot", "contentType": "image/png", "path": "payment.png" }]
+  }]
+}
+```
+
+`atlas report publish --checks file.json --store …` publishes that file, reading `payment.png` from beside it. Nothing in it
+mentions how the page was driven. Because a state is a check, the gate counts it (a `failed` capture fails `atlas report status`),
+`atlas report status` lists it, and a reader that does not know the catalog still shows a check with a screenshot. Evidence never
+decides a result: a picture that cannot be read leaves the check, its status and its slot as they were.
+
+## What is expected
+
+What should exist is a **layer too**, of kind **`ui-plan`**: the same ids, the same profiles (environment and parameters), every
+check `not-run`, published by a planner, by each shard for the states it was given, or both. A plan fills a slot of its own (the
+kind is part of the slot), so a capture never erases it and it never hides a capture; and `not-run` is never a pass.
+
+| A cell (one state in one profile) is | When |
+| --- | --- |
+| **Captured** | the winning capture passed |
+| **Capture failed** | the winning capture failed (its `detail` says why, and a diagnostic picture may be attached) |
+| **Capturing** | the winning result is `running` |
+| **Skipped** | a producer recorded it as `not-run` |
+| **Not captured yet** | a plan expects it and nothing has been recorded |
+
+A captured cell that no plan names is shown as **not in the expected set**. **Without any plan, nothing is "not captured yet" and
+nothing is "not in the expected set"**: what should exist is then unknown, and the catalog says so instead of guessing. "Not yet
+captured" is therefore different from "never declared".
+
+Coverage against an **inventory of the product's surfaces** is a plan entry with `"inventory": true` (the id is the surface;
+nothing is captured for it). A state says which surfaces it shows in `covers`. The catalog lists every surface as *covered* (a
+state that shows it is captured), *states cover it, none captured yet*, or *no state covers it*.
+
+## Publishing from any collector
+
+Any program that writes the layer format feeds the catalog: a shell script, a Go test, a Storybook crawler, a device farm.
+
+```ts
+import { publishChecks } from '@crvouga/atlas/reports/publish-checks';
+import { uiPlanLayer, uiStateLayer } from '@crvouga/atlas/reports/ui-catalog-publish';
+
+// The planner, once: what the product should have, in every profile.
+await publishChecks(store, uiPlanLayer({
+  producer: 'catalog-plan', recordedAtMs: Date.now(), environment: 'mobile-web', profile: { viewport: '390x844', scheme: 'dark' },
+  states: [{ id: 'checkout/payment', covers: ['payment-form'] }, { id: 'settings', inventory: true }]
+}));
+
+// Each collector or shard, with its own captures and no knowledge of the others.
+await publishChecks(store, uiStateLayer({
+  producer: 'web-capture', recordedAtMs: Date.now(), environment: 'mobile-web', profile: { viewport: '390x844', scheme: 'dark' }, shard: '2/8',
+  states: [{ id: 'checkout/payment', route: '/checkout', covers: ['payment-form'], screenshot: { data: png, width: 390, height: 844 } }]
+}));
+```
+
+`uiStateLayer` and `uiPlanLayer` only build the input of `publishChecks`; they use no browser and no test framework. `store` can be
+a directory, a bucket or the server (`httpStore`), exactly as for any layer.
+
+**Playwright is one adapter and not a dependency.** `captureUiState(page, { id, ready })` from `@crvouga/atlas/playwright` waits
+for a readiness check (bounded), photographs the page and returns the capture for `uiStateLayer`. A page that never became ready is
+a `failed` capture that says why (with a picture of what it showed), never an exception that ends the run. Nothing that reads,
+composes or shows the catalog imports a collection framework, and a test keeps it that way.
+
+## Sharding
+
+A shard is **any publisher that does not know the others**: by case, by profile, by area or by machine; Atlas does not care which.
+Each publishes its own layers (a `shard` label records which, and is shown with each capture). The guarantees are those of every
+layer ([the contract](report-contract.md#guarantees)), which is what the tests prove for the catalog:
+
+- The captures of N shards (N is 1 or more), combined, equal the catalog one unsharded run produces.
+- Running a shard again replaces only the cells it produces (a newer `recordedAtMs` wins its slot); every other cell is untouched.
+- A shard that never publishes leaves its cells **not captured yet** when a plan names them, and absent otherwise. They are never
+  shown as captured, and the other shards' cells are unaffected.
+- Atlas does not need the shard count. Completeness is the plan: a cell is expected because a plan says so.
+
+## Streaming and loading
+
+- **Streaming.** There is no second stream. A viewer holds the report over `/api/events` and reads what changed with
+  `/api/reports/evidence/effective?since=<version>`; the catalog is computed from the report it holds, so a capture appears when
+  its layer arrives, with no reload and no file rebuilt. Events carry metadata only; the picture is fetched when a card is on screen.
+- **Reconnecting.** A viewer that was offline or whose stream dropped reads the delta since the version it holds (or the whole report
+  when the server no longer remembers it). Cells are keyed by state and profile, so nothing is missing and nothing is duplicated.
+- **No stream.** A viewer without the stream (or a static export) polls, as the rest of the viewer does, and shows everything that exists.
+- **Slow and failing backends.** Each backend is listed and read on its own, so a slow one never holds back the others: their
+  captures are shown, and the slow backend's appear when it answers. A backend that fails, or a layer that cannot be read, is named
+  on the page and in the document (`sources`); a capture that depended on it is not shown as captured.
+- **Images.** A picture is a file in the layer, addressed by content (`/layers/<layer>/blobs/attachments/<sha256>`), loaded lazily
+  for the cards on screen, and served under the same untrusted-bytes rules as every attachment. A picture given as a `uri` is
+  followed only if it is `http` or `https`. A producer may leave the bytes out (`omitted`), which is told apart from never taken.
+
+## Exploring it
+
+The viewer's **UI catalog** (`/catalog`, in the menu beside the spec types once the report holds a state or a plan) shows a card per
+state with its picture and a coverage badge per profile, and filters by words, coverage, profile, area, route, cohort, feature flag,
+surface and producer. The address holds the filters and the open state, so a link reproduces the view. Opening a state shows its
+profiles side by side with the declared facts, the failure reason, and for each picture the layer, backend, producer, shard, commit
+and time. *Download the manifest* saves the filtered catalog as JSON.
+
+```sh
+curl "$ATLAS_SERVER_URL/api/reports/ui-catalog?coverage=failed,missing&profile=…"   # JSON; add format=markdown for text
+```
+
+`GET /api/reports/ui-catalog` answers the same document the viewer computes (`uiCatalog` from
+`@crvouga/atlas/reports/ui-catalog`), filtered by `q`, `coverage`, `profile`, `area`, `route`, `cohort`, `flag`, `surface` and
+`producer`, paged with `limit` and `offset`, with an ETag. Its picture `path` is relative to `/api/reports`.
+
+## Decisions
+
+The questions the original proposal left open, and how they are answered here. Each is the simplest answer that stays inside the
+[stable contracts](report-contract.md#versioning-and-stability): optional fields only, no change to a slot or a winner.
+
+| Question | Decision |
+| --- | --- |
+| Layer and event schema | An existing check in a `ui-state` part, a `screenshot` attachment, facts under the `atlas.ui-state` extension. No new part type, no new schema version. The report row gained one optional field, `state`, only for `ui-state` and `ui-plan` rows. |
+| Identity and slot | The check id is the state; the slot is the usual (producer, kind, spec, id, environment, parameters). Profile dimensions are `parameters` because they must split the slot; labels never do. |
+| Expected versus captured | A `ui-plan` layer. `ui-plan` is a diagnostic kind (like `performance`): it qualifies a check and is not counted as one when the check has any other result. |
+| Coverage against an inventory | `inventory: true` plan entries plus each state's `covers`. |
+| Carrying images | By content in the layer (`bundleKey`) or by `uri`; loaded lazily; the existing size limits of an upload apply. |
+| The event stream | No new stream or events: the existing `report` event and `?since=` deltas. |
+| Sharding | No shard concept in the format; a shard is any publisher. A `shard` label is provenance. Completeness comes from the plan, so Atlas needs no shard count. |
+| What moves | Reading, composing, the viewer, the document and the API, the publish builders, a Playwright capture adapter. Staying with a product: its state declarations, its inventory, its fixtures and its runner. |
+| Where it lives | `@crvouga/atlas/reports/ui-catalog` (model), `…/ui-catalog-publish` (builders), `…/ui-state` (convention); the viewer's `/catalog`; the server's `/api/reports/ui-catalog`; `@crvouga/atlas/playwright`'s `captureUiState`. |
+| Exploration capabilities | Search, filters, profile comparison, coverage and surface coverage, failure diagnostics, deep links, manifest export. |
+| Static export | The catalog is computed from the layers a viewer holds, so a static export (`atlas client build`) shows it; it just does not stream. |
+
+## What stays in your project, and what is not built yet
+
+**Stays in a project** (it is specific to the product): the list of states and profiles to capture, the inventory of surfaces,
+fixtures and test data, the runner that drives the product to each state, and the readiness conditions of each screen.
+
+**Not built yet:** a command line to list and publish a catalog without a script (use `atlas report publish --checks` today), a
+dedicated static single-file export, diff views that compare two commits of a state, and a hosting-platform publish step beyond
+`atlas serve`. The first slice makes the catalog explorable, streaming, shardable and independent of how it was collected.

package/docs/ui.md

@@ -96,6 +96,20 @@ when Watch is on screen.
 - The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
   none are listed under "No domain declared"; the Tests view reports domain problems).
 
+## UI catalog
+
+`/catalog` (`catalog/`) is the viewer's inventory of captured UI states ([the guide](ui-catalog.md)). It is computed in the page from
+the layers the viewer already holds (`catalog/provider.tsx`, only while the page is open), so states appear as results stream in and
+it works without a server. Cards are Mantine `Card`s in a responsive `SimpleGrid`; the filters live in the shell's navigation column
+(`UiCatalogFilters`) and the open state in a `Drawer`, and both are the address (`catalog/search.ts`).
+
+- Only `captured` is called captured. Every other coverage (`Capture failed`, `Capturing`, `Skipped`, `Not captured yet`) has its own
+  words and mark, and a cell without a picture says so in words instead of showing an empty frame. Never show a plan as a result.
+- A backend that is not available, a notice, and layers that could not be read are shown above the grid; the catalog says it may be
+  incomplete. Waiting follows the loading policy below, and layers still arriving are counted.
+- Pictures are `loading="lazy"` and only the cards on screen are fetched. The grid is bounded with *Show more*.
+- The open state shows every profile side by side with the layer, backend, producer, shard, commit and time of each picture.
+
 ## Reliability
 
 An open test shows its reliability (`explorer/Reliability.tsx`, worked out by `reports/reliability` from the history the page
@@ -129,6 +143,35 @@ environment) is unknown unless the producer declared it under the `atlas.failure
 wording. Records of one source whose outcome changed are listed oldest first, and a later pass is never called a fix.
 In the explorer the chosen source is the `source` search parameter (a layer id; unlike `layer` it does not pin a run).
 
+## Compare two runs
+
+The **Compare** tab (`explorer/CompareView.tsx`, `explorer/ComparePair.tsx`, `@crvouga/atlas/reports/run-compare`) sets a
+baseline run against a candidate run. Both are pinned in the address (`base` and `cand`, run keys; `change` is the group
+shown and `pair` the behavior opened), so a link reproduces the comparison and a newer run never replaces either. A run
+that is no longer in the report is said to be missing and is never swapped for another. A run's page offers "Compare with
+this run as the baseline / candidate".
+
+- Results are paired by the check they record (spec path and check id), then by the condition they were recorded under
+  (kind, producer, environment, parameters). A title never pairs two results. The same check under another condition is
+  paired and marked "Conditions differ".
+- Every behavior is in one group: newly failing, recovered, still failing, promise or conditions differ, not run or
+  unfinished, not in the candidate run, not in the baseline run, same outcome. A result the candidate did not run is
+  unknown, never regressed or passed; one only the baseline has is "not in the candidate run" (not run, renamed or
+  dropped: the comparison cannot tell). A reworded behavior, or one recorded under different conditions, is not framed as
+  a regression or recovery: both outcomes are shown beside the changed promise. A different spec-file revision with the
+  same wording is disclosed beside the movement (a fingerprint is of the whole file); a revision that was not recorded is
+  unknown, never the same.
+- Opening a behavior shows Mantine `Tabs`: **Promise** (wording, spec revision, recorded steps with the ones only one side
+  has), **Outcomes** (each side's own provenance, execution time and failure), **Recordings** (one `<video>` per side with
+  `preload="metadata"`, read only while that tab is on screen; each keeps its own attempt, spec revision and time
+  origin, they are not synchronized, and "Go to a step in both recordings" seeks each to its own recorded step time when
+  the steps are shared by name and timed; otherwise the page says why they are not aligned) and **Figures**.
+- Figures are compared like for like. Product latency is compared only for the same measurement on the same devices,
+  environments and instrumentation, and reliability only when both sides count at least the minimum of independent
+  executions against the same recorded spec revision (and a difference inside the uncertainty is "cannot be told apart").
+  Otherwise the page says "Not compared" with the reasons and computes no change. Test execution time is labelled as such
+  and is never product latency.
+
 ## Watch a filtered set as a playlist
 
 The Artifacts tab and an opened run have a **Watch these N results** button. It opens a Mantine `Modal` (`explorer/playlist.tsx`)
@@ -276,7 +319,8 @@ The narrowly scoped exceptions are:
 | Area | Screens/interactions | Mantine composition |
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
-| Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
+| UI catalog | Captured UI states per profile as cards with coverage badges, filters in the index column, surface coverage, an open state with its profiles side by side and where each picture came from, manifest download | AppShell navbar, SimpleGrid, Card, Image, MultiSelect, TextInput, Drawer, Accordion, Table, Badge, Alert |
+| Project home | Tests/runs/artifacts/compare tabs, a two-run comparison with promise, outcome, recording and figure tabs per behavior, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
 | Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |