@crvouga/atlas 1.11.0 → 1.13.0

package/docs/reference/http-api.md

@@ -414,7 +414,7 @@ Responses: `200` It exists.; `401` No token, or an unknown one. Send `Authorizat
 
 ### Upload an object
 
-`PUT /api/store/{key}` · role: publish
+`PUT /api/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -426,6 +426,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 - `key` (path, required): An object key under `layers/`, such as `layers/manifests/<id>.json`. It contains slashes, which are not percent-encoded; every other character is.
 - `content-type` (header): Stored with the object and returned when it is read.
 - `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
@@ -434,7 +436,7 @@ Responses: `200` Already there with the same bytes.; `201` Stored.; `401` No tok
 
 ### Upload an object
 
-`PUT /api/projects/{project}/store/{key}` · role: publish
+`PUT /api/projects/{project}/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -446,6 +448,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 
 - `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.

package/docs/reference/reports.md

@@ -1366,10 +1366,10 @@ A report: the results laid over the specs, with what was read and what could not
 #### `countResults`
 
 ```ts
-export function countResults(results: readonly EvidenceResult[]): StatusCounts & { total: number }
+export function countResults(all: readonly EvidenceResult[]): StatusCounts & { total: number }
 ```
 
-Counts results by status. Diagnostic results (parts of kind `performance`, which only qualify a check) are left out
+Counts results by status. Opinion results (parts of kind `critique`) are never counted. Diagnostic results (parts of kind `performance`, which only qualify a check) are left out
 whenever any other result is present; if only diagnostics exist, they are counted so the report is not empty.
 
 #### `summarize`
@@ -2371,7 +2371,8 @@ export function checkSlots(results: readonly EvidenceResult[]): CheckSlot[]
 
 Groups the combined results (the winners of the slots) by check, in the order the checks were first seen.
 A check's status is the worst of its non-diagnostic results; if it has only diagnostics (timings), those are its
-outcomes. The title and id shown are the newest result's.
+outcomes. The title and id shown are the newest result's. Opinions (parts of kind `critique`, see `isOpinionKind`) are
+not checks and are left out.
 
 #### `SpecStatus`
 
@@ -4582,6 +4583,17 @@ export function isDiagnosticKind(kind: string): boolean
 
 Whether a part `kind` only qualifies an outcome (today only `performance`) and so is not counted as a check when anything else is recorded.
 
+#### `isOpinionKind`
+
+```ts
+export function isOpinionKind(kind: string): boolean
+```
+
+Whether a part `kind` records an opinion rather than an outcome (today only `critique`, an automated reviewer's
+judgement of a captured UI). An opinion is never counted as a check, never gates, and is not folded into a spec's
+rollup, even when it is the only thing recorded: a report of opinions alone has no test results. The check's `status`
+of such a part says whether the reviewer ran, not whether anything passed.
+
 #### `emptyCounts`
 
 ```ts
@@ -4593,11 +4605,12 @@ A fresh status tally with every status at zero.
 #### `outcomeCounts`
 
 ```ts
-export function outcomeCounts(rows: readonly LayerRow[]): StatusCounts
+export function outcomeCounts(allRows: readonly LayerRow[]): StatusCounts
 ```
 
-Counts rows by status, leaving out the diagnostic rows (see `isDiagnosticKind`) that only qualify an outcome.
-When every row is diagnostic they are counted, so a layer of timings alone is not empty.
+Counts rows by status, leaving out the opinion rows (see `isOpinionKind`) always, and the diagnostic rows (see
+`isDiagnosticKind`) that only qualify an outcome. When every remaining row is diagnostic they are counted, so a
+layer of timings alone is not empty.
 
 #### `layerEvidence`
 
@@ -6981,7 +6994,7 @@ Returns a `ReportStatus` (schemaVersion 1): `ok` is true exactly when `reasons`
 checks (or none completed), failed checks, running checks, specs without results (only with `requireAllSpecs`),
 each required environment with no checks (the name is normalised, so `iOS` means `ios`), then every notice of the
 report (an unreadable layer or backend). `failing` lists each failed or running winner, in a fixed order that does
-not depend on read order; diagnostic `performance` checks are left out unless nothing else was recorded.
+not depend on read order; diagnostic `performance` checks are left out unless nothing else was recorded, and `critique` opinions are always left out.
 `recordings` reports recording completeness of the same results apart from `ok`: an incomplete or unknown
 completeness never changes the gate (use `atlas report publish --require-recordings` to fail on a missing video).
 

package/docs/reference/schema.md

@@ -775,12 +775,128 @@ export function transitionId(source: string, event: string)
 Builds the id of a transition, the key of `transitions` in a manifest: the source state's name,
 `TRANSITION_ID_SEPARATOR`, then the event name.
 
+### The `atlas.critique` evidence schema
+
+Source: `packages/schema/src/critique.ts`
+
+What an automated reviewer (a critic) says about one captured UI. It is not a new layer format: a critique is a
+check inside a part of kind `critique`, and the opinion rides in the check's `extensions["atlas.critique"]`, so by the
+rules of docs/extending.md it can never decide a result. The check's `status` says only whether the critic ran
+(`passed` a verdict was produced, `failed` the critic could not produce one, `running`, `not-run`).
+
+Everything except `schema`, `subject` and `critic` is optional, so a single score (a model-based critic) and rich
+findings (an LLM critic) are one shape. Every object is open, and `kind`, `category`, `severity`, `label` and `reason`
+are open strings: a value nobody has heard of is kept and shown generically, never rejected. A reader that cannot read
+the whole extension drops it and keeps the check and its status (`readCritique` in `@crvouga/atlas/critics`).
+
+See docs/critics.md
+See docs/extending.md
+
+#### `CRITIQUE_EXTENSION`
+
+```ts
+export const CRITIQUE_EXTENSION = …
+```
+
+The key under which a critique is stored in a check's `extensions`.
+
+#### `CRITIQUE_SCHEMA_VERSION`
+
+```ts
+export const CRITIQUE_SCHEMA_VERSION = …
+```
+
+The version of the `atlas.critique` extension a reader of this release understands. A critique of another version is unknown, not wrong.
+
+#### `CRITIQUE_PART_KIND`
+
+```ts
+export const CRITIQUE_PART_KIND = …
+```
+
+The `kind` of the report part that holds critiques. Parts of this kind are opinions: never counted, never part of the gate.
+
+#### `CRITIC_PRODUCER_PREFIX`
+
+```ts
+export const CRITIC_PRODUCER_PREFIX = …
+```
+
+The prefix of the `producer` of a layer of critiques: `critic.<critic name>`.
+
+#### `CritiqueRegionSchema`
+
+```ts
+export const CritiqueRegionSchema = …
+```
+
+The region of the image a finding is about, in the stated unit (`px` of the stored image, or `normalized` 0..1).
+
+#### `CritiqueFindingSchema`
+
+```ts
+export const CritiqueFindingSchema = …
+```
+
+One thing a critic found: where, what kind, how bad, why, and what to do. Nothing is required.
+
+#### `CritiqueRelativeSchema`
+
+```ts
+export const CritiqueRelativeSchema = …
+```
+
+How a score reads in context. A raw score is not absolute: a loading overlay is compared with other loading overlays,
+not with full screens, and a change against the same entry's previous image is more telling than the level.
+
+#### `CritiqueVerdictSchema`
+
+```ts
+export const CritiqueVerdictSchema = …
+```
+
+The judgement: a score and how to read it, a label, a confidence. A critic that cannot score may give only a label.
+
+#### `CritiqueFailureSchema`
+
+```ts
+export const CritiqueFailureSchema = …
+```
+
+Why a critic produced no verdict: a reason code in an open vocabulary (`error`, `timeout`, `rate-limited`, `invalid-output`) and a message that holds no credential.
+
+#### `CritiqueSchema`
+
+```ts
+export const CritiqueSchema = …
+```
+
+The `atlas.critique` extension, version 1: the image judged (`subject`), who judged (`critic`, `model`,
+`parameters`), the opinion (`verdict`, `findings`) or why there is none (`failure`), and what it cost (`usage`).
+The prompt is recorded only as a hash (`parameters.rubric`). Emitted as `critique.schema.json`.
+
+#### `Critique`
+
+```ts
+export type Critique = z.output<typeof CritiqueSchema>;
+```
+
+A critique as stored in a check's `extensions["atlas.critique"]`.
+
+#### `CritiqueFinding`
+
+```ts
+export type CritiqueFinding = z.output<typeof CritiqueFindingSchema>;
+```
+
+One finding of a critique.
+
 ### Schema package entry point (`@crvouga/atlas/schema`)
 
 Source: `packages/schema/src/index.ts`. Import: `@crvouga/atlas/schema`
 
 Re-exports every Zod schema, constant and inferred type of the Atlas contract: the layer format and report status
-(`layer`), the statechart run manifest (`manifest`, `common`), the spec file formats (`spec`), the run index and
+(`layer`), the `atlas.critique` evidence of automated UI reviewers (`critique`), the statechart run manifest (`manifest`, `common`), the spec file formats (`spec`), the run index and
 report sources the viewer reads (`runs-index`, `report-sources`) and the server configuration (`server-config`).
 The JSON Schemas shipped in `schemas/` are generated from these by `bun run schemas`.
 
@@ -1869,7 +1985,8 @@ See docs/server.md
 export const ServerRoleSchema = …
 ```
 
-What a bearer token may do: `read` (look), `publish` (also upload) or `admin` (also remove scratch objects).
+What a bearer token may do: `read` (look), `critic` (also publish critiques, and nothing else), `publish` (also upload
+anything) or `admin` (also remove scratch objects).
 
 #### `ServerRole`
 

package/docs/reference/server.md

@@ -93,7 +93,7 @@ What it knows lives in its backends (by default a folder on its own disk), so re
 
 Source: `packages/server/src/auth.ts`
 
-Who is asking, and may they. Tokens have a role (`read` < `publish` < `admin`) and are compared in constant time. Reading
+Who is asking, and may they. Tokens have a role (`read` < `critic` < `publish` < `admin`) and are compared in constant time. Reading
 accepts a Bearer token or the browser's sign-in box (any user name, the token as the password); changing anything needs
 the Bearer header, because a browser sends the sign-in box's credentials to any page that asks, which would let another
 site upload with them. A server with no tokens is open, and refuses to listen beyond this machine unless told it may.
@@ -530,10 +530,10 @@ The version of the API described, `major.minor.patch`: the minor grows when an o
 #### `Role`
 
 ```ts
-export type Role = 'none' | 'read' | 'publish' | 'admin';
+export type Role = 'none' | 'read' | 'critic' | 'publish' | 'admin';
 ```
 
-Who may call an operation: `none` (anyone), or the least role that may (`read` < `publish` < `admin`).
+Who may call an operation: `none` (anyone), or the least role that may (`read` < `critic` < `publish` < `admin`).
 
 #### `Operation`
 

package/docs/report-contract.md

@@ -230,6 +230,10 @@ An Atlas path that passed on retry (`flaky`) is `passed`, with the detail "Passe
 Timings that only qualify a check (parts of kind `performance`) are not counted as checks when
 anything else is recorded. A measurement over its allowance is reported and never fails a check.
 
+Opinions (parts of kind `critique`, the verdicts of [automated UI reviewers](critics.md)) are never counted as checks,
+even when they are all that was recorded, and are never part of the gate: a low score, or a critic that crashed, leaves
+`atlas report status` as it was. The `status` of a critique check says only whether the critic ran.
+
 **Freshness never changes a status.** A layer whose spec fingerprint no longer matches, whose
 checkout was modified, or whose related code changed still shows its recorded outcome, with a
 warning ("Potentially outdated: spec changed"). Atlas favours quick, possibly stale evidence over

package/docs/rich-evidence.md

@@ -27,6 +27,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
 | `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
 | `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
 | `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
+| `extensions['atlas.media']` | attachment (video) | Where the check's steps are in the recording, so a click on a step seeks the video. See [Place steps in a recording](#place-steps-in-a-recording). |
 | `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
 | `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
 | `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
@@ -109,6 +110,44 @@ Every field is optional, and every value except the booleans is an open string.
 - **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
   dropped, and the attachment, check and layer stay.
 
+## Place steps in a recording
+
+A step says when it started on the check's clock (`offsetMs`, after the check started). A video has its own clock, which
+began when the recorder did, and the two are not assumed to agree: a recorder usually starts before the first step, and a
+paced demo may have delays put into the video that the check never waited for. To let a reviewer click a step and land on
+its moment in the video, say how the clocks relate in the `atlas.media` extension of the video attachment:
+
+```ts
+{
+  name: 'checkout.webm', role: 'video', durationMs: 9000,
+  extensions: {
+    'atlas.media': {
+      checkStartMs: 1200,                                           // the check started 1.2 s into the video
+      pauses: [{ atMs: 900, durationMs: 2000, reason: 'title card' }], // 2 s put into the video before the step at +900 ms
+      marks: [{ step: 'pay', mediaMs: 4100 }]                        // a step whose moment in the video you know exactly
+    }
+  }
+}
+```
+
+Every field is optional and none of it decides a status, a slot or a winner. A step is placed at `checkStartMs` + its
+`offsetMs` + the `durationMs` of every pause whose `atMs` is at or before the step; a mark names the moment outright and
+wins. Anything else is unavailable and said so, never guessed.
+
+- **Without timing, nothing is placed.** A video with no `atlas.media`, no `checkStartMs`, or a step with no `offsetMs` and
+  no mark, is never assumed to start with the check, and a step is never placed from the wording of its name. Selecting
+  such a step says why it cannot be shown, and moves nothing.
+- **Outside the video is not clamped.** A moment before the start, or after `durationMs` (or the length the player reports),
+  is unavailable. A declaration with a field Atlas cannot read (a pause without a duration) is reported and places nothing.
+- **Pacing is not latency.** A video with `pauses`, or with a `scope` of `pacing: 'paced'`, is labelled as paced: the time
+  between its steps includes deliberate pauses and is not application latency, and no measurement is changed by it.
+- **Each try has its own clock.** Put the video and the `atlas.media` of a retry on its `attempts[]` entry, with that
+  try's `steps`. A try is aligned only by its own video, never by the check's or an earlier try's.
+- **Link to a moment.** The viewer copies a link that names the run (`layer`), the exact recording (`clip`, its digest or
+  stored key) and the position (`clipAt`, milliseconds into the video). Opening it selects that recording of that run, even
+  when a newer run exists, and goes to the position; a recording the run does not hold is reported, never replaced.
+- **Older layers.** A layer without `atlas.media` reads exactly as before; the video plays and its steps list as ever.
+
 ## Product latency
 
 A test's `durationMs` includes fixture setup, deliberate waits, demo pacing, retries and cleanup, so it says how long the

package/docs/server.md

@@ -205,7 +205,7 @@ is said in words and the server does not start. The file has a published
 | How often (seconds) the backends are looked at for layers written by something else | `refreshSeconds` | `--refresh <seconds>` | `ATLAS_REFRESH_SECONDS` | `10` |
 | Let anyone look, not upload | `auth.anonymousRead` | `--anonymous-read` | `ATLAS_ANONYMOUS_READ` | no |
 | Listen publicly with no tokens (something in front signs people in) | `auth.insecure` | `--insecure` | `ATLAS_INSECURE` | no |
-| The admin token (also `ATLAS_PUBLISH_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
+| The admin token (also `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
 | The configuration file | `(the file itself)` | `--config <file>` | `ATLAS_CONFIG` | `./atlas.server.json` when it exists |
 | Largest uploaded object in MiB | `limits.maxObjectMiB` |  | `ATLAS_MAX_OBJECT_MIB` | `256` |
 | Largest JSON upload in MiB | `limits.maxJsonMiB` |  | `ATLAS_MAX_JSON_MIB` | `32` |
@@ -238,16 +238,18 @@ backend in the client and `/readyz`, and recovers by itself.
 | Role | May |
 | --- | --- |
 | `read` | look: the client, the report, `/api/status` |
+| `critic` | also publish [critiques](critics.md#authentication-and-authorization) and nothing else: blobs and layers whose producer is `critic.<name>` and whose parts are all of kind `critique`. It cannot post test results, start live runs or change settings |
 | `publish` | also upload layers, specs and runs, and remove the scratch objects `atlas report doctor` writes |
 | `admin` | all server operations |
 
 Tokens are at least 12 characters (`openssl rand -hex 24`) and different from one another. Give them as
-`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
+`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
 with `tokenEnv` so the file holds no secret.
 
 - **Sending a token:** `Authorization: Bearer <token>`. A browser's sign-in box also works for *looking* (any
   user name, the token as the password). **Uploading needs the Bearer header, never the sign-in box**: a browser
   sends those credentials by itself to any page that asks, which would let another site upload with them.
+- **Run critics against a server with tokens.** With no tokens every caller is an administrator, a critic included.
 - **No tokens** means open, and then the server refuses to listen beyond this machine unless told
   `--insecure` (for when a proxy in front does the signing in).
 - `/healthz` and `/readyz` never need a token, so a load balancer can use them.
@@ -490,8 +492,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/store/{key}` | read | Read an object. |
 | `HEAD /api/store/{key}` | read | Does an object exist. |
 | `HEAD /api/projects/{project}/store/{key}` | read | Does an object exist. |
-| `PUT /api/store/{key}` | publish | Upload an object. |
-| `PUT /api/projects/{project}/store/{key}` | publish | Upload an object. |
+| `PUT /api/store/{key}` | critic | Upload an object. |
+| `PUT /api/projects/{project}/store/{key}` | critic | Upload an object. |
 | `DELETE /api/store/{key}` | publish | Remove a scratch object. |
 | `DELETE /api/projects/{project}/store/{key}` | publish | Remove a scratch object. |
 | `GET /api/reports/tests` | read | Every test, with insights, facets and history statistics. |

package/docs/ui.md

@@ -150,6 +150,28 @@ over the same filtered set the page lists, in the same order (`explorer/playlist
   (paced, scripted, silent, synthetic) and the sentence that the length is the recording's, not production latency. Developer
   details opens the test pinned to that run's record and closes the playlist.
 
+## Steps and moments in a recording
+
+Wherever a result's steps sit beside its recording (a result's Timeline under Inspect, the Watch tab of a behavior, an
+opened watch playlist), `evidence/moments.tsx` joins them. Placement is from the recording's own declared timing
+(`evidence/media-timing.ts`, the `atlas.media` extension; see [Place steps in a recording](rich-evidence.md#place-steps-in-a-recording)),
+never from an assumption that the recording starts with the check.
+
+- A step has a "Recording m:ss.s" button that seeks the selected recording; the step's test-clock offset stays visible beside it.
+  A step that cannot be placed (no timing, no origin, no offset, outside the recording, unreadable declaration) is "Not aligned"
+  and selecting it states the reason in a status alert. Nothing is guessed and the video is not moved.
+- While playing, the step under the playhead gets `aria-current="step"` and "Playing now". It never moves focus, scrolls
+  the page or announces itself. Seeking only scrolls the player into view.
+- A paced recording (pauses declared, or a `pacing: paced` scope) says that gaps between steps include deliberate pauses and are
+  not application latency.
+- A failed step has "Jump to the failing moment" in the bar above the timeline, and the failure summary has the same action
+  (`requestFailureMoment`). If the recording is not on screen, or the step cannot be placed, it says so.
+- "Copy link to this moment" names the run (`layer`), the exact recording (`clip`) and the position (`clipAt`, ms), as the
+  explorer address `/?test=…&layer=…&clip=…&clipAt=…` (the playlist links to the result it is playing). Opening it selects that
+  recording of that exact run and seeks once the player knows its length; a position past the end or a recording that the run
+  does not hold is reported and nothing else is played in its place. If the browser refuses to copy, the link is shown to copy.
+- One provider serves one holder: the check, or one try. A try's steps are placed by that try's recording and timing only.
+
 ## Review collections
 
 The **Collections** tab (`explorer/collections.tsx`, `?tab=collections&collection=<id>&rev=<n>`) lists and opens review