@crvouga/atlas 1.14.0 → 1.15.0

package/docs/reference/reports.md

@@ -1694,6 +1694,16 @@ Like {@link summarizeEvidence} for the checks of a part: a video that declares n
 declares none, takes the scope `inherited` from its part or layer (the nearest declaration of the two, see
 `effectiveScope`). Without `inherited` it is the same as `summarizeEvidence`.
 
+#### `summarizeStoredEvidence`
+
+```ts
+export function summarizeStoredEvidence(inherited: EvidenceScope | undefined, unstored: ReadonlySet<string> | undefined, ...holders: readonly EvidenceHolder[]): EvidenceSummary | undefined
+```
+
+Like {@link summarizeEvidenceIn}, knowing which bundle keys the layer names but does not hold (`unstored`, see
+`unstoredKeys`): a video whose only bytes are such a key is `available: false`, never a recording. Without
+`unstored` it is the same as `summarizeEvidenceIn`.
+
 #### `summarizeLayerEvidence`
 
 ```ts
@@ -1703,6 +1713,14 @@ export function summarizeLayerEvidence(...holders: readonly EvidenceHolder[]): E
 Like {@link summarizeEvidence} for a layer and its parts: the same summary, but a video attached to the layer or a
 part is owned by the `layer`, not by a check.
 
+#### `summarizeStoredLayerEvidence`
+
+```ts
+export function summarizeStoredLayerEvidence(unstored: ReadonlySet<string> | undefined, ...holders: readonly EvidenceHolder[]): EvidenceSummary | undefined
+```
+
+Like {@link summarizeLayerEvidence}, knowing which bundle keys the layer names but does not hold (see {@link summarizeStoredEvidence}).
+
 #### `isVideoAttachment`
 
 ```ts
@@ -1804,6 +1822,16 @@ export function bundleKeysOf(holder: EvidenceHolder): string[]
 
 Every distinct bundle key (`bundleKey`) that the attachments of a holder, its steps and its attempts point to, so a layer's files can be copied. Attachments held by `uri` or inline `text` have none.
 
+#### `unstoredKeys`
+
+```ts
+export function unstoredKeys(layer: EvidenceHolder & { readonly parts: readonly (EvidenceHolder & { readonly checks: readonly EvidenceHolder[] })[]; readonly bundle: Readonly<Record<string, unknown>> }): string[]
+```
+
+The bundle keys that attachments of a layer, its parts and their checks name, but that the layer's `bundle` index does
+not hold: files the producer referenced and never stored. Empty for a layer whose every file is there. This reads only
+the index, never a blob, so it is cheap enough to ask of every layer as it is read.
+
 ### Live evidence from running suites
 
 Source: `packages/reports/src/evidence-live.ts`. Import: `@crvouga/atlas/reports/evidence-live`
@@ -2109,7 +2137,7 @@ export type VideoRef = {
   readonly tags?: readonly string[];
   /** What identifies the bytes (digest, bundle key or address); references with the same one are the same recording. Absent when `available` is false. */
   readonly recording?: string;
-  /** False when the reference names no bytes at all (no bundle key, address or inline text), so it can never be played. */
+  /** False when the reference names no bytes at all (no bundle key, address or inline text), or a bundle key that the layer's own bundle does not hold, so it can never be played. */
   readonly available: boolean;
   /** What this recording shows, as declared for it: its own `scope`, else its attempt's, check's, part's or layer's. Absent when none declared one: its scope is unknown, not live, physical or audible. */
   readonly scope?: EvidenceScope;
@@ -2128,7 +2156,7 @@ export type VideoCounts = {
   readonly references: number;
   /** Distinct playable recordings: identical bytes referenced twice are one. */
   readonly recordings: number;
-  /** References that name no bytes. A report that predates availability metadata has none of these and none of the others. */
+  /** References that name no bytes, or bytes the layer does not hold. A report that predates availability metadata has none of these and none of the others. */
   readonly unavailable: number;
 };
 ```
@@ -4870,7 +4898,7 @@ Reads one captured file (a screenshot, a frame, a recorded report) of a layer, o
 export function layerSummary(layer: ReportLayer): LayerSummary
 ```
 
-Drops a layer's bundle index, leaving the summary composition reads.
+Drops a layer's bundle index, leaving the summary composition reads, plus the bundle keys its attachments name that the bundle does not hold (`unstored`), so a video that was never stored is not counted as a recording.
 
 ### Layer types
 
@@ -4931,6 +4959,8 @@ export type LayerSummary = LayerExtras & {
   readonly source: SourceRecord | null;
   /** Where the layer's timelines and captured files are read from: its own bundle (`layer`), an older commit-keyed recording (`commit`), or nowhere (`null`). */
   readonly recording: { readonly type: 'layer' } | { readonly type: 'commit'; readonly sha: string; readonly source: string } | null;
+  /** Bundle keys that attachments name but the layer's bundle does not hold, read from the layer's index when its summary was made (see `unstoredKeys`). Absent when there are none. In memory only: never part of the layer document. */
+  readonly unstored?: readonly string[];
 };
 ```
 
@@ -8162,6 +8192,319 @@ export async function readRunFile(store: ObjectStore, id: string, relative: stri
 
 Reads one file of a run, or `null` when it is missing. Also `null`, without reading, for an `id` that is not a plain run name or a `relative` path with an empty, `.`, `..` or backslash segment, so a path can never leave the run.
 
+### Run comparison
+
+Source: `packages/reports/src/run-compare.ts`. Import: `@crvouga/atlas/reports/run-compare`
+
+Answers "what changed between these two runs?" without claiming more than the evidence supports. A run is a pinned
+source: the comparison reads two {@link RunDetail}s as they are and never reaches for a newer run. Pure functions,
+safe in a browser; nothing here changes a status, a slot or a winner.
+
+The rules, each of which keeps a comparison from looking more certain than it is:
+
+- **Identity is explicit.** Results are paired by the check they record (`TestSummary.key`: spec path and check id),
+  then by the condition they were recorded under (kind, producer, environment, parameters). A title is never what
+  pairs two results. A result with the same check but another condition is paired as well, marked as recorded under
+  different conditions, and is not framed as the same experiment. Results of one side with no partner are `added`
+  (only the candidate has them) or `removed` (only the baseline has them); a `removed` result is not a regression,
+  because the candidate may not have run, renamed or dropped it, and the comparison cannot tell which.
+- **Not run is not regressed.** A result that did not run, or is still running, on either side is `unknown`; it is
+  counted as neither a pass nor a failure. Only a completed result against a completed result is a movement.
+- **A changed promise is not the same experiment.** When the wording of a behavior differs between the two sides, the
+  pair is `not-comparable` and both outcomes are shown beside the changed promise instead of as a regression or a
+  recovery. A different spec-file revision with the same wording is disclosed beside the movement: a fingerprint is
+  of the whole file, so it cannot say that this behavior's own promise changed. A revision that was not recorded is
+  unknown, never the same.
+- **Figures are compared like for like.** Product latency and reliability are compared only when both sides were
+  recorded under the same conditions and support the figure; otherwise the reason is given and no improvement is
+  computed. A metric only one side has is named, not compared.
+
+See docs/ui.md
+
+#### `COMPARE_GROUPS`
+
+```ts
+export const COMPARE_GROUPS = …
+```
+
+Where a pair of results (or one result with no partner) stands, in the order a review reads them.
+
+#### `CompareGroup`
+
+```ts
+export type CompareGroup = (typeof COMPARE_GROUPS)[number];
+```
+
+One of {@link COMPARE_GROUPS}.
+
+#### `COMPARE_GROUP_TEXT`
+
+```ts
+export const COMPARE_GROUP_TEXT: Readonly<Record<CompareGroup, { readonly label: string; readonly meaning: string }>>
+```
+
+What each group is called and what it does and does not say.
+
+#### `PromiseState`
+
+```ts
+export type PromiseState = 'same' | 'changed' | 'revision-differs' | 'unknown';
+```
+
+Whether the promise (what the behavior says should happen) is the same on both sides: `same` (same wording and the
+same recorded spec revision), `changed` (the wording differs), `revision-differs` (same wording, another spec-file
+revision), or `unknown` (a revision was not recorded on a side, so nothing can be said).
+
+#### `PromiseReading`
+
+```ts
+export type PromiseReading = {
+  readonly state: PromiseState;
+  readonly baselineTitle: string | null;
+  readonly candidateTitle: string | null;
+  readonly baselineRevision: string | null;
+  readonly candidateRevision: string | null;
+  /** In words, for a person. */
+  readonly reasons: readonly string[];
+};
+```
+
+What is known about whether the promise changed, and why.
+
+#### `Condition`
+
+```ts
+export type Condition = {
+  readonly kind: string | null;
+  readonly producer: string;
+  readonly environment: string | null;
+  readonly parameters: Readonly<Record<string, string>>;
+};
+```
+
+The condition a result was recorded under: with the check, what makes it its own result.
+
+#### `conditionOf`
+
+```ts
+export function conditionOf(result: Pick<RunResult, 'kind' | 'producer' | 'environment' | 'parameters'>): Condition
+```
+
+The condition of a result. A server that does not send `kind` leaves it null: unknown, equal only to another unknown.
+
+#### `conditionId`
+
+```ts
+export function conditionId(condition: Condition): string
+```
+
+A stable text for a condition, equal exactly when two conditions are.
+
+#### `conditionDifferences`
+
+```ts
+export function conditionDifferences(baseline: Condition, candidate: Condition): string[]
+```
+
+How two conditions differ, in words; empty when they are the same.
+
+#### `ComparedResult`
+
+```ts
+export type ComparedResult = {
+  /** Identifies the pair within one comparison: the check, and the condition of each side. Stable for the same two runs. */
+  readonly id: string;
+  /** The check's identity (`TestSummary.key`). */
+  readonly key: string;
+  /** The check's title: the candidate's when it has one, else the baseline's. */
+  readonly title: string;
+  readonly specPath: string | null;
+  readonly baseline: RunResult | null;
+  readonly candidate: RunResult | null;
+  readonly group: CompareGroup;
+  /** One sentence saying what happened, and what is not known. */
+  readonly summary: string;
+  readonly promise: PromiseReading;
+  /** How the two conditions differ, in words, when both sides have a result and they were recorded under different conditions. */
+  readonly conditions: readonly string[];
+  /** What a reader must not assume about this pair, in words. */
+  readonly caveats: readonly string[];
+};
+```
+
+One behavior's results on the two sides.
+
+#### `RunComparison`
+
+```ts
+export type RunComparison = {
+  readonly baseline: RunSummary;
+  readonly candidate: RunSummary;
+  /** Every pair and every unpaired result, in the order of {@link COMPARE_GROUPS}, then spec and title. */
+  readonly results: readonly ComparedResult[];
+  readonly counts: Readonly<Record<CompareGroup, number>>;
+  /** What a reader must not assume about the two runs as a whole: the same run on both sides, different branches, a candidate older than the baseline. */
+  readonly notes: readonly string[];
+};
+```
+
+The whole comparison of two pinned runs.
+
+#### `shortRevision`
+
+```ts
+export function shortRevision(revision: string | null | undefined): string
+```
+
+Short form of a spec fingerprint, to show.
+
+#### `recordedTitle`
+
+```ts
+export function recordedTitle(result: Pick<RunResult, 'title' | 'recordedTitle'>): string
+```
+
+The wording a run recorded for a check: what that run's layer called it, else the check's newest title.
+
+#### `compareRuns`
+
+```ts
+export function compareRuns(baseline: RunDetail, candidate: RunDetail): RunComparison
+```
+
+Pairs the results of two runs and reads what changed. Results are paired by check identity and then by condition (see
+the module comment); titles never pair two results. Each side's results are those the run recorded last for each
+check and condition (`RunDetail.results`).
+
+- `baseline`: The run to compare against.
+
+- `candidate`: The run under review. Both are used as given: a newer run never replaces either.
+
+**Returns** Every result of either run, grouped by what changed, with the reasons.
+
+**Example**
+
+```ts
+const comparison = compareRuns(await runDetail('run:ci-100'), await runDetail('run:ci-101'));
+comparison.results.filter(item => item.group === 'regressed');
+```
+
+#### `MetricComparison`
+
+```ts
+export type MetricComparison =
+  | { readonly state: 'compared'; readonly name: string; readonly baseline: LatencySummary; readonly candidate: LatencySummary; readonly change: LatencyChange; readonly reasons: readonly string[] }
+  | { readonly state: 'incompatible'; readonly name: string; readonly baseline: LatencySummary; readonly candidate: LatencySummary; readonly change: LatencyChange | null; readonly reasons: readonly string[] }
+  | { readonly state: 'baseline-only' | 'candidate-only'; readonly name: string; readonly summary: LatencySummary; readonly reasons: readonly string[] };
+```
+
+What one product-latency metric shows on the two sides.
+
+#### `compareProductLatency`
+
+```ts
+export function compareProductLatency(baseline: readonly LatencyEntry[], candidate: readonly LatencyEntry[], conditionsDiffer: readonly string[] = []): MetricComparison[]
+```
+
+Compares the product latency recorded in two sets of results of one check, one per side. A metric is compared only
+when it is the same measurement (name, unit, boundaries) and both sides were measured on the same devices, in the
+same environments, with the same instrumentation and other conditions; otherwise it is `incompatible` and no change
+is computed, with the reasons. A metric only one side recorded is named as such. A side without any latency is unknown,
+not fast: it contributes no metric. `conditionsDiffer` names differences of the results' own conditions, which make
+every metric incompatible.
+
+#### `ReliabilityComparison`
+
+```ts
+export type ReliabilityComparison = {
+  /** True only when both figures are established, were counted in the same scope and against the same recorded spec revision. */
+  readonly comparable: boolean;
+  readonly baseline: ReliabilityRate;
+  readonly candidate: ReliabilityRate;
+  /** Candidate minus baseline, in percentage points, only when comparable. */
+  readonly points?: number;
+  /** `higher` or `lower` only when the two intervals do not overlap; `indistinguishable` when they do. Absent when not comparable. */
+  readonly direction?: 'higher' | 'lower' | 'indistinguishable';
+  /** Why no change is stated, or what qualifies one. */
+  readonly reasons: readonly string[];
+};
+```
+
+How reliable a check was on each side, and whether the two figures can be set against each other.
+
+#### `reliabilityScopeAt`
+
+```ts
+export function reliabilityScopeAt(result: Pick<RunResult, 'environment' | 'producer' | 'parameters' | 'specRevision'>, run: Pick<RunSummary, 'endedAtMs'>): ReliabilityScope
+```
+
+The scope that counts the independent executions of one result's check up to the end of its run: the same
+environment, producer and parameters, and the spec revision that result was recorded against (executions against
+another revision are other promises). A result that recorded no revision is counted across revisions, which
+{@link compareReliability} will not compare. Use it for each side, with that side's result and run.
+
+#### `compareReliability`
+
+```ts
+export function compareReliability(baseline: ReliabilityDocument, candidate: ReliabilityDocument, conditionsDiffer: readonly string[] = []): ReliabilityComparison
+```
+
+Sets the reliability of a check up to the baseline run against its reliability up to the candidate run. A change is
+stated only when both samples are established (at least `minimumExecutions` independent executions), both counted the
+same recorded spec revision (see {@link reliabilityScopeAt}) and the conditions match; whether the difference exceeds the uncertainty is
+said by `direction`. Otherwise each figure stands alone and the reasons say why.
+
+#### `StepLike`
+
+```ts
+export type StepLike = { readonly name: string; readonly offsetMs?: number | undefined };
+```
+
+The part of a recorded step this module reads.
+
+#### `AlignedStep`
+
+```ts
+export type AlignedStep = {
+  readonly name: string;
+  /** Milliseconds after the check began, on each side; absent when that side did not record it or its time. */
+  readonly baselineOffsetMs?: number;
+  readonly candidateOffsetMs?: number;
+  readonly onBaseline: boolean;
+  readonly onCandidate: boolean;
+};
+```
+
+One step, found on one or both sides.
+
+#### `StepAlignment`
+
+```ts
+export type StepAlignment = {
+  /** `aligned`: some step is on both sides with a time on both (a recording can be sought to it); `unaligned`: otherwise. */
+  readonly state: 'aligned' | 'unaligned';
+  readonly steps: readonly AlignedStep[];
+  /** The steps with a time on both sides: where both recordings can be sought together. */
+  readonly seekable: readonly (AlignedStep & { readonly baselineOffsetMs: number; readonly candidateOffsetMs: number })[];
+  /** Whether the two lists of step names differ (the steps themselves are part of the promise). */
+  readonly differs: boolean;
+  /** In words, why the recordings are or are not aligned. */
+  readonly explanation: string;
+};
+```
+
+How the recorded steps of two results line up.
+
+#### `alignSteps`
+
+```ts
+export function alignSteps(baseline: readonly StepLike[], candidate: readonly StepLike[]): StepAlignment
+```
+
+Lines up the recorded steps of two results by name (the n-th step of a name on one side with the n-th of that name on
+the other), keeping each side's own time origin: an offset is milliseconds after that side's check began. Nothing is
+aligned by position or by fuzzy wording. When nothing lines up, `explanation` says why.
+
 ### S3 object store
 
 Source: `packages/reports/src/s3-store.ts`. Import: `@crvouga/atlas/reports/s3-store`
@@ -9620,6 +9963,14 @@ export type RunResult = {
   readonly detail: string;
   /** Every video its evidence references, with who holds it (check, retry attempt or step). Absent when there is none. */
   readonly videos?: readonly VideoRef[];
+  /** The kind of the part that held it (`behavior`, `checks`…). With the producer, environment and parameters it is the condition the result was recorded under. Absent from an older server. */
+  readonly kind?: string;
+  /** The other choices that made it its own result (a device size, a locale…). Absent when there are none, and from an older server. */
+  readonly parameters?: Readonly<Record<string, string>>;
+  /** The fingerprint of its spec file the layer recorded: results with different ones were recorded against different revisions of the spec. Absent means not recorded, never "the same". */
+  readonly specRevision?: string;
+  /** The title this run's layer gave the check. `title` is the newest title the check has in the whole history, so a check that was reworded shows its newest wording in both runs; this is the wording as recorded. Absent from an older server. */
+  readonly recordedTitle?: string;
 };
 ```
 

package/docs/ui.md

@@ -143,6 +143,35 @@ environment) is unknown unless the producer declared it under the `atlas.failure
 wording. Records of one source whose outcome changed are listed oldest first, and a later pass is never called a fix.
 In the explorer the chosen source is the `source` search parameter (a layer id; unlike `layer` it does not pin a run).
 
+## Compare two runs
+
+The **Compare** tab (`explorer/CompareView.tsx`, `explorer/ComparePair.tsx`, `@crvouga/atlas/reports/run-compare`) sets a
+baseline run against a candidate run. Both are pinned in the address (`base` and `cand`, run keys; `change` is the group
+shown and `pair` the behavior opened), so a link reproduces the comparison and a newer run never replaces either. A run
+that is no longer in the report is said to be missing and is never swapped for another. A run's page offers "Compare with
+this run as the baseline / candidate".
+
+- Results are paired by the check they record (spec path and check id), then by the condition they were recorded under
+  (kind, producer, environment, parameters). A title never pairs two results. The same check under another condition is
+  paired and marked "Conditions differ".
+- Every behavior is in one group: newly failing, recovered, still failing, promise or conditions differ, not run or
+  unfinished, not in the candidate run, not in the baseline run, same outcome. A result the candidate did not run is
+  unknown, never regressed or passed; one only the baseline has is "not in the candidate run" (not run, renamed or
+  dropped: the comparison cannot tell). A reworded behavior, or one recorded under different conditions, is not framed as
+  a regression or recovery: both outcomes are shown beside the changed promise. A different spec-file revision with the
+  same wording is disclosed beside the movement (a fingerprint is of the whole file); a revision that was not recorded is
+  unknown, never the same.
+- Opening a behavior shows Mantine `Tabs`: **Promise** (wording, spec revision, recorded steps with the ones only one side
+  has), **Outcomes** (each side's own provenance, execution time and failure), **Recordings** (one `<video>` per side with
+  `preload="metadata"`, read only while that tab is on screen; each keeps its own attempt, spec revision and time
+  origin, they are not synchronized, and "Go to a step in both recordings" seeks each to its own recorded step time when
+  the steps are shared by name and timed; otherwise the page says why they are not aligned) and **Figures**.
+- Figures are compared like for like. Product latency is compared only for the same measurement on the same devices,
+  environments and instrumentation, and reliability only when both sides count at least the minimum of independent
+  executions against the same recorded spec revision (and a difference inside the uncertainty is "cannot be told apart").
+  Otherwise the page says "Not compared" with the reasons and computes no change. Test execution time is labelled as such
+  and is never product latency.
+
 ## Watch a filtered set as a playlist
 
 The Artifacts tab and an opened run have a **Watch these N results** button. It opens a Mantine `Modal` (`explorer/playlist.tsx`)
@@ -291,7 +320,7 @@ The narrowly scoped exceptions are:
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
 | UI catalog | Captured UI states per profile as cards with coverage badges, filters in the index column, surface coverage, an open state with its profiles side by side and where each picture came from, manifest download | AppShell navbar, SimpleGrid, Card, Image, MultiSelect, TextInput, Drawer, Accordion, Table, Badge, Alert |
-| Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
+| Project home | Tests/runs/artifacts/compare tabs, a two-run comparison with promise, outcome, recording and figure tabs per behavior, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
 | Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |

package/llms-full.txt

@@ -4562,6 +4562,35 @@ environment) is unknown unless the producer declared it under the `atlas.failure
 wording. Records of one source whose outcome changed are listed oldest first, and a later pass is never called a fix.
 In the explorer the chosen source is the `source` search parameter (a layer id; unlike `layer` it does not pin a run).
 
+## Compare two runs
+
+The **Compare** tab (`explorer/CompareView.tsx`, `explorer/ComparePair.tsx`, `@crvouga/atlas/reports/run-compare`) sets a
+baseline run against a candidate run. Both are pinned in the address (`base` and `cand`, run keys; `change` is the group
+shown and `pair` the behavior opened), so a link reproduces the comparison and a newer run never replaces either. A run
+that is no longer in the report is said to be missing and is never swapped for another. A run's page offers "Compare with
+this run as the baseline / candidate".
+
+- Results are paired by the check they record (spec path and check id), then by the condition they were recorded under
+  (kind, producer, environment, parameters). A title never pairs two results. The same check under another condition is
+  paired and marked "Conditions differ".
+- Every behavior is in one group: newly failing, recovered, still failing, promise or conditions differ, not run or
+  unfinished, not in the candidate run, not in the baseline run, same outcome. A result the candidate did not run is
+  unknown, never regressed or passed; one only the baseline has is "not in the candidate run" (not run, renamed or
+  dropped: the comparison cannot tell). A reworded behavior, or one recorded under different conditions, is not framed as
+  a regression or recovery: both outcomes are shown beside the changed promise. A different spec-file revision with the
+  same wording is disclosed beside the movement (a fingerprint is of the whole file); a revision that was not recorded is
+  unknown, never the same.
+- Opening a behavior shows Mantine `Tabs`: **Promise** (wording, spec revision, recorded steps with the ones only one side
+  has), **Outcomes** (each side's own provenance, execution time and failure), **Recordings** (one `<video>` per side with
+  `preload="metadata"`, read only while that tab is on screen; each keeps its own attempt, spec revision and time
+  origin, they are not synchronized, and "Go to a step in both recordings" seeks each to its own recorded step time when
+  the steps are shared by name and timed; otherwise the page says why they are not aligned) and **Figures**.
+- Figures are compared like for like. Product latency is compared only for the same measurement on the same devices,
+  environments and instrumentation, and reliability only when both sides count at least the minimum of independent
+  executions against the same recorded spec revision (and a difference inside the uncertainty is "cannot be told apart").
+  Otherwise the page says "Not compared" with the reasons and computes no change. Test execution time is labelled as such
+  and is never product latency.
+
 ## Watch a filtered set as a playlist
 
 The Artifacts tab and an opened run have a **Watch these N results** button. It opens a Mantine `Modal` (`explorer/playlist.tsx`)
@@ -4710,7 +4739,7 @@ The narrowly scoped exceptions are:
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
 | UI catalog | Captured UI states per profile as cards with coverage badges, filters in the index column, surface coverage, an open state with its profiles side by side and where each picture came from, manifest download | AppShell navbar, SimpleGrid, Card, Image, MultiSelect, TextInput, Drawer, Accordion, Table, Badge, Alert |
-| Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
+| Project home | Tests/runs/artifacts/compare tabs, a two-run comparison with promise, outcome, recording and figure tabs per behavior, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
 | Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/atlas",
-  "version": "1.14.0",
+  "version": "1.15.0",
   "description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
   "license": "MIT",
   "type": "module",

package/packages/schema/schemas/effective-delta.schema.json

@@ -262,7 +262,7 @@
                           },
                           "available": {
                             "type": "boolean",
-                            "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                            "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                           },
                           "scope": {
                             "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",
@@ -741,7 +741,7 @@
                         },
                         "available": {
                           "type": "boolean",
-                          "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                          "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                         },
                         "scope": {
                           "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",

package/packages/schema/schemas/effective-evidence.schema.json

@@ -262,7 +262,7 @@
                           },
                           "available": {
                             "type": "boolean",
-                            "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                            "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                           },
                           "scope": {
                             "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",
@@ -741,7 +741,7 @@
                         },
                         "available": {
                           "type": "boolean",
-                          "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                          "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                         },
                         "scope": {
                           "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",

package/packages/schema/schemas/evidence-page.schema.json

@@ -249,7 +249,7 @@
                     },
                     "available": {
                       "type": "boolean",
-                      "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                      "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                     },
                     "scope": {
                       "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",

package/packages/schema/schemas/layer-rows.schema.json

@@ -253,7 +253,7 @@
                     },
                     "available": {
                       "type": "boolean",
-                      "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                      "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                     },
                     "scope": {
                       "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",

package/packages/schema/schemas/report-run.schema.json

@@ -161,7 +161,7 @@
               "type": "integer",
               "minimum": 0,
               "maximum": 9007199254740991,
-              "description": "References that name no bytes"
+              "description": "References that name no bytes, or bytes the layer does not hold"
             }
           },
           "required": [
@@ -314,7 +314,7 @@
                 },
                 "available": {
                   "type": "boolean",
-                  "description": "False when the reference names no bytes (no bundle key, address or inline text)"
+                  "description": "False when the reference names no bytes: no bundle key, address or inline text, or a bundle key that the layer's own bundle does not hold"
                 },
                 "scope": {
                   "description": "What this recording shows as its producer declared it: its own `scope`, else its attempt's, check's, part's or layer's. Absent: the scope is unknown, never live, physical or audible",
@@ -420,6 +420,28 @@
               ],
               "additionalProperties": {}
             }
+          },
+          "kind": {
+            "description": "The kind of the part that held it. With the producer, environment and parameters it is the condition the result was recorded under. Absent from an older server",
+            "type": "string"
+          },
+          "parameters": {
+            "description": "The other choices that made it its own result (a device size, a locale…). Absent when there are none, and from an older server",
+            "type": "object",
+            "propertyNames": {
+              "type": "string"
+            },
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "specRevision": {
+            "description": "The fingerprint of its spec file the layer recorded. Absent means not recorded, never \"the same\"",
+            "type": "string"
+          },
+          "recordedTitle": {
+            "description": "The title this run's layer gave the check. `title` is the check's newest title in the whole history; this is the wording as recorded. Absent from an older server",
+            "type": "string"
           }
         },
         "required": [
@@ -494,7 +516,7 @@
                 "type": "integer",
                 "minimum": 0,
                 "maximum": 9007199254740991,
-                "description": "References that name no bytes"
+                "description": "References that name no bytes, or bytes the layer does not hold"
               }
             },
             "required": [

package/packages/schema/schemas/report-runs.schema.json

@@ -169,7 +169,7 @@
                 "type": "integer",
                 "minimum": 0,
                 "maximum": 9007199254740991,
-                "description": "References that name no bytes"
+                "description": "References that name no bytes, or bytes the layer does not hold"
               }
             },
             "required": [