@crvouga/atlas 1.7.0 → 1.9.0

package/docs/reference/schema.md

@@ -300,6 +300,17 @@ export const TestHistorySchema = …
 
 `GET /api/reports/tests/history`: one test and every result recorded for it (at most 100), newest first.
 
+#### `TestReliabilitySchema`
+
+```ts
+export const TestReliabilitySchema = …
+```
+
+`GET /api/reports/tests/reliability`: how reliable one test is inside a scope, and exactly how that figure was counted.
+The snapshots one run recorded for one producer, environment and parameter set are one independent execution; only
+`passed` and `failed` executions are trials; `notes` say when the sample is too small, pooled, selectively rerun or
+bounded. It reads results and never changes a status. It is not a production failure probability.
+
 #### `ReportRunSchema`
 
 ```ts
@@ -742,6 +753,22 @@ export type LayerRun = z.output<typeof LayerRunSchema>;
 
 The execution of a layer, parsed from `LayerRunSchema`.
 
+#### `PRODUCT_LATENCY_KIND`
+
+```ts
+export const PRODUCT_LATENCY_KIND = …
+```
+
+The `kind` of a measurement that is a named, user-visible product latency (see docs/rich-evidence.md#product-latency).
+
+#### `MeasurementBoundarySchema`
+
+```ts
+export const MeasurementBoundarySchema = …
+```
+
+The two ends of a measured interval, named in the producer's words: `{ from: 'submit pressed', to: 'confirmation visible' }`.
+
 #### `MeasurementSchema`
 
 ```ts
@@ -751,6 +778,18 @@ export const MeasurementSchema = …
 A number a check measured, such as a latency or a size, with its unit and an optional allowance. A measurement over
 its `limit` is reported and never changes the check's status.
 
+A measurement of `kind` `product-latency` is how quickly the product responded to a user, between two named events.
+It is not the check's duration (which includes setup, waits and pacing) and a number with no `kind` is never read as
+product latency. All the fields after `limit` are optional and describe the measurement; none decides a status.
+
+#### `Measurement`
+
+```ts
+export type Measurement = z.output<typeof MeasurementSchema>;
+```
+
+A measurement as stored in a layer, parsed from `MeasurementSchema`.
+
 #### `ExtensionsSchema`
 
 ```ts

package/docs/rich-evidence.md

@@ -109,6 +109,49 @@ Every field is optional, and every value except the booleans is an open string.
 - **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
   dropped, and the attachment, check and layer stay.
 
+## Product latency
+
+A test's `durationMs` includes fixture setup, deliberate waits, demo pacing, retries and cleanup, so it says how long the
+test ran and nothing about how quickly the product responded. Name the numbers that do, as measurements of `kind`
+`product-latency` between two events:
+
+```ts
+measurements: [
+  { name: 'Fixture setup', value: 20_000, unit: 'ms' },              // just a number: never read as latency
+  {
+    name: 'Confirmation', value: 500, unit: 'ms', kind: 'product-latency',
+    from: 'submit pressed', to: 'confirmation visible',             // the boundaries are part of the claim
+    limit: 800,                                                     // the budget
+    source: 'app-instrumentation', instrumentation: 'spans-v2',
+    labels: { device: 'pixel-8', build: '4821' }                    // what it is comparable with
+  }
+]
+```
+
+The test page shows **Product latency** apart from the test statistics, whose durations are labelled as test execution
+time. Every field after `unit` is optional and none of them decides a status, a slot or a winner.
+
+- **Only named measurements are latency.** A check's duration, a video's length, a step and a number with no `kind` are
+  never product latency. A test that recorded none shows that its latency is unknown, not fast.
+- **Boundaries are part of the claim.** Without both `from` and `to` the numbers are shown but no percentile or product
+  claim is made, and the page says so. When the interval you measured is only part of what the name claims, say what it
+  claims in `intended` (`{ "to": "audible response" }` on a measurement that ends at `tts dispatch`) and the viewer labels
+  it a partial interval, never the complete one.
+- **Percentiles need samples.** p50 needs 5 observations and p95 needs 20; fewer shows `Withheld` with the count it has.
+  One observation shows no spread or confidence.
+- **A timeout is a sample.** `outcome: "timeout"` or `"failed"` (anything but `observed`) means `value` is how long the
+  observation waited. It stays in the sample count and the percentiles, counts as over budget whenever a `limit` is set,
+  and marks the figures as lower bounds, so a stuck product never looks healthy.
+- **Groups.** The viewer groups observations by `device`, `build` or the UTC day they were recorded, taking `device` and
+  `build` from the measurement's `labels`, else the result's labels, parameters, environment and commit. Each group has
+  its own sample count, p50/p95, budget and breaches. A group that names nothing is `unknown device` or `unknown build`.
+- **Baselines.** A group is compared with a chosen baseline by absolute and relative change of p50 (and p95), or of the
+  newest observation when percentiles are not supported on both sides. Measurements with different names, units or
+  boundaries are not compared. A change of `instrumentation`, environment, other labels or parameters, or budget between
+  the two sides is listed beside the change.
+- **Reading it from the API.** `GET /api/reports/tests/history` entries carry `latency` (their product-latency
+  measurements) next to `durationMs`. The pure functions are in `@crvouga/atlas/reports/latency`.
+
 ## Finding runs and artifacts
 
 The viewer's **Runs** and **Artifacts** tabs and the API answer one query, so a link means the same thing everywhere.

package/docs/server.md

@@ -374,6 +374,7 @@ with `format=markdown` (or `Accept: text/markdown`), and takes the project prefi
 | Every test, a summary (by status, environment, producer, domain), insights (failing, flaky, slowest) and facets | `GET /api/reports/tests` |
 | Only some: `q` (all words), `status` (or `flaky`), `environment`, `producer`, `domain`, `spec`, `tag`, `run`; `sort`, `offset`, `limit` | `GET /api/reports/tests?status=failed&environment=ios` |
 | One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
+| How reliable one test is in a scope (environment, producer, parameters, branch, commit, spec revision, time window): independent executions, first-try and eventual success, what was left out and why | `GET /api/reports/tests/reliability?key=<key>&environment=phone` |
 | Why Specs, domains and counts disagree: ambiguous spec paths, shared domains, gaps, a reconciliation of the totals | `GET /api/reports/tests/health` |
 | The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
 | Recorded results that carry evidence, across the whole history, found by tag, media type, label, feature, environment, branch, revision, outcome and time | `GET /api/reports/artifacts?artifactTag=demo&type=video` |
@@ -388,6 +389,21 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
   `stats` describes its recorded history: pass rate, flips, whether it is flaky and why (it passed and failed on the
   same commit, passed only after a retry, or its producer said so), durations (avg, p50, p95, min, max, last), runs
   and branches.
+- The **reliability** document qualifies `stats.passRate` instead of repeating it. The snapshots one run recorded for one
+  producer, environment and parameter set (running, finished, enriched) are **one independent execution**, counted by its last
+  word with the tries it recorded; `firstAttempt` counts executions whose first try passed, `eventual` those that passed in the
+  end, and an execution that recorded no tries (`attemptStatuses` on its evidence) is never assumed to have passed first time.
+  Choose the scope with `environment`, `producer`, `parameter=key=value`, `branch`, `sha`, `specRevision`, `since` and `until`:
+  other environments never add to a phone's sample. By default only the newest recorded spec revision (the `fingerprint` of its
+  spec text, `specRevision` on each result) counts and earlier ones appear as `historical` context; `revision=any` pools them
+  and says so. Only `passed` and `failed` executions are trials; `not-run`, unfinished and unknown ones are named under
+  `excluded` and never become a pass. `sample` is `none`, `insufficient` (under `minimumExecutions`) or `sufficient`, a count
+  only: `notes` say when the figure pools environments, producers or branches, when failed cases were published again
+  (`selected-reruns`), and when history is bounded to the newest 100 results. `executions` is the denominator, one row per
+  independent execution with its tries and snapshots, so the same query can be inspected and shared. The Wilson interval states
+  its assumptions. It describes the checks Atlas holds; it is not a production failure probability. The viewer shows the same
+  figures under a test (scope in the address as `relEnv`, `relProducer`, `relBranch`, `relParam`, `relSince`, `relUntil`,
+  `relRevision=any`, `relOpen=true`).
 - The **health** document explains the numbers instead of adding one. It keeps spec files, defined specs, specs with an
   executed test, tests, outlines, examples and per-environment observations apart, and walks from the Specs total to
   the Tests total. Two spec paths are one logical spec only when a mapping says so
@@ -453,6 +469,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/reports/tests/health` | read | Why specs, domains and counts disagree. |
 | `GET /api/reports/tests/history` | read | One test and every result recorded for it. |
 | `GET /api/projects/{project}/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/reports/tests/reliability` | read | How reliable one test is in a scope, and how that was counted. |
+| `GET /api/projects/{project}/reports/tests/reliability` | read | How reliable one test is in a scope, and how that was counted. |
 | `GET /api/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
 | `GET /api/projects/{project}/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
 | `POST /api/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |

package/docs/ui.md

@@ -87,6 +87,25 @@ when Watch is on screen.
 - The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
   none are listed under "No domain declared"; the Tests view reports domain problems).
 
+## Reliability
+
+An open test shows its reliability (`explorer/Reliability.tsx`, worked out by `reports/reliability` from the history the page
+already holds). Never show a bare percentage: it always comes with the number of independent executions it rests on, the
+"Insufficient evidence" badge below the minimum (one pass is not established reliability), the scope (environment,
+producer, branch, parameters, time window, spec revision) and the warning notes (pooled environments, failed cases
+published again, bounded history). First-try success is over executions that recorded their tries only; a result that
+did not run or did not finish is excluded by name and never counted as a pass. The scope and the open executions table
+are in the address (`rel*`), so a link reproduces the figure and the executions behind it.
+
+## Product latency
+
+A test's duration and its product latency are different things and the viewer never mixes them (`explorer/ProductLatency.tsx`,
+`@crvouga/atlas/reports/latency`). Statistics and the trend are labelled test execution time. Product latency is read only from
+measurements named `product-latency`; it shows boundaries, group (device, build, day), samples, budget and breaches, and a
+baseline comparison that lists changes of instrumentation or conditions. Percentiles and product claims that the evidence
+cannot support are shown as `Withheld` with the reason, a partial interval is badged, timeouts count as samples and breaches,
+and a test with no latency says it is unknown, not fast.
+
 ## Outcome summary
 
 A test or behavior detail states what the **selected** result says apart from what the other current sources say
@@ -170,7 +189,7 @@ The narrowly scoped exceptions are:
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
 | Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
-| Test detail | Current outcomes, statistics, duration graph, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
+| Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
 | Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |

package/llms-full.txt

@@ -1096,6 +1096,49 @@ Every field is optional, and every value except the booleans is an open string.
 - **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
   dropped, and the attachment, check and layer stay.
 
+## Product latency
+
+A test's `durationMs` includes fixture setup, deliberate waits, demo pacing, retries and cleanup, so it says how long the
+test ran and nothing about how quickly the product responded. Name the numbers that do, as measurements of `kind`
+`product-latency` between two events:
+
+```ts
+measurements: [
+  { name: 'Fixture setup', value: 20_000, unit: 'ms' },              // just a number: never read as latency
+  {
+    name: 'Confirmation', value: 500, unit: 'ms', kind: 'product-latency',
+    from: 'submit pressed', to: 'confirmation visible',             // the boundaries are part of the claim
+    limit: 800,                                                     // the budget
+    source: 'app-instrumentation', instrumentation: 'spans-v2',
+    labels: { device: 'pixel-8', build: '4821' }                    // what it is comparable with
+  }
+]
+```
+
+The test page shows **Product latency** apart from the test statistics, whose durations are labelled as test execution
+time. Every field after `unit` is optional and none of them decides a status, a slot or a winner.
+
+- **Only named measurements are latency.** A check's duration, a video's length, a step and a number with no `kind` are
+  never product latency. A test that recorded none shows that its latency is unknown, not fast.
+- **Boundaries are part of the claim.** Without both `from` and `to` the numbers are shown but no percentile or product
+  claim is made, and the page says so. When the interval you measured is only part of what the name claims, say what it
+  claims in `intended` (`{ "to": "audible response" }` on a measurement that ends at `tts dispatch`) and the viewer labels
+  it a partial interval, never the complete one.
+- **Percentiles need samples.** p50 needs 5 observations and p95 needs 20; fewer shows `Withheld` with the count it has.
+  One observation shows no spread or confidence.
+- **A timeout is a sample.** `outcome: "timeout"` or `"failed"` (anything but `observed`) means `value` is how long the
+  observation waited. It stays in the sample count and the percentiles, counts as over budget whenever a `limit` is set,
+  and marks the figures as lower bounds, so a stuck product never looks healthy.
+- **Groups.** The viewer groups observations by `device`, `build` or the UTC day they were recorded, taking `device` and
+  `build` from the measurement's `labels`, else the result's labels, parameters, environment and commit. Each group has
+  its own sample count, p50/p95, budget and breaches. A group that names nothing is `unknown device` or `unknown build`.
+- **Baselines.** A group is compared with a chosen baseline by absolute and relative change of p50 (and p95), or of the
+  newest observation when percentiles are not supported on both sides. Measurements with different names, units or
+  boundaries are not compared. A change of `instrumentation`, environment, other labels or parameters, or budget between
+  the two sides is listed beside the change.
+- **Reading it from the API.** `GET /api/reports/tests/history` entries carry `latency` (their product-latency
+  measurements) next to `durationMs`. The pure functions are in `@crvouga/atlas/reports/latency`.
+
 ## Finding runs and artifacts
 
 The viewer's **Runs** and **Artifacts** tabs and the API answer one query, so a link means the same thing everywhere.
@@ -1720,6 +1763,7 @@ with `format=markdown` (or `Accept: text/markdown`), and takes the project prefi
 | Every test, a summary (by status, environment, producer, domain), insights (failing, flaky, slowest) and facets | `GET /api/reports/tests` |
 | Only some: `q` (all words), `status` (or `flaky`), `environment`, `producer`, `domain`, `spec`, `tag`, `run`; `sort`, `offset`, `limit` | `GET /api/reports/tests?status=failed&environment=ios` |
 | One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
+| How reliable one test is in a scope (environment, producer, parameters, branch, commit, spec revision, time window): independent executions, first-try and eventual success, what was left out and why | `GET /api/reports/tests/reliability?key=<key>&environment=phone` |
 | Why Specs, domains and counts disagree: ambiguous spec paths, shared domains, gaps, a reconciliation of the totals | `GET /api/reports/tests/health` |
 | The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
 | Recorded results that carry evidence, across the whole history, found by tag, media type, label, feature, environment, branch, revision, outcome and time | `GET /api/reports/artifacts?artifactTag=demo&type=video` |
@@ -1734,6 +1778,21 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
   `stats` describes its recorded history: pass rate, flips, whether it is flaky and why (it passed and failed on the
   same commit, passed only after a retry, or its producer said so), durations (avg, p50, p95, min, max, last), runs
   and branches.
+- The **reliability** document qualifies `stats.passRate` instead of repeating it. The snapshots one run recorded for one
+  producer, environment and parameter set (running, finished, enriched) are **one independent execution**, counted by its last
+  word with the tries it recorded; `firstAttempt` counts executions whose first try passed, `eventual` those that passed in the
+  end, and an execution that recorded no tries (`attemptStatuses` on its evidence) is never assumed to have passed first time.
+  Choose the scope with `environment`, `producer`, `parameter=key=value`, `branch`, `sha`, `specRevision`, `since` and `until`:
+  other environments never add to a phone's sample. By default only the newest recorded spec revision (the `fingerprint` of its
+  spec text, `specRevision` on each result) counts and earlier ones appear as `historical` context; `revision=any` pools them
+  and says so. Only `passed` and `failed` executions are trials; `not-run`, unfinished and unknown ones are named under
+  `excluded` and never become a pass. `sample` is `none`, `insufficient` (under `minimumExecutions`) or `sufficient`, a count
+  only: `notes` say when the figure pools environments, producers or branches, when failed cases were published again
+  (`selected-reruns`), and when history is bounded to the newest 100 results. `executions` is the denominator, one row per
+  independent execution with its tries and snapshots, so the same query can be inspected and shared. The Wilson interval states
+  its assumptions. It describes the checks Atlas holds; it is not a production failure probability. The viewer shows the same
+  figures under a test (scope in the address as `relEnv`, `relProducer`, `relBranch`, `relParam`, `relSince`, `relUntil`,
+  `relRevision=any`, `relOpen=true`).
 - The **health** document explains the numbers instead of adding one. It keeps spec files, defined specs, specs with an
   executed test, tests, outlines, examples and per-environment observations apart, and walks from the Specs total to
   the Tests total. Two spec paths are one logical spec only when a mapping says so
@@ -1799,6 +1858,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/reports/tests/health` | read | Why specs, domains and counts disagree. |
 | `GET /api/reports/tests/history` | read | One test and every result recorded for it. |
 | `GET /api/projects/{project}/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/reports/tests/reliability` | read | How reliable one test is in a scope, and how that was counted. |
+| `GET /api/projects/{project}/reports/tests/reliability` | read | How reliable one test is in a scope, and how that was counted. |
 | `GET /api/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
 | `GET /api/projects/{project}/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
 | `POST /api/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
@@ -2830,7 +2891,7 @@ carry any of these. Every one is optional.
 | `scope{}` | What the evidence does and does not show: surface, device kind, synthetic data, scripted dependencies, audio, pacing, what was asserted and its limits. Unknown when absent. See [Evidence scope](rich-evidence.md#evidence-scope). |
 | `links[]` | Where to read more: the CI run, the issue, the trace client. |
 | `extensions{}` | Structured data of your own or a tool's, under a name you own. |
-| `measurements[]` | Numbers, with units and an allowance (already part of the contract). |
+| `measurements[]` | Numbers, with units and an allowance (already part of the contract). A measurement of `kind` `product-latency` names a user-visible response time between two events; see [Product latency](rich-evidence.md#product-latency). |
 
 An attachment may carry its own `tags[]` (`demo`): shown as `#demo` and searchable; see
 [Tagging artifacts and demos](rich-evidence.md#tagging-artifacts-and-demos).
@@ -3936,6 +3997,25 @@ when Watch is on screen.
 - The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
   none are listed under "No domain declared"; the Tests view reports domain problems).
 
+## Reliability
+
+An open test shows its reliability (`explorer/Reliability.tsx`, worked out by `reports/reliability` from the history the page
+already holds). Never show a bare percentage: it always comes with the number of independent executions it rests on, the
+"Insufficient evidence" badge below the minimum (one pass is not established reliability), the scope (environment,
+producer, branch, parameters, time window, spec revision) and the warning notes (pooled environments, failed cases
+published again, bounded history). First-try success is over executions that recorded their tries only; a result that
+did not run or did not finish is excluded by name and never counted as a pass. The scope and the open executions table
+are in the address (`rel*`), so a link reproduces the figure and the executions behind it.
+
+## Product latency
+
+A test's duration and its product latency are different things and the viewer never mixes them (`explorer/ProductLatency.tsx`,
+`@crvouga/atlas/reports/latency`). Statistics and the trend are labelled test execution time. Product latency is read only from
+measurements named `product-latency`; it shows boundaries, group (device, build, day), samples, budget and breaches, and a
+baseline comparison that lists changes of instrumentation or conditions. Percentiles and product claims that the evidence
+cannot support are shown as `Withheld` with the reason, a partial interval is badged, timeouts count as samples and breaches,
+and a test with no latency says it is unknown, not fast.
+
 ## Outcome summary
 
 A test or behavior detail states what the **selected** result says apart from what the other current sources say
@@ -4019,7 +4099,7 @@ The narrowly scoped exceptions are:
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
 | Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
-| Test detail | Current outcomes, statistics, duration graph, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
+| Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
 | Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |
@@ -5156,6 +5236,54 @@ Scoped to one project. The same project may be selected on the unprefixed API us
 
 Responses: `200` The test and its history.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No test in the report has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
 
+### How reliable one test is in a scope, and how that was counted
+
+`GET /api/reports/tests/reliability` · role: read
+
+A success rate for one test that says what it counted. The snapshots one run recorded for one producer, environment and parameter set (running, finished, enriched) are one independent execution; its last word counts once with the tries it recorded. `eventual` is the share of executions that passed in the end, `firstAttempt` the share whose first try passed (over executions that recorded their tries; the rest are `unrecordedAttempts`, never assumed to have passed first time).
+
+Choose the scope with `environment`, `producer`, `parameter` (`key=value`), `branch`, `sha`, `specRevision`, `since` and `until`; executions elsewhere never count. By default only the newest recorded spec revision counts and earlier ones appear under `historical`; send `revision=any` to pool them. Only `passed` and `failed` executions are trials: `not-run` and unfinished ones are listed under `excluded` and never counted as a pass. `sample` is `none`, `insufficient` (fewer than `minimumExecutions`) or `sufficient`, a count only: `notes` say when environments, producers or branches are pooled, failed cases were published again, or history is bounded to the newest 100 results. `executions` is the denominator, open to inspection. The interval states its assumptions. This is not a production failure probability. Send `format=markdown` for Markdown.
+
+- `key` (query, required): The test's `key` from `/api/reports/tests`.
+- `environment` (query): Environments or devices to count. Comma-separated.
+- `producer` (query): Producers to count. Comma-separated.
+- `parameter` (query): Parameter choices `key=value`; all must match. Comma-separated.
+- `branch` (query): Branches to count. Comma-separated.
+- `sha` (query): Commits: a full id or a prefix of at least seven characters. Comma-separated.
+- `specRevision` (query): Only executions recorded against these spec fingerprints (overrides `revision`). Comma-separated.
+- `revision` (query): `any` pools every spec revision; otherwise only the newest recorded one counts.
+- `since` (query): Only executions at or after this time: epoch milliseconds or a date.
+- `until` (query): Only executions at or before this time: epoch milliseconds or a date.
+- `format` (query): `markdown` for Markdown instead of JSON.
+- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+
+Responses: `200` The reliability of the test in the scope.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No test in the report has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### How reliable one test is in a scope, and how that was counted
+
+`GET /api/projects/{project}/reports/tests/reliability` · role: read
+
+A success rate for one test that says what it counted. The snapshots one run recorded for one producer, environment and parameter set (running, finished, enriched) are one independent execution; its last word counts once with the tries it recorded. `eventual` is the share of executions that passed in the end, `firstAttempt` the share whose first try passed (over executions that recorded their tries; the rest are `unrecordedAttempts`, never assumed to have passed first time).
+
+Choose the scope with `environment`, `producer`, `parameter` (`key=value`), `branch`, `sha`, `specRevision`, `since` and `until`; executions elsewhere never count. By default only the newest recorded spec revision counts and earlier ones appear under `historical`; send `revision=any` to pool them. Only `passed` and `failed` executions are trials: `not-run` and unfinished ones are listed under `excluded` and never counted as a pass. `sample` is `none`, `insufficient` (fewer than `minimumExecutions`) or `sufficient`, a count only: `notes` say when environments, producers or branches are pooled, failed cases were published again, or history is bounded to the newest 100 results. `executions` is the denominator, open to inspection. The interval states its assumptions. This is not a production failure probability. Send `format=markdown` for Markdown.
+
+Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
+
+- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+- `key` (query, required): The test's `key` from `/api/reports/tests`.
+- `environment` (query): Environments or devices to count. Comma-separated.
+- `producer` (query): Producers to count. Comma-separated.
+- `parameter` (query): Parameter choices `key=value`; all must match. Comma-separated.
+- `branch` (query): Branches to count. Comma-separated.
+- `sha` (query): Commits: a full id or a prefix of at least seven characters. Comma-separated.
+- `specRevision` (query): Only executions recorded against these spec fingerprints (overrides `revision`). Comma-separated.
+- `revision` (query): `any` pools every spec revision; otherwise only the newest recorded one counts.
+- `since` (query): Only executions at or after this time: epoch milliseconds or a date.
+- `until` (query): Only executions at or before this time: epoch milliseconds or a date.
+- `format` (query): `markdown` for Markdown instead of JSON.
+
+Responses: `200` The reliability of the test in the scope.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No test in the report has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
 ### Reviewer tags of attachments: every recorded edit
 
 `GET /api/reports/artifact-tags` · role: read

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/atlas",
-  "version": "1.7.0",
+  "version": "1.9.0",
   "description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
   "license": "MIT",
   "type": "module",

package/packages/schema/schemas/check-detail.schema.json

@@ -56,8 +56,63 @@
                 "type": "string"
               },
               "limit": {
-                "description": "The allowance. A measurement over it is reported, and never fails the check by itself",
+                "description": "The allowance, or budget. A measurement over it is reported, and never fails the check by itself",
                 "type": "number"
+              },
+              "kind": {
+                "description": "What the number is, in an open vocabulary. `product-latency`: a user-visible response time between `from` and `to`. Absent: just a number, never read as product latency",
+                "type": "string",
+                "minLength": 1
+              },
+              "from": {
+                "description": "The event the interval starts at. Without both `from` and `to` a latency is shown but no product claim is made from it",
+                "type": "string",
+                "minLength": 1
+              },
+              "to": {
+                "description": "The event the interval ends at",
+                "type": "string",
+                "minLength": 1
+              },
+              "intended": {
+                "description": "The interval the name of the measurement claims, when what was measured is only part of it. If `intended.to` differs from `to` (or `intended.from` from `from`) the measurement is shown as partial, never as the complete interval",
+                "type": "object",
+                "properties": {
+                  "from": {
+                    "description": "The event the interval starts at: `end of speech`, `submit pressed`",
+                    "type": "string",
+                    "minLength": 1
+                  },
+                  "to": {
+                    "description": "The event the interval ends at: `audible response`, `confirmation visible`",
+                    "type": "string",
+                    "minLength": 1
+                  }
+                },
+                "additionalProperties": {}
+              },
+              "source": {
+                "description": "How it was measured: `app-instrumentation`, `trace`, `recording-analysis`",
+                "type": "string"
+              },
+              "outcome": {
+                "description": "`observed` (the default), `timeout` or `failed`. For anything but `observed` the value is how long the observation waited: at least that much, so it counts against a budget and is never dropped",
+                "type": "string",
+                "minLength": 1
+              },
+              "instrumentation": {
+                "description": "The version or method of the measurement. A change of it between two sets of observations is disclosed when they are compared",
+                "type": "string"
+              },
+              "labels": {
+                "description": "What the observation is comparable with: `device`, `build`, `network`. `device` and `build` group the observations and, when two groups differ in the others, the comparison says so",
+                "type": "object",
+                "propertyNames": {
+                  "type": "string"
+                },
+                "additionalProperties": {
+                  "type": "string"
+                }
               }
             },
             "required": [

package/packages/schema/schemas/effective-delta.schema.json

@@ -57,16 +57,74 @@
                     "type": "object",
                     "properties": {
                       "name": {
-                        "type": "string"
+                        "type": "string",
+                        "minLength": 1
                       },
                       "value": {
-                        "type": "number"
+                        "type": "number",
+                        "description": "A finite number"
                       },
                       "unit": {
                         "type": "string"
                       },
                       "limit": {
+                        "description": "The allowance, or budget. A measurement over it is reported, and never fails the check by itself",
                         "type": "number"
+                      },
+                      "kind": {
+                        "description": "What the number is, in an open vocabulary. `product-latency`: a user-visible response time between `from` and `to`. Absent: just a number, never read as product latency",
+                        "type": "string",
+                        "minLength": 1
+                      },
+                      "from": {
+                        "description": "The event the interval starts at. Without both `from` and `to` a latency is shown but no product claim is made from it",
+                        "type": "string",
+                        "minLength": 1
+                      },
+                      "to": {
+                        "description": "The event the interval ends at",
+                        "type": "string",
+                        "minLength": 1
+                      },
+                      "intended": {
+                        "description": "The interval the name of the measurement claims, when what was measured is only part of it. If `intended.to` differs from `to` (or `intended.from` from `from`) the measurement is shown as partial, never as the complete interval",
+                        "type": "object",
+                        "properties": {
+                          "from": {
+                            "description": "The event the interval starts at: `end of speech`, `submit pressed`",
+                            "type": "string",
+                            "minLength": 1
+                          },
+                          "to": {
+                            "description": "The event the interval ends at: `audible response`, `confirmation visible`",
+                            "type": "string",
+                            "minLength": 1
+                          }
+                        },
+                        "additionalProperties": {}
+                      },
+                      "source": {
+                        "description": "How it was measured: `app-instrumentation`, `trace`, `recording-analysis`",
+                        "type": "string"
+                      },
+                      "outcome": {
+                        "description": "`observed` (the default), `timeout` or `failed`. For anything but `observed` the value is how long the observation waited: at least that much, so it counts against a budget and is never dropped",
+                        "type": "string",
+                        "minLength": 1
+                      },
+                      "instrumentation": {
+                        "description": "The version or method of the measurement. A change of it between two sets of observations is disclosed when they are compared",
+                        "type": "string"
+                      },
+                      "labels": {
+                        "description": "What the observation is comparable with: `device`, `build`, `network`. `device` and `build` group the observations and, when two groups differ in the others, the comparison says so",
+                        "type": "object",
+                        "propertyNames": {
+                          "type": "string"
+                        },
+                        "additionalProperties": {
+                          "type": "string"
+                        }
                       }
                     },
                     "required": [
@@ -121,6 +179,19 @@
                       "minimum": 0,
                       "maximum": 9007199254740991
                     },
+                    "attemptStatuses": {
+                      "description": "The status of each recorded try, in the order they were made. Absent when no try was recorded, and in reports written before this was recorded: a first try is then unknown, never assumed to have passed",
+                      "type": "array",
+                      "items": {
+                        "type": "string",
+                        "enum": [
+                          "passed",
+                          "failed",
+                          "running",
+                          "not-run"
+                        ]
+                      }
+                    },
                     "links": {
                       "type": "integer",
                       "minimum": 0,
@@ -514,6 +585,19 @@
                     "minimum": 0,
                     "maximum": 9007199254740991
                   },
+                  "attemptStatuses": {
+                    "description": "The status of each recorded try, in the order they were made. Absent when no try was recorded, and in reports written before this was recorded: a first try is then unknown, never assumed to have passed",
+                    "type": "array",
+                    "items": {
+                      "type": "string",
+                      "enum": [
+                        "passed",
+                        "failed",
+                        "running",
+                        "not-run"
+                      ]
+                    }
+                  },
                   "links": {
                     "type": "integer",
                     "minimum": 0,