@crvouga/atlas 1.2.0 → 1.4.0

package/docs/rich-evidence.md

@@ -27,6 +27,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
 | `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
 | `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
 | `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
+| `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
 | `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
 | `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
 | `specs[]` (`path`, `domain`, `kind`, `title`) | layer | Names each spec and its group (`domain`), so a server can group checks by spec with a title even when no [spec set](integrating.md#tell-atlas-which-spec-a-check-belongs-to) has been published. |
@@ -62,6 +63,52 @@ attachments with that tag, and says how many match. Tags describe evidence; they
 - **Older layers.** A layer without attachment tags reads exactly as before. Tags on an attachment that cannot be read
   (not a list of strings) are dropped, and the attachment, check and layer stay.
 
+## Evidence scope
+
+A green result and a video do not say whether the speech was real, the replies scripted, the audio recorded, the device
+physical or a vendor substituted. Declare it where the evidence is, in an optional `scope`, and the viewer shows it as
+short badges above the recording (before it is played), with the exact declaration on demand:
+
+```ts
+{
+  name: 'checkout.webm', role: 'video',
+  scope: {
+    surface: 'browser',               // where the product ran: browser, native-app, api…
+    deviceKind: 'simulator',          // simulator, emulator, physical, virtual-machine, desktop…
+    syntheticData: true,              // made-up data (false: real, absent: unknown)
+    dependencies: 'scripted',         // scripted, live or mixed
+    audioCaptured: false,             // false: silent, absent: unknown
+    pacing: 'paced',                  // paced, real-time
+    asserted: ['the receipt is shown'],           // what the result stands on
+    limitations: ['Replies are scripted'],
+    declaredBy: 'checkout e2e suite',
+    verified: ['deviceKind'],         // fields something independent checked
+    verifiedBy: 'device farm session log',
+    provenance: [{ url: 'https://farm.example/session/1', name: 'Device session' }]
+  }
+}
+```
+
+Every field is optional, and every value except the booleans is an open string.
+
+- **Unknown is not live.** A recording with no scope, or a scope that does not say a thing, shows `Scope unknown` or
+  `Not declared`. It is never assumed live, physical, audible or end to end.
+- **Declared is not verified.** Atlas shows `scope` as the producer's word. Only the fields named in `verified` are
+  marked verified (with `verifiedBy`), and `provenance` links where to check them. Atlas does not check any of it.
+- **Only what was asserted.** `asserted` lists the boundaries the result stands on. The viewer says that other activity
+  visible in the clip is not verified behavior; with no `asserted` it says that no boundary was declared.
+- **The nearest declaration, whole.** A recording takes the `scope` of the attachment, else its attempt, its check, its
+  part, its layer, in that order. It is never merged field by field, so a retry on a physical phone does not borrow
+  the limitations declared for the simulator try, and the badges change with the exact recording opened.
+- **Filtering.** `GET /api/reports/tests?scope=device:physical,audio:recorded` (and the explorer's *Recording scope*
+  filter) lists tests with one playable recording that states every token: `surface:<v>`, `device:<v>`,
+  `data:synthetic` or `data:real`, `dependencies:<v>`, `audio:recorded` or `audio:silent`, `pacing:<v>`. A simulator
+  and a physical phone are separate values. Evidence that did not declare a value is never included, and `unknown`
+  asks for recordings that declared nothing. Each test lists its recordings' tokens as `scopeProfiles`, the facets as
+  `scopes`, and the search box finds the words in them.
+- **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
+  dropped, and the attachment, check and layer stay.
+
 ## Finding runs and artifacts
 
 The viewer's **Runs** and **Artifacts** tabs and the API answer one query, so a link means the same thing everywhere.
@@ -256,6 +303,29 @@ or the command line, an attachment may name a `path` instead; `atlas report publ
 `atlas report publish --ctrf` keeps it ([native runner reporters](runners.md) cover CodeceptJS, Mocha, Jest and Node's
 test runner).
 
+## Run artifacts and direct links
+
+Files attached to a run as a whole (top-level `attachments`: a review movie, a ZIP, an HTML gallery, a summary) are
+shown under **Artifacts** when you open a run in the viewer's Runs tab, without opening any test. The movie plays
+inline, an archive is a download, and each file keeps its name, role, type, size, duration and tags. They are grouped
+by the layer that carries them, with who recorded it and when. A layer may hold no checks at all, only files: it is
+listed with `No checks in this layer`, and a run of only such layers shows `No checks recorded`, never `0 passed`.
+
+- **Direct link.** *Link to this file* gives an address that names the project, the run, one immutable layer and one
+  attachment: `#/?tab=runs&openRun=<run key>&layer=<layer id>&artifact=<id:… | key:… | name:…>`. Opened in a fresh tab
+  it selects that same file of that same layer, whatever runs arrived since. A link whose file is not in the layer says
+  so instead of showing another file.
+- **Shared recordings.** Tests that reference the same bytes (same SHA-256 or bundle key) are one recording. The run's
+  *Recordings in tests* lists each once, with every test that references it, and a run-level file that a test also
+  references says so.
+- **Unplayable files.** A file with no stored bytes, with an address that is not a web address, in a type the viewer
+  has no preview for, or that the browser cannot play, shows what is wrong and, when it can, a download.
+- **Safety is unchanged.** Files are served only through the attachment route (never sniffed, sandboxed), and an HTML
+  archive renders in an iframe with an empty `sandbox`: it cannot run script.
+- **API.** `GET /api/reports/runs/detail` lists the run's `layers` (backend, check count, file count) and
+  `GET /api/reports/evidence/detail/{backend}/{id}` returns what a layer carries as a whole (and, as `parts`, what its
+  parts carry). Both only gained optional fields.
+
 ## Recording completeness
 
 A green run does not say whether the videos you expected were kept and published. Atlas answers that separately from

package/docs/ui.md

@@ -125,7 +125,7 @@ The narrowly scoped exceptions are:
 | Area | Screens/interactions | Mantine composition |
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
-| Project home | Tests/runs/artifacts tabs, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
+| Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
 | Test detail | Current outcomes, statistics, duration graph, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index, Gherkin rules/scenarios/examples, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |

package/llms-full.txt

@@ -1014,6 +1014,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
 | `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
 | `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
 | `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
+| `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
 | `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
 | `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
 | `specs[]` (`path`, `domain`, `kind`, `title`) | layer | Names each spec and its group (`domain`), so a server can group checks by spec with a title even when no [spec set](integrating.md#tell-atlas-which-spec-a-check-belongs-to) has been published. |
@@ -1049,6 +1050,52 @@ attachments with that tag, and says how many match. Tags describe evidence; they
 - **Older layers.** A layer without attachment tags reads exactly as before. Tags on an attachment that cannot be read
   (not a list of strings) are dropped, and the attachment, check and layer stay.
 
+## Evidence scope
+
+A green result and a video do not say whether the speech was real, the replies scripted, the audio recorded, the device
+physical or a vendor substituted. Declare it where the evidence is, in an optional `scope`, and the viewer shows it as
+short badges above the recording (before it is played), with the exact declaration on demand:
+
+```ts
+{
+  name: 'checkout.webm', role: 'video',
+  scope: {
+    surface: 'browser',               // where the product ran: browser, native-app, api…
+    deviceKind: 'simulator',          // simulator, emulator, physical, virtual-machine, desktop…
+    syntheticData: true,              // made-up data (false: real, absent: unknown)
+    dependencies: 'scripted',         // scripted, live or mixed
+    audioCaptured: false,             // false: silent, absent: unknown
+    pacing: 'paced',                  // paced, real-time
+    asserted: ['the receipt is shown'],           // what the result stands on
+    limitations: ['Replies are scripted'],
+    declaredBy: 'checkout e2e suite',
+    verified: ['deviceKind'],         // fields something independent checked
+    verifiedBy: 'device farm session log',
+    provenance: [{ url: 'https://farm.example/session/1', name: 'Device session' }]
+  }
+}
+```
+
+Every field is optional, and every value except the booleans is an open string.
+
+- **Unknown is not live.** A recording with no scope, or a scope that does not say a thing, shows `Scope unknown` or
+  `Not declared`. It is never assumed live, physical, audible or end to end.
+- **Declared is not verified.** Atlas shows `scope` as the producer's word. Only the fields named in `verified` are
+  marked verified (with `verifiedBy`), and `provenance` links where to check them. Atlas does not check any of it.
+- **Only what was asserted.** `asserted` lists the boundaries the result stands on. The viewer says that other activity
+  visible in the clip is not verified behavior; with no `asserted` it says that no boundary was declared.
+- **The nearest declaration, whole.** A recording takes the `scope` of the attachment, else its attempt, its check, its
+  part, its layer, in that order. It is never merged field by field, so a retry on a physical phone does not borrow
+  the limitations declared for the simulator try, and the badges change with the exact recording opened.
+- **Filtering.** `GET /api/reports/tests?scope=device:physical,audio:recorded` (and the explorer's *Recording scope*
+  filter) lists tests with one playable recording that states every token: `surface:<v>`, `device:<v>`,
+  `data:synthetic` or `data:real`, `dependencies:<v>`, `audio:recorded` or `audio:silent`, `pacing:<v>`. A simulator
+  and a physical phone are separate values. Evidence that did not declare a value is never included, and `unknown`
+  asks for recordings that declared nothing. Each test lists its recordings' tokens as `scopeProfiles`, the facets as
+  `scopes`, and the search box finds the words in them.
+- **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
+  dropped, and the attachment, check and layer stay.
+
 ## Finding runs and artifacts
 
 The viewer's **Runs** and **Artifacts** tabs and the API answer one query, so a link means the same thing everywhere.
@@ -1243,6 +1290,29 @@ or the command line, an attachment may name a `path` instead; `atlas report publ
 `atlas report publish --ctrf` keeps it ([native runner reporters](runners.md) cover CodeceptJS, Mocha, Jest and Node's
 test runner).
 
+## Run artifacts and direct links
+
+Files attached to a run as a whole (top-level `attachments`: a review movie, a ZIP, an HTML gallery, a summary) are
+shown under **Artifacts** when you open a run in the viewer's Runs tab, without opening any test. The movie plays
+inline, an archive is a download, and each file keeps its name, role, type, size, duration and tags. They are grouped
+by the layer that carries them, with who recorded it and when. A layer may hold no checks at all, only files: it is
+listed with `No checks in this layer`, and a run of only such layers shows `No checks recorded`, never `0 passed`.
+
+- **Direct link.** *Link to this file* gives an address that names the project, the run, one immutable layer and one
+  attachment: `#/?tab=runs&openRun=<run key>&layer=<layer id>&artifact=<id:… | key:… | name:…>`. Opened in a fresh tab
+  it selects that same file of that same layer, whatever runs arrived since. A link whose file is not in the layer says
+  so instead of showing another file.
+- **Shared recordings.** Tests that reference the same bytes (same SHA-256 or bundle key) are one recording. The run's
+  *Recordings in tests* lists each once, with every test that references it, and a run-level file that a test also
+  references says so.
+- **Unplayable files.** A file with no stored bytes, with an address that is not a web address, in a type the viewer
+  has no preview for, or that the browser cannot play, shows what is wrong and, when it can, a download.
+- **Safety is unchanged.** Files are served only through the attachment route (never sniffed, sandboxed), and an HTML
+  archive renders in an iframe with an empty `sandbox`: it cannot run script.
+- **API.** `GET /api/reports/runs/detail` lists the run's `layers` (backend, check count, file count) and
+  `GET /api/reports/evidence/detail/{backend}/{id}` returns what a layer carries as a whole (and, as `parts`, what its
+  parts carry). Both only gained optional fields.
+
 ## Recording completeness
 
 A green run does not say whether the videos you expected were kept and published. Atlas answers that separately from
@@ -2757,6 +2827,7 @@ carry any of these. Every one is optional.
 | `attempts[]` | Every try, when the check was retried. |
 | `tags[]` | Free labels to filter by: `smoke`, `payments`. |
 | `labels{}` | Facts about the result that are not its identity: `owner`, `severity`, `browser`, `build`. |
+| `scope{}` | What the evidence does and does not show: surface, device kind, synthetic data, scripted dependencies, audio, pacing, what was asserted and its limits. Unknown when absent. See [Evidence scope](rich-evidence.md#evidence-scope). |
 | `links[]` | Where to read more: the CI run, the issue, the trace client. |
 | `extensions{}` | Structured data of your own or a tool's, under a name you own. |
 | `measurements[]` | Numbers, with units and an allowance (already part of the contract). |
@@ -3886,7 +3957,7 @@ The narrowly scoped exceptions are:
 | Area | Screens/interactions | Mantine composition |
 | --- | --- | --- |
 | Shared shell | Project switching, spec navigation, run selection, connection status, diagnostics, mobile index | AppShell, Burger, NavLink, Menu, Popover, NativeSelect, Drawer |
-| Project home | Tests/runs/artifacts tabs, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
+| Project home | Tests/runs/artifacts tabs, a run's Artifacts (files of its layers, shared recordings, direct links), metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion |
 | Test detail | Current outcomes, statistics, duration graph, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
 | Specs | All six existing kinds, overview/index, Gherkin rules/scenarios/examples, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Table, CodeHighlightTabs, ResultRow |
 | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
@@ -4923,6 +4994,7 @@ Send `format=markdown` (or `Accept: text/markdown`) for the same as a Markdown r
 - `tag` (query): Only tests with this tag. Repeatable or comma-separated.
 - `artifactTag` (query): Only tests with an attachment tagged this (`demo` and `#demo` are the same tag), not counting tags inherited from a check or run: the exact artifact query. `inherited=1` widens it to the tags inherited from the check and run. Repeatable or comma-separated.
 - `inherited` (query): With `artifactTag`: `1` also matches the tags an attachment inherits from its check and run, so a recording without a tag of its own under a run tagged `demo` is found. Without it the query is exact.
+- `scope` (query): Only tests with one playable recording that states every one of these producer-declared qualifications: `surface:<value>`, `device:<value>` (`device:physical` and `device:simulator` stay apart), `data:synthetic` or `data:real`, `dependencies:<value>`, `audio:recorded` or `audio:silent`, `pacing:<value>`. All must hold for the same recording. A recording that did not declare a value never matches it, and `unknown` matches only a recording that declared no scope. Repeatable or comma-separated.
 - `run` (query): Only tests recorded in this run (a `key` from `/api/reports/runs`).
 - `sort` (query): The order: `spec` (default: domain, spec, title), `title`, `recent`, `slowest`, `failures`, `flaky` or `runs`.
 - `offset` (query): Skip this many matching tests.
@@ -4954,6 +5026,7 @@ Scoped to one project. The same project may be selected on the unprefixed API us
 - `tag` (query): Only tests with this tag. Repeatable or comma-separated.
 - `artifactTag` (query): Only tests with an attachment tagged this (`demo` and `#demo` are the same tag), not counting tags inherited from a check or run: the exact artifact query. `inherited=1` widens it to the tags inherited from the check and run. Repeatable or comma-separated.
 - `inherited` (query): With `artifactTag`: `1` also matches the tags an attachment inherits from its check and run, so a recording without a tag of its own under a run tagged `demo` is found. Without it the query is exact.
+- `scope` (query): Only tests with one playable recording that states every one of these producer-declared qualifications: `surface:<value>`, `device:<value>` (`device:physical` and `device:simulator` stay apart), `data:synthetic` or `data:real`, `dependencies:<value>`, `audio:recorded` or `audio:silent`, `pacing:<value>`. All must hold for the same recording. A recording that did not declare a value never matches it, and `unknown` matches only a recording that declared no scope. Repeatable or comma-separated.
 - `run` (query): Only tests recorded in this run (a `key` from `/api/reports/runs`).
 - `sort` (query): The order: `spec` (default: domain, spec, title), `title`, `recent`, `slowest`, `failures`, `flaky` or `runs`.
 - `offset` (query): Skip this many matching tests.
@@ -5180,7 +5253,7 @@ Responses: `200` The matching results and the facets.; `304` Unchanged since the
 
 `GET /api/reports/runs/detail` · role: read
 
-One run and its last result for every test, environment and producer it recorded, failures first. Send `format=markdown` for Markdown.
+One run and its last result for every test, environment and producer it recorded, failures first, and `layers`: each layer of the run with its backend, how many checks it recorded (zero means evidence only, never a pass) and how many files it carries as a whole. Open a layer's files with `/api/reports/evidence/detail/{backend}/{id}`. Send `format=markdown` for Markdown.
 
 - `key` (query, required): The run's `key` from `/api/reports/runs`, URL-encoded.
 - `format` (query): `markdown` for Markdown instead of JSON.
@@ -5192,7 +5265,7 @@ Responses: `200` The run and its results.; `304` Unchanged since the `If-None-Ma
 
 `GET /api/projects/{project}/reports/runs/detail` · role: read
 
-One run and its last result for every test, environment and producer it recorded, failures first. Send `format=markdown` for Markdown.
+One run and its last result for every test, environment and producer it recorded, failures first, and `layers`: each layer of the run with its backend, how many checks it recorded (zero means evidence only, never a pass) and how many files it carries as a whole. Open a layer's files with `/api/reports/evidence/detail/{backend}/{id}`. Send `format=markdown` for Markdown.
 
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 
@@ -5302,7 +5375,7 @@ Responses: `200` The rows.; `401` No token, or an unknown one. Send `Authorizati
 
 `GET /api/reports/evidence/detail/{backend}/{id}` · role: read
 
-The timeline, annotations, retries, attachments and extensions of one check, and what its run carries as a whole. Without `check`, only the run's. Attachment bytes are fetched from `/api/reports/layers/{id}/blobs/{bundleKey}`.
+The timeline, annotations, retries, attachments and extensions of one check, and what its run carries as a whole. Without `check`, only the run's, and `parts`: what each part of the layer that carries files or links of its own holds. Attachment bytes are fetched from `/api/reports/layers/{id}/blobs/{bundleKey}`.
 
 - `backend` (path, required): A backend id from `/api/server`.
 - `id` (path, required): A layer id: 64 lowercase hex digits.
@@ -5316,7 +5389,7 @@ Responses: `200` The detail.; `401` No token, or an unknown one. Send `Authoriza
 
 `GET /api/projects/{project}/reports/evidence/detail/{backend}/{id}` · role: read
 
-The timeline, annotations, retries, attachments and extensions of one check, and what its run carries as a whole. Without `check`, only the run's. Attachment bytes are fetched from `/api/reports/layers/{id}/blobs/{bundleKey}`.
+The timeline, annotations, retries, attachments and extensions of one check, and what its run carries as a whole. Without `check`, only the run's, and `parts`: what each part of the layer that carries files or links of its own holds. Attachment bytes are fetched from `/api/reports/layers/{id}/blobs/{bundleKey}`.
 
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/atlas",
-  "version": "1.2.0",
+  "version": "1.4.0",
   "description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
   "license": "MIT",
   "type": "module",