@crvouga/atlas 1.17.0 → 1.19.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (66) hide show
  1. package/CHANGELOG.md +14 -1
  2. package/dist/reports/src/artifact-query.d.ts +14 -2
  3. package/dist/reports/src/artifact-query.js +13 -4
  4. package/dist/reports/src/evidence-detail.js +19 -6
  5. package/dist/reports/src/evidence-types.d.ts +8 -0
  6. package/dist/reports/src/recording-receipt.d.ts +23 -0
  7. package/dist/reports/src/recording-receipt.js +16 -0
  8. package/dist/reports/src/recording-verify.d.ts +74 -0
  9. package/dist/reports/src/recording-verify.js +142 -0
  10. package/dist/reports/src/report-cli.d.ts +2 -2
  11. package/dist/reports/src/report-cli.js +53 -3
  12. package/dist/reports/src/test-catalog.d.ts +7 -1
  13. package/dist/reports/src/test-catalog.js +10 -4
  14. package/dist/schema/src/api.d.ts +134 -0
  15. package/dist/schema/src/api.js +13 -2
  16. package/dist/schema/src/cli-results.d.ts +51 -1
  17. package/dist/schema/src/cli-results.js +25 -1
  18. package/dist/server/src/openapi.js +1 -0
  19. package/docs/agents.md +1 -0
  20. package/docs/reference/cli.md +11 -2
  21. package/docs/reference/http-api.md +2 -0
  22. package/docs/reference/reports.md +189 -9
  23. package/docs/reference/schema.md +19 -1
  24. package/docs/rich-evidence.md +33 -5
  25. package/docs/server.md +1 -1
  26. package/docs/ui.md +40 -1
  27. package/llms-full.txt +88 -9
  28. package/package.json +1 -1
  29. package/packages/schema/schemas/effective-delta.schema.json +26 -0
  30. package/packages/schema/schemas/effective-evidence.schema.json +26 -0
  31. package/packages/schema/schemas/evidence-page.schema.json +13 -0
  32. package/packages/schema/schemas/layer-rows.schema.json +13 -0
  33. package/packages/schema/schemas/report-run.schema.json +83 -0
  34. package/packages/schema/schemas/report-runs.schema.json +86 -0
  35. package/packages/schema/schemas/review-collection.schema.json +13 -0
  36. package/packages/schema/schemas/review-manifest.schema.json +13 -0
  37. package/packages/schema/schemas/test-catalog.schema.json +13 -0
  38. package/packages/schema/schemas/test-history.schema.json +26 -0
  39. package/packages/schema/schemas/verify-recordings-result.schema.json +97 -0
  40. package/packages/server/openapi.json +378 -0
  41. package/packages/visualizer/src/app/AppShell.tsx +2 -2
  42. package/packages/visualizer/src/app/router.tsx +25 -1
  43. package/packages/visualizer/src/components/Header.tsx +15 -4
  44. package/packages/visualizer/src/evidence/behavior-index.test.ts +119 -0
  45. package/packages/visualizer/src/evidence/behavior-index.ts +162 -0
  46. package/packages/visualizer/src/evidence/behavior-panel.tsx +36 -22
  47. package/packages/visualizer/src/evidence/behavior-return.test.ts +41 -0
  48. package/packages/visualizer/src/evidence/behavior-return.ts +59 -0
  49. package/packages/visualizer/src/evidence/behavior.ts +32 -1
  50. package/packages/visualizer/src/evidence/behaviors-page.tsx +465 -0
  51. package/packages/visualizer/src/evidence/feature-page.tsx +5 -26
  52. package/packages/visualizer/src/evidence/scoped-summary-view.tsx +110 -0
  53. package/packages/visualizer/src/evidence/scoped-summary.test.ts +62 -0
  54. package/packages/visualizer/src/evidence/scoped-summary.ts +118 -0
  55. package/packages/visualizer/src/evidence/spec-texts.ts +88 -0
  56. package/packages/visualizer/src/explorer/ArtifactsView.tsx +7 -6
  57. package/packages/visualizer/src/explorer/Explorer.tsx +1 -1
  58. package/packages/visualizer/src/explorer/RunsView.tsx +32 -7
  59. package/packages/visualizer/src/explorer/collections.tsx +5 -5
  60. package/packages/visualizer/src/explorer/filters.tsx +11 -3
  61. package/packages/visualizer/src/explorer/navigate.ts +17 -0
  62. package/packages/visualizer/src/explorer/playlist-entries.test.ts +10 -1
  63. package/packages/visualizer/src/explorer/playlist-entries.ts +10 -0
  64. package/packages/visualizer/src/explorer/playlist.tsx +43 -8
  65. package/packages/visualizer/src/explorer/search.ts +13 -3
  66. package/packages/visualizer/src/find/Finder.tsx +1 -62
@@ -105,6 +105,8 @@ atlas report — layered evidence for any kind of spec
105
105
  [--specs <dir>] [--format text|json|markdown]
106
106
  [--require-environments a,b] [--require-all-specs]
107
107
  [--branch <name>]... [--ignore-modified]
108
+ atlas report verify-recordings [--store <dir|s3://…|id=url>]... [--backends <module>] [--layer <id>]...
109
+ [--limit <n>] [--branch <name>]... [--ignore-modified] [--format text|json]
108
110
  atlas report merge [--store …]... [--backends <module>] --into <dir|s3://…>
109
111
  atlas report doctor [--store …]... [--role publisher|reader|both]
110
112
  atlas report policy --store s3://bucket/prefix [--role publisher|reader|both]
@@ -148,6 +150,13 @@ atlas report — layered evidence for any kind of spec
148
150
  --require-all-specs fail while any spec has no results
149
151
  --branch <name> only layers recorded on this branch (repeatable)
150
152
  --ignore-modified leave out layers from modified checkouts and unknown origins
153
+ verify-recordings Read back the videos the layers published and check they are still there as published: the
154
+ layer still lists the file, its blob is still stored, its size and SHA-256 are the recorded ones, and it
155
+ begins like the container its media type names. Exit 1 when any video is missing, unreadable, corrupt or
156
+ not that container, naming the layer and check; test results are never changed. This checks stored bytes,
157
+ not playback in a browser. Videos given only by address are counted, not checked.
158
+ --layer <id> only the layer whose id starts with this (repeatable; at least 7 digits)
159
+ --limit <n> only the newest n layers
151
160
  merge Copy the layers of several stores into one.
152
161
  doctor Check that a store works: credentials, listing, writing, and what it holds.
153
162
  policy Print the least-privilege IAM policy for publishers, readers, or both.
@@ -156,8 +165,8 @@ atlas report — layered evidence for any kind of spec
156
165
  { id, label?, store } (or a function returning it): a device lab, anything
157
166
  that implements ObjectStore.
158
167
 
159
- --format json (alias --json) publish, stream, merge, doctor and status print one JSON document instead of prose
160
- (schemas: publish-result, stream-result, merge-result, doctor-result, report-status). A command
168
+ --format json (alias --json) publish, stream, merge, doctor, status and verify-recordings print one JSON document instead of prose
169
+ (schemas: publish-result, stream-result, merge-result, doctor-result, report-status, verify-recordings-result). A command
161
170
  that is wrong is {"ok":false,"error":{"message":…,"code":"usage"}} on standard error.
162
171
  A store may be an Atlas server (https://…): ATLAS_STORE=https://atlas.example.com, with ATLAS_TOKEN.
163
172
 
@@ -833,6 +833,7 @@ Every run, newest first: layers grouped by their CI run URL (else branch and com
833
833
  - `branch` (query): Branches.
834
834
  - `sha` (query): Revisions: a full commit id or a prefix of at least seven characters.
835
835
  - `outcome` (query): Statuses: `passed`, `failed`, `running`, `not-run`. A run's outcome is `failed` when any result failed, else `running`, else `passed`, else `not-run`.
836
+ - `recordings` (query): Recording completeness of the run, apart from its outcome: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no recording expectations is `unknown`, never `complete`). Repeatable or comma-separated. Answered from the layers' receipts; no recording is read. A server that does not know it ignores it and does not name it in `filters`.
836
837
  - `q` (query): Words that must all appear in the title, id, spec, branch, producer, environment, tags or labels.
837
838
  - `since` (query): Only records at or after this time: epoch milliseconds or an ISO date (a date alone is UTC). A result's recording time; a run's end.
838
839
  - `until` (query): Only records at or before this time (same forms as `since`).
@@ -861,6 +862,7 @@ Scoped to one project. The same project may be selected on the unprefixed API us
861
862
  - `branch` (query): Branches.
862
863
  - `sha` (query): Revisions: a full commit id or a prefix of at least seven characters.
863
864
  - `outcome` (query): Statuses: `passed`, `failed`, `running`, `not-run`. A run's outcome is `failed` when any result failed, else `running`, else `passed`, else `not-run`.
865
+ - `recordings` (query): Recording completeness of the run, apart from its outcome: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no recording expectations is `unknown`, never `complete`). Repeatable or comma-separated. Answered from the layers' receipts; no recording is read. A server that does not know it ignores it and does not name it in `filters`.
864
866
  - `q` (query): Words that must all appear in the title, id, spec, branch, producer, environment, tags or labels.
865
867
  - `since` (query): Only records at or after this time: epoch milliseconds or an ISO date (a date alone is UTC). A result's recording time; a run's end.
866
868
  - `until` (query): Only records at or before this time (same forms as `since`).
@@ -10,7 +10,7 @@ Source: `packages/reports/src/artifact-query.ts`. Import: `@crvouga/atlas/report
10
10
 
11
11
  One shareable query for the runs of a project and for the recorded results that carry evidence (recordings,
12
12
  screenshots, traces): by artifact tag, label, media type, feature (domain, spec), environment or device, producer,
13
- branch, revision, outcome and time. The same rules answer `GET /api/reports/runs`, `GET /api/reports/artifacts`
13
+ branch, revision, outcome, time and (runs only) recording completeness. The same rules answer `GET /api/reports/runs`, `GET /api/reports/artifacts`
14
14
  and the viewer's Runs and Artifacts views, so a link means the same thing everywhere.
15
15
 
16
16
  The semantics are fixed, not inferred:
@@ -81,6 +81,12 @@ export type ArtifactFilter = {
81
81
  readonly domain?: readonly string[];
82
82
  /** Spec paths. Not applicable to runs. */
83
83
  readonly spec?: readonly string[];
84
+ /**
85
+ * Recording completeness of a run: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no
86
+ * recording expectations is `unknown`, never `complete`). Answered from the layers' receipts alone, without reading any
87
+ * recording. Applies to runs only; it is independent of `outcome`.
88
+ */
89
+ readonly recordings?: readonly string[];
84
90
  /** Only records at or after this time (epoch ms): a result's recording time, a run's end. */
85
91
  readonly since?: number;
86
92
  /** Only records at or before this time (epoch ms). */
@@ -124,6 +130,8 @@ export type FilterFields = {
124
130
  /** Absent when the dimension does not apply (runs have no domain). */
125
131
  readonly domains?: readonly string[];
126
132
  readonly specs?: readonly string[];
133
+ /** The recording completeness of a run (one value). Absent when the dimension does not apply (results). */
134
+ readonly recordings?: readonly string[];
127
135
  readonly at: number;
128
136
  };
129
137
  ```
@@ -143,6 +151,8 @@ export type QueryFacets = {
143
151
  readonly branches: readonly QueryFacetValue[];
144
152
  readonly outcomes: readonly QueryFacetValue[];
145
153
  readonly domains?: readonly QueryFacetValue[];
154
+ /** Runs only: `complete`, `incomplete`, `none-expected` and `unknown`, with how many runs have each. */
155
+ readonly recordings?: readonly QueryFacetValue[];
146
156
  };
147
157
  ```
148
158
 
@@ -204,6 +214,14 @@ export function usedFilters(filter: ArtifactFilter): FilterName[]
204
214
 
205
215
  Which filters a query uses, by the names an address gives them.
206
216
 
217
+ #### `forArtifacts`
218
+
219
+ ```ts
220
+ export function forArtifacts(filter: ArtifactFilter): ArtifactFilter
221
+ ```
222
+
223
+ Every filter in `filter` that may be applied to a recorded result, as the artifacts query applies them (no recording completeness, which is a property of a run).
224
+
207
225
  #### `isFiltered`
208
226
 
209
227
  ```ts
@@ -218,7 +236,7 @@ Whether a query narrows anything.
218
236
  export function forRuns(filter: ArtifactFilter): ArtifactFilter
219
237
  ```
220
238
 
221
- Every filter in `filter` that may be applied to a record with these fields, as the runs query applies them (no domain or spec).
239
+ Every filter in `filter` that may be applied to a run, as the runs query applies them (no domain or spec).
222
240
 
223
241
  #### `matchesFilter`
224
242
 
@@ -2161,6 +2179,12 @@ export type VideoRef = {
2161
2179
  readonly recording?: string;
2162
2180
  /** False when the reference names no bytes at all (no bundle key, address or inline text), or a bundle key that the layer's own bundle does not hold, so it can never be played. */
2163
2181
  readonly available: boolean;
2182
+ /** Why it is unavailable, when the report can tell: `not-in-bundle` names a file the layer should hold (its bundle key) but does not, a recording that was acknowledged and is no longer there. Absent otherwise, and for an available video. An open vocabulary: treat an unknown reason as plain unavailable. */
2183
+ readonly reason?: 'not-in-bundle';
2184
+ /** How long the recording is, in milliseconds, as its producer declared (`Attachment.durationMs`): a hint to show before the file is opened, never measured here and never a latency. Absent when none was declared, and in documents of older servers. */
2185
+ readonly durationMs?: number;
2186
+ /** The bundle key, in the same layer, of the image the producer named as this recording's poster (`Attachment.poster`): a hint to show before the file is opened. Only an image the layer's bundle holds; absent when there is none, and in documents of older servers. */
2187
+ readonly poster?: string;
2164
2188
  /** What this recording shows, as declared for it: its own `scope`, else its attempt's, check's, part's or layer's. Absent when none declared one: its scope is unknown, not live, physical or audible. */
2165
2189
  readonly scope?: EvidenceScope;
2166
2190
  };
@@ -2180,6 +2204,8 @@ export type VideoCounts = {
2180
2204
  readonly recordings: number;
2181
2205
  /** References that name no bytes, or bytes the layer does not hold. A report that predates availability metadata has none of these and none of the others. */
2182
2206
  readonly unavailable: number;
2207
+ /** Of `unavailable`, those that name a file the layer's bundle does not hold (stored, then gone). Absent when there are none. */
2208
+ readonly missing?: number;
2183
2209
  };
2184
2210
  ```
2185
2211
 
@@ -6408,6 +6434,160 @@ skipped or unresolved; otherwise `unknown` when a contributing layer has no rece
6408
6434
  report with no results at all is unknown too); otherwise `complete` when something was expected and `none-expected` when
6409
6435
  nothing was. Gaps are sorted so the output never depends on read order.
6410
6436
 
6437
+ #### `RunRecordings`
6438
+
6439
+ ```ts
6440
+ export type RunRecordings = {
6441
+ /** `incomplete`: a recording is skipped or unresolved. `unknown`: no gap is known, but a layer declared no expectations. `complete`: everything expected was published. `none-expected`: every layer declared expectations and none required a recording. */
6442
+ readonly state: 'complete' | 'incomplete' | 'none-expected' | 'unknown';
6443
+ /** Recordings the layers expected. */
6444
+ readonly expected: number;
6445
+ /** Of those, uploaded or referenced by address. */
6446
+ readonly published: number;
6447
+ readonly skipped: number;
6448
+ readonly unresolved: number;
6449
+ /** Intentionally unrecorded checks: neither expected nor missing. */
6450
+ readonly notApplicable: number;
6451
+ /** Layers of the run that declared no expectations: nothing is claimed for them. */
6452
+ readonly layersWithoutReceipt: number;
6453
+ };
6454
+ ```
6455
+
6456
+ A run's recording completeness, from the receipts of its layers (see {@link runRecordings}).
6457
+
6458
+ #### `runRecordings`
6459
+
6460
+ ```ts
6461
+ export function runRecordings(layers: readonly (RecordingCounts | undefined)[]): RunRecordings | undefined
6462
+ ```
6463
+
6464
+ The recording completeness of a run, from the `recordings` counts of each of its layers (`undefined` for a layer that
6465
+ has no receipt). `undefined` when no layer has a receipt: the run claims nothing, which a reader treats as `unknown`.
6466
+ A gap always wins (`incomplete`); otherwise a layer without a receipt keeps the run `unknown`, never `complete`. Reads
6467
+ only the counts, never a recording, and never changes a status.
6468
+
6469
+ ### Recording verification
6470
+
6471
+ Source: `packages/reports/src/recording-verify.ts`. Import: `@crvouga/atlas/reports/recording-verify`
6472
+
6473
+ Answers "is the recording we published still there, and does it still look like the video we published?" apart from
6474
+ "did the tests pass?" and apart from "was it uploaded?" (the receipt, see `recording-receipt.ts`). `verifyLayerRecordings`
6475
+ reads each video a layer keeps in its bundle back from the store and checks, in order, that the layer's bundle still
6476
+ lists its file, that the blob it lives in is still stored, that its size and SHA-256 are the ones recorded at upload,
6477
+ and that its first bytes are the container header its media type promises. A problem is a gap in the evidence, named
6478
+ with the check that holds the video; it never changes a status, a slot or a winner.
6479
+
6480
+ Trap: this is a stored-bytes check, not a playback test. A file with a valid header can still fail to play in a given
6481
+ browser (codec, truncation), and acknowledging an upload is not playback either. Videos given only by `uri` are counted
6482
+ as `referenced`, not checked: Atlas holds no bytes to read back.
6483
+
6484
+ See docs/rich-evidence.md#recording-completeness
6485
+
6486
+ #### `RecordingProblem`
6487
+
6488
+ ```ts
6489
+ export type RecordingProblem =
6490
+ /** The layer names the file but its bundle does not list it: stored, then gone (or never arrived). */
6491
+ | 'not-in-bundle'
6492
+ /** The bundle lists the file but a blob holding its bytes is no longer in the store (a 404 when played). */
6493
+ | 'blob-missing'
6494
+ /** The store could not be read just now. Not proof the bytes are gone, and not a pass. */
6495
+ | 'unreadable'
6496
+ /** The stored bytes are not as many as were recorded. */
6497
+ | 'size-mismatch'
6498
+ /** The stored bytes do not hash to the SHA-256 recorded at upload. */
6499
+ | 'checksum-mismatch'
6500
+ /** The stored bytes do not begin like the container their media type names, or are empty: incompatible or corrupt. */
6501
+ | 'undecodable';
6502
+ ```
6503
+
6504
+ What is wrong with a stored recording. An open vocabulary: a reader treats an unknown problem as a gap.
6505
+
6506
+ #### `RecordingGap`
6507
+
6508
+ ```ts
6509
+ export type RecordingGap = {
6510
+ /** The check that holds the video, or `null` for a video attached to the layer or a part. */
6511
+ readonly check: string | null;
6512
+ /** The attachment's name. */
6513
+ readonly name: string;
6514
+ readonly bundleKey: string;
6515
+ readonly problem: RecordingProblem;
6516
+ /** A short, non-secret explanation: sizes or the container found. */
6517
+ readonly reason: string;
6518
+ };
6519
+ ```
6520
+
6521
+ One video of a layer that failed verification.
6522
+
6523
+ #### `LayerRecordingVerification`
6524
+
6525
+ ```ts
6526
+ export type LayerRecordingVerification = {
6527
+ readonly layer: string;
6528
+ /** Videos read back from the store: those that name a bundle key. */
6529
+ readonly checked: number;
6530
+ /** Of those, the ones that passed every check. */
6531
+ readonly ok: number;
6532
+ /** Videos given only by `uri`, which Atlas cannot read back. */
6533
+ readonly referenced: number;
6534
+ readonly gaps: readonly RecordingGap[];
6535
+ };
6536
+ ```
6537
+
6538
+ What verifying one layer found.
6539
+
6540
+ #### `containerProblem`
6541
+
6542
+ ```ts
6543
+ export function containerProblem(bytes: Uint8Array, contentType: string): string | null
6544
+ ```
6545
+
6546
+ Whether the first bytes of a stored video are the container its media type names, as a reason it is not, or `null` when
6547
+ it looks right or the type is one this check has no signature for (then nothing is claimed). Recognizes WebM and
6548
+ Matroska (EBML), MP4, QuickTime and 3GPP (an ISO box), Ogg and AVI. An empty file is never a video.
6549
+
6550
+ #### `verifyLayerRecordings`
6551
+
6552
+ ```ts
6553
+ export async function verifyLayerRecordings(store: ObjectStore, layer: ReportLayer): Promise<LayerRecordingVerification>
6554
+ ```
6555
+
6556
+ Reads back every video of `layer` that is kept in its bundle and reports the ones that are no longer retrievable or no
6557
+ longer look like what was published (see {@link RecordingProblem}). Each blob is read once however many checks
6558
+ reference it. A store that fails to answer is `unreadable`, never a pass. It only reads: nothing is written, and no
6559
+ status changes.
6560
+
6561
+ **Example**
6562
+
6563
+ ```ts
6564
+ const layer = await readLayer(store, id);
6565
+ const { gaps } = await verifyLayerRecordings(store, layer!);
6566
+ gaps.map(gap => `${gap.check}: ${gap.problem}`);
6567
+ ```
6568
+
6569
+ #### `StoreRecordingVerification`
6570
+
6571
+ ```ts
6572
+ export type StoreRecordingVerification = {
6573
+ readonly layers: readonly LayerRecordingVerification[];
6574
+ /** Layers that could not be read at all (their manifest is unreadable or gone): their recordings are unknown, not verified. */
6575
+ readonly notices: readonly string[];
6576
+ };
6577
+ ```
6578
+
6579
+ What verifying the layers of one store found.
6580
+
6581
+ #### `verifyStoreRecordings`
6582
+
6583
+ ```ts
6584
+ export async function verifyStoreRecordings(store: ObjectStore, options: { readonly ids?: readonly string[]; readonly limit?: number; readonly include?: (layer: ReportLayer) => boolean } = {}): Promise<StoreRecordingVerification>
6585
+ ```
6586
+
6587
+ Verifies the recordings of the layers a store lists, newest first. `ids` narrows to layers whose id starts with one of
6588
+ them (at least 7 hex digits); `limit` keeps only that many of the newest. A layer that cannot be read is a notice, never
6589
+ skipped silently and never counted as verified. A store that cannot be listed rejects.
6590
+
6411
6591
  ### Reliability of a test
6412
6592
 
6413
6593
  Source: `packages/reports/src/reliability.ts`. Import: `@crvouga/atlas/reports/reliability`
@@ -6767,7 +6947,7 @@ Call this inside a Solid root: it creates signals and a store.
6767
6947
 
6768
6948
  Source: `packages/reports/src/report-cli.ts`. Import: `@crvouga/atlas/reports/report-cli`
6769
6949
 
6770
- `reportMain` is `atlas report publish | stream | status | merge | doctor | policy`, written so it can be called from code and
6950
+ `reportMain` is `atlas report publish | stream | status | verify-recordings | merge | doctor | policy`, written so it can be called from code and
6771
6951
  tests as well as a terminal: it takes its arguments, environment, working directory, clock, git and standard input as
6772
6952
  parameters and returns the exit code. Every command can print a JSON document (`--format json`) instead of prose, and a
6773
6953
  command that is wrong throws, which `runReportCommand` turns into a message (or a JSON error) and exit code 2.
@@ -9942,14 +10122,14 @@ export type RunSummary = {
9942
10122
  * and references that cannot be played. Absent when it has none, and in a report that carries no video information.
9943
10123
  */
9944
10124
  readonly videos?: VideoCounts;
10125
+ /**
10126
+ * Whether the recordings its layers expected were published, apart from its outcomes (see `runRecordings`). Absent when
10127
+ * none of its layers declared a recording expectation: its completeness is unknown, never complete. Never changes a status.
10128
+ */
10129
+ readonly recordings?: RunRecordings;
9945
10130
  /**
9946
10131
  * Files attached to the run itself, in its layers (and their parts), not to any one check: review movies, archives,
9947
10132
  * summaries. All kinds, videos included. Absent when none, and in a report that carries no such information.
9948
- */
9949
- readonly attachments?: number;
9950
- /** The normalized tags of the attachments of its results and layers, sorted and distinct. Absent when none is tagged. */
9951
- readonly artifactTags?: readonly string[];
9952
- /** The tags of its checks and layers, which its attachments inherit (what `inherited=1` also matches), sorted and distinct. Absent when none. */
9953
10133
  …
9954
10134
  ```
9955
10135
 
@@ -10281,7 +10461,7 @@ What a query looks at in a run: see {@link ArtifactFilter}. A run's outcome is `
10281
10461
  #### `artifactList`
10282
10462
 
10283
10463
  ```ts
10284
- export function artifactList(history: TestHistory, version: string, query: ArtifactFilter = {}, page: { readonly limit?: number; readonly offset?: number; readonly domainOf?: (specPath: string) => string | undefined } = {}): ArtifactList
10464
+ export function artifactList(history: TestHistory, version: string, asked: ArtifactFilter = {}, page: { readonly limit?: number; readonly offset?: number; readonly domainOf?: (specPath: string) => string | undefined } = {}): ArtifactList
10285
10465
  ```
10286
10466
 
10287
10467
  The recorded results that carry evidence and match a query, newest first, across the whole history (`limit` default
@@ -535,7 +535,7 @@ of every layer. States are in id order and cells in profile order, so the same l
535
535
 
536
536
  Source: `packages/schema/src/cli-results.ts`
537
537
 
538
- What `atlas report publish`, `stream`, `merge` and `doctor` print on standard output with `--format json` (alias
538
+ What `atlas report publish`, `stream`, `merge`, `doctor` and `verify-recordings` print on standard output with `--format json` (alias
539
539
  `--json`), and what any `atlas` command prints on standard error for a command that was wrong: one JSON document,
540
540
  so an agent or a script never has to parse prose. (`status` and `policy` already print documents:
541
541
  `ReportStatusSchema`, and an IAM policy.) Each is also published as a JSON Schema.
@@ -594,6 +594,24 @@ export const DoctorResultSchema = …
594
594
 
595
595
  What `atlas report doctor --format json` prints.
596
596
 
597
+ #### `RecordingGapSchema`
598
+
599
+ ```ts
600
+ export const RecordingGapSchema = …
601
+ ```
602
+
603
+ A published video that could not be read back from the store as published (see `atlas report verify-recordings`).
604
+
605
+ #### `VerifyRecordingsResultSchema`
606
+
607
+ ```ts
608
+ export const VerifyRecordingsResultSchema = …
609
+ ```
610
+
611
+ What `atlas report verify-recordings --format json` prints: whether the videos the layers published can still be read
612
+ back from the store as they were published. It describes stored bytes, never playback in a browser, and it never changes
613
+ a check\'s result.
614
+
597
615
  #### `CommandErrorSchema`
598
616
 
599
617
  ```ts
@@ -27,6 +27,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
27
27
  | `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
28
28
  | `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
29
29
  | `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
30
+ | `poster` (the `id` of an image in the same list) and `durationMs` | attachment (video) | Shown beside the recording in a watch playlist before anything is opened: the report carries them as the optional `poster` (a bundle key) and `durationMs` of each video reference, so no evidence is read per entry. Neither is ever a latency. |
30
31
  | `extensions['atlas.media']` | attachment (video) | Where the check's steps are in the recording, so a click on a step seeks the video. See [Place steps in a recording](#place-steps-in-a-recording). |
31
32
  | `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
32
33
  | `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
@@ -208,7 +209,8 @@ curl "$ATLAS_SERVER_URL/api/reports/artifacts?artifactTag=demo&type=video&domain
208
209
  - **Dimensions.** `artifactTag`, `type` (the media type as a role: `video`, `image`, `audio`, `snapshot`, `text`,
209
210
  `file`; `video` means a playable recording), `label` (`key=value`, or `key`), `domain` and `spec` (artifacts only),
210
211
  `environment` (the device or client), `producer`, `branch`, `sha` (seven or more characters), `outcome` (`passed`,
211
- `failed`, `running`, `not-run`), `since` and `until` (epoch milliseconds or an ISO date), and `q` (words).
212
+ `failed`, `running`, `not-run`), `recordings` (runs only: `complete`, `incomplete`, `none-expected` or `unknown`, see
213
+ [recording completeness](#recording-completeness)), `since` and `until` (epoch milliseconds or an ISO date), and `q` (words).
212
214
  - **Semantics.** Filters combine with AND across dimensions and OR within one (`environment=ios,android`). With both
213
215
  `artifactTag` and `type`, one attachment must be both where the report says which video carries which tag: a tagged
214
216
  screenshot beside an untagged video is not a tagged video. `inherited=1` also counts the tags an attachment inherits
@@ -475,7 +477,33 @@ publish; it never contains a path or a token.
475
477
  retry (publishing again reuses the stored files).
476
478
  - `atlas report status --format json` has `recordings`: `state` is `complete`, `incomplete`, `none-expected` or
477
479
  `unknown`, with counts and each gap. It never changes `ok`. A layer without a receipt is `unknown`, never `complete`.
478
- - The viewer shows `Recordings 2 of 3: incomplete` beside the run's origin, apart from its green or red outcomes.
479
-
480
- This is a statement about publication. Atlas does not check that a stored video plays in a browser, and does not yet
481
- re-check later that a stored file is still readable.
480
+ - The viewer shows `Recordings 2 of 3: incomplete` beside the run's origin and in each row of the Runs tab, apart from
481
+ its green or red outcomes.
482
+ - **Find runs by completeness.** `GET /api/reports/runs?recordings=incomplete` (and the Runs tab's *Recordings* filter,
483
+ `?tab=runs&recordings=incomplete`) keep the runs whose layers have a skipped or unresolved expected recording. The
484
+ values are `complete`, `incomplete`, `none-expected` and `unknown`, comma-separated for OR, and they combine with the
485
+ other filters by AND. A run whose layers declared no expectations is `unknown`, never `complete`, and a run with any
486
+ layer without a receipt cannot be `complete`. It is answered from the layers' receipts, before the page limit, and
487
+ never reads a recording; the facet counts say how many runs have each state. It is a property of a run, so the
488
+ artifacts list does not take it. A server that predates it ignores the parameter and does not name it in `filters`, and
489
+ the viewer then says the list is not narrowed by it.
490
+
491
+ ### Is the recording still there?
492
+
493
+ A receipt says a file was published. Whether it is still retrievable later is a separate question, and Atlas answers it
494
+ two ways without touching a check's result:
495
+
496
+ - **Layer bundle.** A video that names a `bundleKey` the layer's bundle does not list (the file was acknowledged and is
497
+ gone, or never arrived) is not offered to play. The report marks it `available: false` with `reason: "not-in-bundle"`,
498
+ the run counts it in `videos.unavailable` and `videos.missing`, and the Runs tab explains that the run stored the file
499
+ but no longer holds it, which is different from a reference that never named any bytes.
500
+ - **`atlas report verify-recordings`** reads the videos of the stored layers back and checks, for each, that the bundle
501
+ lists it, that its blob is still in the store, that its size and SHA-256 are the recorded ones, and that it begins like
502
+ the container its media type names (WebM, MP4, QuickTime, Ogg, AVI). It exits 1 and names the layer and check for
503
+ every gap (`not-in-bundle`, `blob-missing`, `unreadable`, `size-mismatch`, `checksum-mismatch`, `undecodable`);
504
+ `--format json` prints the `verify-recordings-result` document. `--layer <id>` and `--limit <n>` narrow the work, since
505
+ it reads blobs. A store that cannot answer is `unreadable`, never verified. Test results are unchanged either way.
506
+
507
+ Both are statements about stored bytes. Atlas does not check that a stored video plays in a given browser (a valid header
508
+ can still have an unsupported codec or be truncated), and acknowledging an upload is not playback. Videos given only by
509
+ `uri` are counted as referenced and not checked.
package/docs/server.md CHANGED
@@ -410,7 +410,7 @@ with `format=markdown` (or `Accept: text/markdown`), and takes the project prefi
410
410
  | One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
411
411
  | How reliable one test is in a scope (environment, producer, parameters, branch, commit, spec revision, time window): independent executions, first-try and eventual success, what was left out and why | `GET /api/reports/tests/reliability?key=<key>&environment=phone` |
412
412
  | Why Specs, domains and counts disagree: ambiguous spec paths, shared domains, gaps, a reconciliation of the totals | `GET /api/reports/tests/health` |
413
- | The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
413
+ | The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `recordings` (completeness of the recordings the run's layers expected), `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
414
414
  | Recorded results that carry evidence, across the whole history, found by tag, media type, label, feature, environment, branch, revision, outcome and time | `GET /api/reports/artifacts?artifactTag=demo&type=video` |
415
415
  | Every test one run recorded | `GET /api/reports/runs/detail?key=<run key>` |
416
416
 
package/docs/ui.md CHANGED
@@ -104,6 +104,37 @@ when Watch is on screen.
104
104
  - The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
105
105
  none are listed under "No domain declared"; the Tests view reports domain problems).
106
106
 
107
+ ## Behaviors across the project
108
+
109
+ `/behaviors` (`evidence/behaviors-page.tsx`, **Behaviors** in the spec-type menu) lists every behavior of every Gherkin feature in one
110
+ place, so the question "what does this product promise?" needs no knowledge of which file holds the answer. It adds no source of
111
+ truth: `evidence/behavior-index.ts` reads each feature through the same `featureBehaviors` the feature page uses, so a behavior has the
112
+ same coverage state, revisions and recording in both.
113
+
114
+ - The list is searched with the shared matcher (`find/`: words, quoted phrases, `status:`, `domain:`, `tag:`, `has:video`), grouped by
115
+ the domain spec metadata declares ("No domain declared" when it declares none), or ranked best match first when words are typed.
116
+ A coverage state is a filter with its own count; `demonstrated` is always named. A long list is bounded with *Show more*.
117
+ - Features whose text has not arrived are counted ("3 of 12 read"), and features that cannot be parsed are named, so an empty answer is
118
+ never taken for "no such behavior" while it is still incomplete.
119
+ - The search words (`q`), the coverage filter (`coverage`), the selection (`behavior`) and the expanded state (`workspace`) are in the
120
+ address. Selecting shows the Read / Watch / Inspect panel beside the list (below it on narrow screens), and *Read and watch* in the
121
+ selected-behavior bar is reachable before the list in keyboard order.
122
+ - **Read and watch workspace.** Expanding a behavior replaces the index with a page that uses the width: a two-column
123
+ `SimpleGrid` (one column below `md`) with *What should happen* (the steps, an outline's example filled in) beside *Recording* (the
124
+ player, status and provenance, promise and evidence revisions, applicability), the scoped summary below, and Inspect behind a
125
+ disclosure that reads the technical record only when opened. Keyboard focus moves to the behavior's heading on arrival, and
126
+ *Previous behavior* / *Next behavior* move through the same search. A recording is labelled illustrative.
127
+ - **Returning** from the workspace restores the search words, the filter and the selection from the address, and the scroll position,
128
+ the number of rows shown and the focused row from `evidence/behavior-return.ts`, which remembers them for the session against the
129
+ scope they were taken in and gives them back once, only to that scope. Storage that is unavailable means the list opens at its top.
130
+ - **Scoped summary** (`evidence/scoped-summary.ts`, `scoped-summary-view.tsx`, on Watch and in the workspace) answers three questions
131
+ apart, for the check in the environment of the recording shown (environments are not pooled; with none named it says they are):
132
+ the **outcome** of the shown result (`outcomeHeadline`), **reliability** (`reports/reliability`: always with the independent
133
+ executions it rests on, "Insufficient evidence" below the minimum, first-try success only from recorded tries, the library's
134
+ warnings) and **performance** (`reports/latency`: only measurements named `product-latency`, percentiles withheld with the reason,
135
+ budget and breaches). Absent evidence is stated as unknown, never as good; the test's duration is not product latency, and the
136
+ summary says it describes the checks Atlas holds, not production. The full reliability and latency detail is one link away.
137
+
107
138
  ## Catalog health
108
139
 
109
140
  The **Catalog health** tab (`explorer/HealthView.tsx`, `?tab=health`; `@crvouga/atlas/reports/catalog-health`) explains why the Specs
@@ -209,7 +240,14 @@ over the same filtered set the page lists, in the same order (`explorer/playlist
209
240
  demonstrated. A set longer than the page's limit says it is only the newest results.
210
241
  - Only the selected clip is read: the list is the metadata the catalog already sent, the selected result's evidence is read
211
242
  when it is selected, and the one `<video>` uses `preload="metadata"`. Opening the playlist downloads no recordings in bulk.
212
- Posters and durations of unselected entries are not shown for that reason.
243
+ An unselected entry still shows a poster and a length when the catalog already carries them: the producer's declared
244
+ `Attachment.durationMs` and `Attachment.poster` (an image of the same list that the layer's bundle holds) travel as the
245
+ optional `durationMs` and `poster` of the result's video reference. The poster is an `<img loading="lazy">`, fetched when
246
+ its row nears the screen; an entry whose producer declared neither shows neither, and nothing is read to find out.
247
+ - The open playlist is in the address (`watch=true`, and `watchClip` for the entry playing), so a link restores it over
248
+ the same filters, run or collection. Opening it adds a history entry (Back closes it); moving between clips and closing
249
+ it replace the current one. A `watchClip` the set no longer holds opens the first watchable entry and the address then
250
+ names that one.
213
251
  - One player: selecting another clip replaces it, and the position of the clip left is kept for the session so returning
214
252
  resumes there. Previous/next and advancing on end move over the watchable entries; a result with nothing to play is
215
253
  selected from the list to read why.
@@ -350,6 +388,7 @@ The narrowly scoped exceptions are:
350
388
  | Project home | Tests/runs/artifacts/compare/catalog health tabs, a two-run comparison with promise, outcome, recording and figure tabs per behavior, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
351
389
  | Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
352
390
  | Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
391
+ | Behaviors | Project-wide behavior index with search, coverage filters and a selected behavior's Read / Watch / Inspect; an expanded read-and-watch workspace with a scoped outcome, reliability and performance summary that restores scope, scroll, selection and focus on return | Container, Grid, SimpleGrid, NavLink, TextInput, Button, Paper, Accordion, Tabs |
353
392
  | Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
354
393
  | Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |
355
394
  | Recordings | Pictures/video/timeline, actor selection, moments, failure jumps, timing, model-state differences | Tabs, Slider, Button, ScrollArea, Table, Progress |