@crvouga/atlas 1.17.0 → 1.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -1
- package/dist/reports/src/artifact-query.d.ts +14 -2
- package/dist/reports/src/artifact-query.js +13 -4
- package/dist/reports/src/evidence-detail.js +19 -6
- package/dist/reports/src/evidence-types.d.ts +8 -0
- package/dist/reports/src/recording-receipt.d.ts +23 -0
- package/dist/reports/src/recording-receipt.js +16 -0
- package/dist/reports/src/recording-verify.d.ts +74 -0
- package/dist/reports/src/recording-verify.js +142 -0
- package/dist/reports/src/report-cli.d.ts +2 -2
- package/dist/reports/src/report-cli.js +53 -3
- package/dist/reports/src/test-catalog.d.ts +7 -1
- package/dist/reports/src/test-catalog.js +10 -4
- package/dist/schema/src/api.d.ts +134 -0
- package/dist/schema/src/api.js +13 -2
- package/dist/schema/src/cli-results.d.ts +51 -1
- package/dist/schema/src/cli-results.js +25 -1
- package/dist/server/src/openapi.js +1 -0
- package/docs/agents.md +1 -0
- package/docs/reference/cli.md +11 -2
- package/docs/reference/http-api.md +2 -0
- package/docs/reference/reports.md +189 -9
- package/docs/reference/schema.md +19 -1
- package/docs/rich-evidence.md +33 -5
- package/docs/server.md +1 -1
- package/docs/ui.md +40 -1
- package/llms-full.txt +88 -9
- package/package.json +1 -1
- package/packages/schema/schemas/effective-delta.schema.json +26 -0
- package/packages/schema/schemas/effective-evidence.schema.json +26 -0
- package/packages/schema/schemas/evidence-page.schema.json +13 -0
- package/packages/schema/schemas/layer-rows.schema.json +13 -0
- package/packages/schema/schemas/report-run.schema.json +83 -0
- package/packages/schema/schemas/report-runs.schema.json +86 -0
- package/packages/schema/schemas/review-collection.schema.json +13 -0
- package/packages/schema/schemas/review-manifest.schema.json +13 -0
- package/packages/schema/schemas/test-catalog.schema.json +13 -0
- package/packages/schema/schemas/test-history.schema.json +26 -0
- package/packages/schema/schemas/verify-recordings-result.schema.json +97 -0
- package/packages/server/openapi.json +378 -0
- package/packages/visualizer/src/app/AppShell.tsx +2 -2
- package/packages/visualizer/src/app/router.tsx +25 -1
- package/packages/visualizer/src/components/Header.tsx +15 -4
- package/packages/visualizer/src/evidence/behavior-index.test.ts +119 -0
- package/packages/visualizer/src/evidence/behavior-index.ts +162 -0
- package/packages/visualizer/src/evidence/behavior-panel.tsx +36 -22
- package/packages/visualizer/src/evidence/behavior-return.test.ts +41 -0
- package/packages/visualizer/src/evidence/behavior-return.ts +59 -0
- package/packages/visualizer/src/evidence/behavior.ts +32 -1
- package/packages/visualizer/src/evidence/behaviors-page.tsx +465 -0
- package/packages/visualizer/src/evidence/feature-page.tsx +5 -26
- package/packages/visualizer/src/evidence/scoped-summary-view.tsx +110 -0
- package/packages/visualizer/src/evidence/scoped-summary.test.ts +62 -0
- package/packages/visualizer/src/evidence/scoped-summary.ts +118 -0
- package/packages/visualizer/src/evidence/spec-texts.ts +88 -0
- package/packages/visualizer/src/explorer/ArtifactsView.tsx +7 -6
- package/packages/visualizer/src/explorer/Explorer.tsx +1 -1
- package/packages/visualizer/src/explorer/RunsView.tsx +32 -7
- package/packages/visualizer/src/explorer/collections.tsx +5 -5
- package/packages/visualizer/src/explorer/filters.tsx +11 -3
- package/packages/visualizer/src/explorer/navigate.ts +17 -0
- package/packages/visualizer/src/explorer/playlist-entries.test.ts +10 -1
- package/packages/visualizer/src/explorer/playlist-entries.ts +10 -0
- package/packages/visualizer/src/explorer/playlist.tsx +43 -8
- package/packages/visualizer/src/explorer/search.ts +13 -3
- package/packages/visualizer/src/find/Finder.tsx +1 -62
package/docs/reference/cli.md
CHANGED
|
@@ -105,6 +105,8 @@ atlas report — layered evidence for any kind of spec
|
|
|
105
105
|
[--specs <dir>] [--format text|json|markdown]
|
|
106
106
|
[--require-environments a,b] [--require-all-specs]
|
|
107
107
|
[--branch <name>]... [--ignore-modified]
|
|
108
|
+
atlas report verify-recordings [--store <dir|s3://…|id=url>]... [--backends <module>] [--layer <id>]...
|
|
109
|
+
[--limit <n>] [--branch <name>]... [--ignore-modified] [--format text|json]
|
|
108
110
|
atlas report merge [--store …]... [--backends <module>] --into <dir|s3://…>
|
|
109
111
|
atlas report doctor [--store …]... [--role publisher|reader|both]
|
|
110
112
|
atlas report policy --store s3://bucket/prefix [--role publisher|reader|both]
|
|
@@ -148,6 +150,13 @@ atlas report — layered evidence for any kind of spec
|
|
|
148
150
|
--require-all-specs fail while any spec has no results
|
|
149
151
|
--branch <name> only layers recorded on this branch (repeatable)
|
|
150
152
|
--ignore-modified leave out layers from modified checkouts and unknown origins
|
|
153
|
+
verify-recordings Read back the videos the layers published and check they are still there as published: the
|
|
154
|
+
layer still lists the file, its blob is still stored, its size and SHA-256 are the recorded ones, and it
|
|
155
|
+
begins like the container its media type names. Exit 1 when any video is missing, unreadable, corrupt or
|
|
156
|
+
not that container, naming the layer and check; test results are never changed. This checks stored bytes,
|
|
157
|
+
not playback in a browser. Videos given only by address are counted, not checked.
|
|
158
|
+
--layer <id> only the layer whose id starts with this (repeatable; at least 7 digits)
|
|
159
|
+
--limit <n> only the newest n layers
|
|
151
160
|
merge Copy the layers of several stores into one.
|
|
152
161
|
doctor Check that a store works: credentials, listing, writing, and what it holds.
|
|
153
162
|
policy Print the least-privilege IAM policy for publishers, readers, or both.
|
|
@@ -156,8 +165,8 @@ atlas report — layered evidence for any kind of spec
|
|
|
156
165
|
{ id, label?, store } (or a function returning it): a device lab, anything
|
|
157
166
|
that implements ObjectStore.
|
|
158
167
|
|
|
159
|
-
--format json (alias --json) publish, stream, merge, doctor and
|
|
160
|
-
(schemas: publish-result, stream-result, merge-result, doctor-result, report-status). A command
|
|
168
|
+
--format json (alias --json) publish, stream, merge, doctor, status and verify-recordings print one JSON document instead of prose
|
|
169
|
+
(schemas: publish-result, stream-result, merge-result, doctor-result, report-status, verify-recordings-result). A command
|
|
161
170
|
that is wrong is {"ok":false,"error":{"message":…,"code":"usage"}} on standard error.
|
|
162
171
|
A store may be an Atlas server (https://…): ATLAS_STORE=https://atlas.example.com, with ATLAS_TOKEN.
|
|
163
172
|
|
|
@@ -833,6 +833,7 @@ Every run, newest first: layers grouped by their CI run URL (else branch and com
|
|
|
833
833
|
- `branch` (query): Branches.
|
|
834
834
|
- `sha` (query): Revisions: a full commit id or a prefix of at least seven characters.
|
|
835
835
|
- `outcome` (query): Statuses: `passed`, `failed`, `running`, `not-run`. A run's outcome is `failed` when any result failed, else `running`, else `passed`, else `not-run`.
|
|
836
|
+
- `recordings` (query): Recording completeness of the run, apart from its outcome: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no recording expectations is `unknown`, never `complete`). Repeatable or comma-separated. Answered from the layers' receipts; no recording is read. A server that does not know it ignores it and does not name it in `filters`.
|
|
836
837
|
- `q` (query): Words that must all appear in the title, id, spec, branch, producer, environment, tags or labels.
|
|
837
838
|
- `since` (query): Only records at or after this time: epoch milliseconds or an ISO date (a date alone is UTC). A result's recording time; a run's end.
|
|
838
839
|
- `until` (query): Only records at or before this time (same forms as `since`).
|
|
@@ -861,6 +862,7 @@ Scoped to one project. The same project may be selected on the unprefixed API us
|
|
|
861
862
|
- `branch` (query): Branches.
|
|
862
863
|
- `sha` (query): Revisions: a full commit id or a prefix of at least seven characters.
|
|
863
864
|
- `outcome` (query): Statuses: `passed`, `failed`, `running`, `not-run`. A run's outcome is `failed` when any result failed, else `running`, else `passed`, else `not-run`.
|
|
865
|
+
- `recordings` (query): Recording completeness of the run, apart from its outcome: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no recording expectations is `unknown`, never `complete`). Repeatable or comma-separated. Answered from the layers' receipts; no recording is read. A server that does not know it ignores it and does not name it in `filters`.
|
|
864
866
|
- `q` (query): Words that must all appear in the title, id, spec, branch, producer, environment, tags or labels.
|
|
865
867
|
- `since` (query): Only records at or after this time: epoch milliseconds or an ISO date (a date alone is UTC). A result's recording time; a run's end.
|
|
866
868
|
- `until` (query): Only records at or before this time (same forms as `since`).
|
|
@@ -10,7 +10,7 @@ Source: `packages/reports/src/artifact-query.ts`. Import: `@crvouga/atlas/report
|
|
|
10
10
|
|
|
11
11
|
One shareable query for the runs of a project and for the recorded results that carry evidence (recordings,
|
|
12
12
|
screenshots, traces): by artifact tag, label, media type, feature (domain, spec), environment or device, producer,
|
|
13
|
-
branch, revision, outcome and
|
|
13
|
+
branch, revision, outcome, time and (runs only) recording completeness. The same rules answer `GET /api/reports/runs`, `GET /api/reports/artifacts`
|
|
14
14
|
and the viewer's Runs and Artifacts views, so a link means the same thing everywhere.
|
|
15
15
|
|
|
16
16
|
The semantics are fixed, not inferred:
|
|
@@ -81,6 +81,12 @@ export type ArtifactFilter = {
|
|
|
81
81
|
readonly domain?: readonly string[];
|
|
82
82
|
/** Spec paths. Not applicable to runs. */
|
|
83
83
|
readonly spec?: readonly string[];
|
|
84
|
+
/**
|
|
85
|
+
* Recording completeness of a run: `complete`, `incomplete`, `none-expected` or `unknown` (a run whose layers declared no
|
|
86
|
+
* recording expectations is `unknown`, never `complete`). Answered from the layers' receipts alone, without reading any
|
|
87
|
+
* recording. Applies to runs only; it is independent of `outcome`.
|
|
88
|
+
*/
|
|
89
|
+
readonly recordings?: readonly string[];
|
|
84
90
|
/** Only records at or after this time (epoch ms): a result's recording time, a run's end. */
|
|
85
91
|
readonly since?: number;
|
|
86
92
|
/** Only records at or before this time (epoch ms). */
|
|
@@ -124,6 +130,8 @@ export type FilterFields = {
|
|
|
124
130
|
/** Absent when the dimension does not apply (runs have no domain). */
|
|
125
131
|
readonly domains?: readonly string[];
|
|
126
132
|
readonly specs?: readonly string[];
|
|
133
|
+
/** The recording completeness of a run (one value). Absent when the dimension does not apply (results). */
|
|
134
|
+
readonly recordings?: readonly string[];
|
|
127
135
|
readonly at: number;
|
|
128
136
|
};
|
|
129
137
|
```
|
|
@@ -143,6 +151,8 @@ export type QueryFacets = {
|
|
|
143
151
|
readonly branches: readonly QueryFacetValue[];
|
|
144
152
|
readonly outcomes: readonly QueryFacetValue[];
|
|
145
153
|
readonly domains?: readonly QueryFacetValue[];
|
|
154
|
+
/** Runs only: `complete`, `incomplete`, `none-expected` and `unknown`, with how many runs have each. */
|
|
155
|
+
readonly recordings?: readonly QueryFacetValue[];
|
|
146
156
|
};
|
|
147
157
|
```
|
|
148
158
|
|
|
@@ -204,6 +214,14 @@ export function usedFilters(filter: ArtifactFilter): FilterName[]
|
|
|
204
214
|
|
|
205
215
|
Which filters a query uses, by the names an address gives them.
|
|
206
216
|
|
|
217
|
+
#### `forArtifacts`
|
|
218
|
+
|
|
219
|
+
```ts
|
|
220
|
+
export function forArtifacts(filter: ArtifactFilter): ArtifactFilter
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Every filter in `filter` that may be applied to a recorded result, as the artifacts query applies them (no recording completeness, which is a property of a run).
|
|
224
|
+
|
|
207
225
|
#### `isFiltered`
|
|
208
226
|
|
|
209
227
|
```ts
|
|
@@ -218,7 +236,7 @@ Whether a query narrows anything.
|
|
|
218
236
|
export function forRuns(filter: ArtifactFilter): ArtifactFilter
|
|
219
237
|
```
|
|
220
238
|
|
|
221
|
-
Every filter in `filter` that may be applied to a
|
|
239
|
+
Every filter in `filter` that may be applied to a run, as the runs query applies them (no domain or spec).
|
|
222
240
|
|
|
223
241
|
#### `matchesFilter`
|
|
224
242
|
|
|
@@ -2161,6 +2179,12 @@ export type VideoRef = {
|
|
|
2161
2179
|
readonly recording?: string;
|
|
2162
2180
|
/** False when the reference names no bytes at all (no bundle key, address or inline text), or a bundle key that the layer's own bundle does not hold, so it can never be played. */
|
|
2163
2181
|
readonly available: boolean;
|
|
2182
|
+
/** Why it is unavailable, when the report can tell: `not-in-bundle` names a file the layer should hold (its bundle key) but does not, a recording that was acknowledged and is no longer there. Absent otherwise, and for an available video. An open vocabulary: treat an unknown reason as plain unavailable. */
|
|
2183
|
+
readonly reason?: 'not-in-bundle';
|
|
2184
|
+
/** How long the recording is, in milliseconds, as its producer declared (`Attachment.durationMs`): a hint to show before the file is opened, never measured here and never a latency. Absent when none was declared, and in documents of older servers. */
|
|
2185
|
+
readonly durationMs?: number;
|
|
2186
|
+
/** The bundle key, in the same layer, of the image the producer named as this recording's poster (`Attachment.poster`): a hint to show before the file is opened. Only an image the layer's bundle holds; absent when there is none, and in documents of older servers. */
|
|
2187
|
+
readonly poster?: string;
|
|
2164
2188
|
/** What this recording shows, as declared for it: its own `scope`, else its attempt's, check's, part's or layer's. Absent when none declared one: its scope is unknown, not live, physical or audible. */
|
|
2165
2189
|
readonly scope?: EvidenceScope;
|
|
2166
2190
|
};
|
|
@@ -2180,6 +2204,8 @@ export type VideoCounts = {
|
|
|
2180
2204
|
readonly recordings: number;
|
|
2181
2205
|
/** References that name no bytes, or bytes the layer does not hold. A report that predates availability metadata has none of these and none of the others. */
|
|
2182
2206
|
readonly unavailable: number;
|
|
2207
|
+
/** Of `unavailable`, those that name a file the layer's bundle does not hold (stored, then gone). Absent when there are none. */
|
|
2208
|
+
readonly missing?: number;
|
|
2183
2209
|
};
|
|
2184
2210
|
```
|
|
2185
2211
|
|
|
@@ -6408,6 +6434,160 @@ skipped or unresolved; otherwise `unknown` when a contributing layer has no rece
|
|
|
6408
6434
|
report with no results at all is unknown too); otherwise `complete` when something was expected and `none-expected` when
|
|
6409
6435
|
nothing was. Gaps are sorted so the output never depends on read order.
|
|
6410
6436
|
|
|
6437
|
+
#### `RunRecordings`
|
|
6438
|
+
|
|
6439
|
+
```ts
|
|
6440
|
+
export type RunRecordings = {
|
|
6441
|
+
/** `incomplete`: a recording is skipped or unresolved. `unknown`: no gap is known, but a layer declared no expectations. `complete`: everything expected was published. `none-expected`: every layer declared expectations and none required a recording. */
|
|
6442
|
+
readonly state: 'complete' | 'incomplete' | 'none-expected' | 'unknown';
|
|
6443
|
+
/** Recordings the layers expected. */
|
|
6444
|
+
readonly expected: number;
|
|
6445
|
+
/** Of those, uploaded or referenced by address. */
|
|
6446
|
+
readonly published: number;
|
|
6447
|
+
readonly skipped: number;
|
|
6448
|
+
readonly unresolved: number;
|
|
6449
|
+
/** Intentionally unrecorded checks: neither expected nor missing. */
|
|
6450
|
+
readonly notApplicable: number;
|
|
6451
|
+
/** Layers of the run that declared no expectations: nothing is claimed for them. */
|
|
6452
|
+
readonly layersWithoutReceipt: number;
|
|
6453
|
+
};
|
|
6454
|
+
```
|
|
6455
|
+
|
|
6456
|
+
A run's recording completeness, from the receipts of its layers (see {@link runRecordings}).
|
|
6457
|
+
|
|
6458
|
+
#### `runRecordings`
|
|
6459
|
+
|
|
6460
|
+
```ts
|
|
6461
|
+
export function runRecordings(layers: readonly (RecordingCounts | undefined)[]): RunRecordings | undefined
|
|
6462
|
+
```
|
|
6463
|
+
|
|
6464
|
+
The recording completeness of a run, from the `recordings` counts of each of its layers (`undefined` for a layer that
|
|
6465
|
+
has no receipt). `undefined` when no layer has a receipt: the run claims nothing, which a reader treats as `unknown`.
|
|
6466
|
+
A gap always wins (`incomplete`); otherwise a layer without a receipt keeps the run `unknown`, never `complete`. Reads
|
|
6467
|
+
only the counts, never a recording, and never changes a status.
|
|
6468
|
+
|
|
6469
|
+
### Recording verification
|
|
6470
|
+
|
|
6471
|
+
Source: `packages/reports/src/recording-verify.ts`. Import: `@crvouga/atlas/reports/recording-verify`
|
|
6472
|
+
|
|
6473
|
+
Answers "is the recording we published still there, and does it still look like the video we published?" apart from
|
|
6474
|
+
"did the tests pass?" and apart from "was it uploaded?" (the receipt, see `recording-receipt.ts`). `verifyLayerRecordings`
|
|
6475
|
+
reads each video a layer keeps in its bundle back from the store and checks, in order, that the layer's bundle still
|
|
6476
|
+
lists its file, that the blob it lives in is still stored, that its size and SHA-256 are the ones recorded at upload,
|
|
6477
|
+
and that its first bytes are the container header its media type promises. A problem is a gap in the evidence, named
|
|
6478
|
+
with the check that holds the video; it never changes a status, a slot or a winner.
|
|
6479
|
+
|
|
6480
|
+
Trap: this is a stored-bytes check, not a playback test. A file with a valid header can still fail to play in a given
|
|
6481
|
+
browser (codec, truncation), and acknowledging an upload is not playback either. Videos given only by `uri` are counted
|
|
6482
|
+
as `referenced`, not checked: Atlas holds no bytes to read back.
|
|
6483
|
+
|
|
6484
|
+
See docs/rich-evidence.md#recording-completeness
|
|
6485
|
+
|
|
6486
|
+
#### `RecordingProblem`
|
|
6487
|
+
|
|
6488
|
+
```ts
|
|
6489
|
+
export type RecordingProblem =
|
|
6490
|
+
/** The layer names the file but its bundle does not list it: stored, then gone (or never arrived). */
|
|
6491
|
+
| 'not-in-bundle'
|
|
6492
|
+
/** The bundle lists the file but a blob holding its bytes is no longer in the store (a 404 when played). */
|
|
6493
|
+
| 'blob-missing'
|
|
6494
|
+
/** The store could not be read just now. Not proof the bytes are gone, and not a pass. */
|
|
6495
|
+
| 'unreadable'
|
|
6496
|
+
/** The stored bytes are not as many as were recorded. */
|
|
6497
|
+
| 'size-mismatch'
|
|
6498
|
+
/** The stored bytes do not hash to the SHA-256 recorded at upload. */
|
|
6499
|
+
| 'checksum-mismatch'
|
|
6500
|
+
/** The stored bytes do not begin like the container their media type names, or are empty: incompatible or corrupt. */
|
|
6501
|
+
| 'undecodable';
|
|
6502
|
+
```
|
|
6503
|
+
|
|
6504
|
+
What is wrong with a stored recording. An open vocabulary: a reader treats an unknown problem as a gap.
|
|
6505
|
+
|
|
6506
|
+
#### `RecordingGap`
|
|
6507
|
+
|
|
6508
|
+
```ts
|
|
6509
|
+
export type RecordingGap = {
|
|
6510
|
+
/** The check that holds the video, or `null` for a video attached to the layer or a part. */
|
|
6511
|
+
readonly check: string | null;
|
|
6512
|
+
/** The attachment's name. */
|
|
6513
|
+
readonly name: string;
|
|
6514
|
+
readonly bundleKey: string;
|
|
6515
|
+
readonly problem: RecordingProblem;
|
|
6516
|
+
/** A short, non-secret explanation: sizes or the container found. */
|
|
6517
|
+
readonly reason: string;
|
|
6518
|
+
};
|
|
6519
|
+
```
|
|
6520
|
+
|
|
6521
|
+
One video of a layer that failed verification.
|
|
6522
|
+
|
|
6523
|
+
#### `LayerRecordingVerification`
|
|
6524
|
+
|
|
6525
|
+
```ts
|
|
6526
|
+
export type LayerRecordingVerification = {
|
|
6527
|
+
readonly layer: string;
|
|
6528
|
+
/** Videos read back from the store: those that name a bundle key. */
|
|
6529
|
+
readonly checked: number;
|
|
6530
|
+
/** Of those, the ones that passed every check. */
|
|
6531
|
+
readonly ok: number;
|
|
6532
|
+
/** Videos given only by `uri`, which Atlas cannot read back. */
|
|
6533
|
+
readonly referenced: number;
|
|
6534
|
+
readonly gaps: readonly RecordingGap[];
|
|
6535
|
+
};
|
|
6536
|
+
```
|
|
6537
|
+
|
|
6538
|
+
What verifying one layer found.
|
|
6539
|
+
|
|
6540
|
+
#### `containerProblem`
|
|
6541
|
+
|
|
6542
|
+
```ts
|
|
6543
|
+
export function containerProblem(bytes: Uint8Array, contentType: string): string | null
|
|
6544
|
+
```
|
|
6545
|
+
|
|
6546
|
+
Whether the first bytes of a stored video are the container its media type names, as a reason it is not, or `null` when
|
|
6547
|
+
it looks right or the type is one this check has no signature for (then nothing is claimed). Recognizes WebM and
|
|
6548
|
+
Matroska (EBML), MP4, QuickTime and 3GPP (an ISO box), Ogg and AVI. An empty file is never a video.
|
|
6549
|
+
|
|
6550
|
+
#### `verifyLayerRecordings`
|
|
6551
|
+
|
|
6552
|
+
```ts
|
|
6553
|
+
export async function verifyLayerRecordings(store: ObjectStore, layer: ReportLayer): Promise<LayerRecordingVerification>
|
|
6554
|
+
```
|
|
6555
|
+
|
|
6556
|
+
Reads back every video of `layer` that is kept in its bundle and reports the ones that are no longer retrievable or no
|
|
6557
|
+
longer look like what was published (see {@link RecordingProblem}). Each blob is read once however many checks
|
|
6558
|
+
reference it. A store that fails to answer is `unreadable`, never a pass. It only reads: nothing is written, and no
|
|
6559
|
+
status changes.
|
|
6560
|
+
|
|
6561
|
+
**Example**
|
|
6562
|
+
|
|
6563
|
+
```ts
|
|
6564
|
+
const layer = await readLayer(store, id);
|
|
6565
|
+
const { gaps } = await verifyLayerRecordings(store, layer!);
|
|
6566
|
+
gaps.map(gap => `${gap.check}: ${gap.problem}`);
|
|
6567
|
+
```
|
|
6568
|
+
|
|
6569
|
+
#### `StoreRecordingVerification`
|
|
6570
|
+
|
|
6571
|
+
```ts
|
|
6572
|
+
export type StoreRecordingVerification = {
|
|
6573
|
+
readonly layers: readonly LayerRecordingVerification[];
|
|
6574
|
+
/** Layers that could not be read at all (their manifest is unreadable or gone): their recordings are unknown, not verified. */
|
|
6575
|
+
readonly notices: readonly string[];
|
|
6576
|
+
};
|
|
6577
|
+
```
|
|
6578
|
+
|
|
6579
|
+
What verifying the layers of one store found.
|
|
6580
|
+
|
|
6581
|
+
#### `verifyStoreRecordings`
|
|
6582
|
+
|
|
6583
|
+
```ts
|
|
6584
|
+
export async function verifyStoreRecordings(store: ObjectStore, options: { readonly ids?: readonly string[]; readonly limit?: number; readonly include?: (layer: ReportLayer) => boolean } = {}): Promise<StoreRecordingVerification>
|
|
6585
|
+
```
|
|
6586
|
+
|
|
6587
|
+
Verifies the recordings of the layers a store lists, newest first. `ids` narrows to layers whose id starts with one of
|
|
6588
|
+
them (at least 7 hex digits); `limit` keeps only that many of the newest. A layer that cannot be read is a notice, never
|
|
6589
|
+
skipped silently and never counted as verified. A store that cannot be listed rejects.
|
|
6590
|
+
|
|
6411
6591
|
### Reliability of a test
|
|
6412
6592
|
|
|
6413
6593
|
Source: `packages/reports/src/reliability.ts`. Import: `@crvouga/atlas/reports/reliability`
|
|
@@ -6767,7 +6947,7 @@ Call this inside a Solid root: it creates signals and a store.
|
|
|
6767
6947
|
|
|
6768
6948
|
Source: `packages/reports/src/report-cli.ts`. Import: `@crvouga/atlas/reports/report-cli`
|
|
6769
6949
|
|
|
6770
|
-
`reportMain` is `atlas report publish | stream | status | merge | doctor | policy`, written so it can be called from code and
|
|
6950
|
+
`reportMain` is `atlas report publish | stream | status | verify-recordings | merge | doctor | policy`, written so it can be called from code and
|
|
6771
6951
|
tests as well as a terminal: it takes its arguments, environment, working directory, clock, git and standard input as
|
|
6772
6952
|
parameters and returns the exit code. Every command can print a JSON document (`--format json`) instead of prose, and a
|
|
6773
6953
|
command that is wrong throws, which `runReportCommand` turns into a message (or a JSON error) and exit code 2.
|
|
@@ -9942,14 +10122,14 @@ export type RunSummary = {
|
|
|
9942
10122
|
* and references that cannot be played. Absent when it has none, and in a report that carries no video information.
|
|
9943
10123
|
*/
|
|
9944
10124
|
readonly videos?: VideoCounts;
|
|
10125
|
+
/**
|
|
10126
|
+
* Whether the recordings its layers expected were published, apart from its outcomes (see `runRecordings`). Absent when
|
|
10127
|
+
* none of its layers declared a recording expectation: its completeness is unknown, never complete. Never changes a status.
|
|
10128
|
+
*/
|
|
10129
|
+
readonly recordings?: RunRecordings;
|
|
9945
10130
|
/**
|
|
9946
10131
|
* Files attached to the run itself, in its layers (and their parts), not to any one check: review movies, archives,
|
|
9947
10132
|
* summaries. All kinds, videos included. Absent when none, and in a report that carries no such information.
|
|
9948
|
-
*/
|
|
9949
|
-
readonly attachments?: number;
|
|
9950
|
-
/** The normalized tags of the attachments of its results and layers, sorted and distinct. Absent when none is tagged. */
|
|
9951
|
-
readonly artifactTags?: readonly string[];
|
|
9952
|
-
/** The tags of its checks and layers, which its attachments inherit (what `inherited=1` also matches), sorted and distinct. Absent when none. */
|
|
9953
10133
|
…
|
|
9954
10134
|
```
|
|
9955
10135
|
|
|
@@ -10281,7 +10461,7 @@ What a query looks at in a run: see {@link ArtifactFilter}. A run's outcome is `
|
|
|
10281
10461
|
#### `artifactList`
|
|
10282
10462
|
|
|
10283
10463
|
```ts
|
|
10284
|
-
export function artifactList(history: TestHistory, version: string,
|
|
10464
|
+
export function artifactList(history: TestHistory, version: string, asked: ArtifactFilter = {}, page: { readonly limit?: number; readonly offset?: number; readonly domainOf?: (specPath: string) => string | undefined } = {}): ArtifactList
|
|
10285
10465
|
```
|
|
10286
10466
|
|
|
10287
10467
|
The recorded results that carry evidence and match a query, newest first, across the whole history (`limit` default
|
package/docs/reference/schema.md
CHANGED
|
@@ -535,7 +535,7 @@ of every layer. States are in id order and cells in profile order, so the same l
|
|
|
535
535
|
|
|
536
536
|
Source: `packages/schema/src/cli-results.ts`
|
|
537
537
|
|
|
538
|
-
What `atlas report publish`, `stream`, `merge` and `
|
|
538
|
+
What `atlas report publish`, `stream`, `merge`, `doctor` and `verify-recordings` print on standard output with `--format json` (alias
|
|
539
539
|
`--json`), and what any `atlas` command prints on standard error for a command that was wrong: one JSON document,
|
|
540
540
|
so an agent or a script never has to parse prose. (`status` and `policy` already print documents:
|
|
541
541
|
`ReportStatusSchema`, and an IAM policy.) Each is also published as a JSON Schema.
|
|
@@ -594,6 +594,24 @@ export const DoctorResultSchema = …
|
|
|
594
594
|
|
|
595
595
|
What `atlas report doctor --format json` prints.
|
|
596
596
|
|
|
597
|
+
#### `RecordingGapSchema`
|
|
598
|
+
|
|
599
|
+
```ts
|
|
600
|
+
export const RecordingGapSchema = …
|
|
601
|
+
```
|
|
602
|
+
|
|
603
|
+
A published video that could not be read back from the store as published (see `atlas report verify-recordings`).
|
|
604
|
+
|
|
605
|
+
#### `VerifyRecordingsResultSchema`
|
|
606
|
+
|
|
607
|
+
```ts
|
|
608
|
+
export const VerifyRecordingsResultSchema = …
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
What `atlas report verify-recordings --format json` prints: whether the videos the layers published can still be read
|
|
612
|
+
back from the store as they were published. It describes stored bytes, never playback in a browser, and it never changes
|
|
613
|
+
a check\'s result.
|
|
614
|
+
|
|
597
615
|
#### `CommandErrorSchema`
|
|
598
616
|
|
|
599
617
|
```ts
|
package/docs/rich-evidence.md
CHANGED
|
@@ -27,6 +27,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
|
|
|
27
27
|
| `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
|
|
28
28
|
| `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
|
|
29
29
|
| `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
|
|
30
|
+
| `poster` (the `id` of an image in the same list) and `durationMs` | attachment (video) | Shown beside the recording in a watch playlist before anything is opened: the report carries them as the optional `poster` (a bundle key) and `durationMs` of each video reference, so no evidence is read per entry. Neither is ever a latency. |
|
|
30
31
|
| `extensions['atlas.media']` | attachment (video) | Where the check's steps are in the recording, so a click on a step seeks the video. See [Place steps in a recording](#place-steps-in-a-recording). |
|
|
31
32
|
| `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
|
|
32
33
|
| `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
|
|
@@ -208,7 +209,8 @@ curl "$ATLAS_SERVER_URL/api/reports/artifacts?artifactTag=demo&type=video&domain
|
|
|
208
209
|
- **Dimensions.** `artifactTag`, `type` (the media type as a role: `video`, `image`, `audio`, `snapshot`, `text`,
|
|
209
210
|
`file`; `video` means a playable recording), `label` (`key=value`, or `key`), `domain` and `spec` (artifacts only),
|
|
210
211
|
`environment` (the device or client), `producer`, `branch`, `sha` (seven or more characters), `outcome` (`passed`,
|
|
211
|
-
`failed`, `running`, `not-run`), `
|
|
212
|
+
`failed`, `running`, `not-run`), `recordings` (runs only: `complete`, `incomplete`, `none-expected` or `unknown`, see
|
|
213
|
+
[recording completeness](#recording-completeness)), `since` and `until` (epoch milliseconds or an ISO date), and `q` (words).
|
|
212
214
|
- **Semantics.** Filters combine with AND across dimensions and OR within one (`environment=ios,android`). With both
|
|
213
215
|
`artifactTag` and `type`, one attachment must be both where the report says which video carries which tag: a tagged
|
|
214
216
|
screenshot beside an untagged video is not a tagged video. `inherited=1` also counts the tags an attachment inherits
|
|
@@ -475,7 +477,33 @@ publish; it never contains a path or a token.
|
|
|
475
477
|
retry (publishing again reuses the stored files).
|
|
476
478
|
- `atlas report status --format json` has `recordings`: `state` is `complete`, `incomplete`, `none-expected` or
|
|
477
479
|
`unknown`, with counts and each gap. It never changes `ok`. A layer without a receipt is `unknown`, never `complete`.
|
|
478
|
-
- The viewer shows `Recordings 2 of 3: incomplete` beside the run's origin
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
480
|
+
- The viewer shows `Recordings 2 of 3: incomplete` beside the run's origin and in each row of the Runs tab, apart from
|
|
481
|
+
its green or red outcomes.
|
|
482
|
+
- **Find runs by completeness.** `GET /api/reports/runs?recordings=incomplete` (and the Runs tab's *Recordings* filter,
|
|
483
|
+
`?tab=runs&recordings=incomplete`) keep the runs whose layers have a skipped or unresolved expected recording. The
|
|
484
|
+
values are `complete`, `incomplete`, `none-expected` and `unknown`, comma-separated for OR, and they combine with the
|
|
485
|
+
other filters by AND. A run whose layers declared no expectations is `unknown`, never `complete`, and a run with any
|
|
486
|
+
layer without a receipt cannot be `complete`. It is answered from the layers' receipts, before the page limit, and
|
|
487
|
+
never reads a recording; the facet counts say how many runs have each state. It is a property of a run, so the
|
|
488
|
+
artifacts list does not take it. A server that predates it ignores the parameter and does not name it in `filters`, and
|
|
489
|
+
the viewer then says the list is not narrowed by it.
|
|
490
|
+
|
|
491
|
+
### Is the recording still there?
|
|
492
|
+
|
|
493
|
+
A receipt says a file was published. Whether it is still retrievable later is a separate question, and Atlas answers it
|
|
494
|
+
two ways without touching a check's result:
|
|
495
|
+
|
|
496
|
+
- **Layer bundle.** A video that names a `bundleKey` the layer's bundle does not list (the file was acknowledged and is
|
|
497
|
+
gone, or never arrived) is not offered to play. The report marks it `available: false` with `reason: "not-in-bundle"`,
|
|
498
|
+
the run counts it in `videos.unavailable` and `videos.missing`, and the Runs tab explains that the run stored the file
|
|
499
|
+
but no longer holds it, which is different from a reference that never named any bytes.
|
|
500
|
+
- **`atlas report verify-recordings`** reads the videos of the stored layers back and checks, for each, that the bundle
|
|
501
|
+
lists it, that its blob is still in the store, that its size and SHA-256 are the recorded ones, and that it begins like
|
|
502
|
+
the container its media type names (WebM, MP4, QuickTime, Ogg, AVI). It exits 1 and names the layer and check for
|
|
503
|
+
every gap (`not-in-bundle`, `blob-missing`, `unreadable`, `size-mismatch`, `checksum-mismatch`, `undecodable`);
|
|
504
|
+
`--format json` prints the `verify-recordings-result` document. `--layer <id>` and `--limit <n>` narrow the work, since
|
|
505
|
+
it reads blobs. A store that cannot answer is `unreadable`, never verified. Test results are unchanged either way.
|
|
506
|
+
|
|
507
|
+
Both are statements about stored bytes. Atlas does not check that a stored video plays in a given browser (a valid header
|
|
508
|
+
can still have an unsupported codec or be truncated), and acknowledging an upload is not playback. Videos given only by
|
|
509
|
+
`uri` are counted as referenced and not checked.
|
package/docs/server.md
CHANGED
|
@@ -410,7 +410,7 @@ with `format=markdown` (or `Accept: text/markdown`), and takes the project prefi
|
|
|
410
410
|
| One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
|
|
411
411
|
| How reliable one test is in a scope (environment, producer, parameters, branch, commit, spec revision, time window): independent executions, first-try and eventual success, what was left out and why | `GET /api/reports/tests/reliability?key=<key>&environment=phone` |
|
|
412
412
|
| Why Specs, domains and counts disagree: ambiguous spec paths, shared domains, gaps, a reconciliation of the totals | `GET /api/reports/tests/health` |
|
|
413
|
-
| The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
|
|
413
|
+
| The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video; `artifactTag`, `type`, `label`, `environment`, `branch`, `outcome`, `recordings` (completeness of the recordings the run's layers expected), `since`, `until` narrow it ([filtering](rich-evidence.md#finding-runs-and-artifacts)) | `GET /api/reports/runs` |
|
|
414
414
|
| Recorded results that carry evidence, across the whole history, found by tag, media type, label, feature, environment, branch, revision, outcome and time | `GET /api/reports/artifacts?artifactTag=demo&type=video` |
|
|
415
415
|
| Every test one run recorded | `GET /api/reports/runs/detail?key=<run key>` |
|
|
416
416
|
|
package/docs/ui.md
CHANGED
|
@@ -104,6 +104,37 @@ when Watch is on screen.
|
|
|
104
104
|
- The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
|
|
105
105
|
none are listed under "No domain declared"; the Tests view reports domain problems).
|
|
106
106
|
|
|
107
|
+
## Behaviors across the project
|
|
108
|
+
|
|
109
|
+
`/behaviors` (`evidence/behaviors-page.tsx`, **Behaviors** in the spec-type menu) lists every behavior of every Gherkin feature in one
|
|
110
|
+
place, so the question "what does this product promise?" needs no knowledge of which file holds the answer. It adds no source of
|
|
111
|
+
truth: `evidence/behavior-index.ts` reads each feature through the same `featureBehaviors` the feature page uses, so a behavior has the
|
|
112
|
+
same coverage state, revisions and recording in both.
|
|
113
|
+
|
|
114
|
+
- The list is searched with the shared matcher (`find/`: words, quoted phrases, `status:`, `domain:`, `tag:`, `has:video`), grouped by
|
|
115
|
+
the domain spec metadata declares ("No domain declared" when it declares none), or ranked best match first when words are typed.
|
|
116
|
+
A coverage state is a filter with its own count; `demonstrated` is always named. A long list is bounded with *Show more*.
|
|
117
|
+
- Features whose text has not arrived are counted ("3 of 12 read"), and features that cannot be parsed are named, so an empty answer is
|
|
118
|
+
never taken for "no such behavior" while it is still incomplete.
|
|
119
|
+
- The search words (`q`), the coverage filter (`coverage`), the selection (`behavior`) and the expanded state (`workspace`) are in the
|
|
120
|
+
address. Selecting shows the Read / Watch / Inspect panel beside the list (below it on narrow screens), and *Read and watch* in the
|
|
121
|
+
selected-behavior bar is reachable before the list in keyboard order.
|
|
122
|
+
- **Read and watch workspace.** Expanding a behavior replaces the index with a page that uses the width: a two-column
|
|
123
|
+
`SimpleGrid` (one column below `md`) with *What should happen* (the steps, an outline's example filled in) beside *Recording* (the
|
|
124
|
+
player, status and provenance, promise and evidence revisions, applicability), the scoped summary below, and Inspect behind a
|
|
125
|
+
disclosure that reads the technical record only when opened. Keyboard focus moves to the behavior's heading on arrival, and
|
|
126
|
+
*Previous behavior* / *Next behavior* move through the same search. A recording is labelled illustrative.
|
|
127
|
+
- **Returning** from the workspace restores the search words, the filter and the selection from the address, and the scroll position,
|
|
128
|
+
the number of rows shown and the focused row from `evidence/behavior-return.ts`, which remembers them for the session against the
|
|
129
|
+
scope they were taken in and gives them back once, only to that scope. Storage that is unavailable means the list opens at its top.
|
|
130
|
+
- **Scoped summary** (`evidence/scoped-summary.ts`, `scoped-summary-view.tsx`, on Watch and in the workspace) answers three questions
|
|
131
|
+
apart, for the check in the environment of the recording shown (environments are not pooled; with none named it says they are):
|
|
132
|
+
the **outcome** of the shown result (`outcomeHeadline`), **reliability** (`reports/reliability`: always with the independent
|
|
133
|
+
executions it rests on, "Insufficient evidence" below the minimum, first-try success only from recorded tries, the library's
|
|
134
|
+
warnings) and **performance** (`reports/latency`: only measurements named `product-latency`, percentiles withheld with the reason,
|
|
135
|
+
budget and breaches). Absent evidence is stated as unknown, never as good; the test's duration is not product latency, and the
|
|
136
|
+
summary says it describes the checks Atlas holds, not production. The full reliability and latency detail is one link away.
|
|
137
|
+
|
|
107
138
|
## Catalog health
|
|
108
139
|
|
|
109
140
|
The **Catalog health** tab (`explorer/HealthView.tsx`, `?tab=health`; `@crvouga/atlas/reports/catalog-health`) explains why the Specs
|
|
@@ -209,7 +240,14 @@ over the same filtered set the page lists, in the same order (`explorer/playlist
|
|
|
209
240
|
demonstrated. A set longer than the page's limit says it is only the newest results.
|
|
210
241
|
- Only the selected clip is read: the list is the metadata the catalog already sent, the selected result's evidence is read
|
|
211
242
|
when it is selected, and the one `<video>` uses `preload="metadata"`. Opening the playlist downloads no recordings in bulk.
|
|
212
|
-
|
|
243
|
+
An unselected entry still shows a poster and a length when the catalog already carries them: the producer's declared
|
|
244
|
+
`Attachment.durationMs` and `Attachment.poster` (an image of the same list that the layer's bundle holds) travel as the
|
|
245
|
+
optional `durationMs` and `poster` of the result's video reference. The poster is an `<img loading="lazy">`, fetched when
|
|
246
|
+
its row nears the screen; an entry whose producer declared neither shows neither, and nothing is read to find out.
|
|
247
|
+
- The open playlist is in the address (`watch=true`, and `watchClip` for the entry playing), so a link restores it over
|
|
248
|
+
the same filters, run or collection. Opening it adds a history entry (Back closes it); moving between clips and closing
|
|
249
|
+
it replace the current one. A `watchClip` the set no longer holds opens the first watchable entry and the address then
|
|
250
|
+
names that one.
|
|
213
251
|
- One player: selecting another clip replaces it, and the position of the clip left is kept for the session so returning
|
|
214
252
|
resumes there. Previous/next and advancing on end move over the watchable entries; a result with nothing to play is
|
|
215
253
|
selected from the list to read why.
|
|
@@ -350,6 +388,7 @@ The narrowly scoped exceptions are:
|
|
|
350
388
|
| Project home | Tests/runs/artifacts/compare/catalog health tabs, a two-run comparison with promise, outcome, recording and figure tabs per behavior, a run's Artifacts (files of its layers, shared recordings, direct links), Watch playlist of a filtered set, metrics, search, status/environment/domain/artifact-tag filters, run and artifact facets with saved views, grouping, sorting, show more | Container, Tabs, SimpleGrid, Paper, TextInput, MultiSelect, Popover, Button, Accordion, Modal, Grid, Switch |
|
|
351
389
|
| Test detail | Current outcomes, statistics, duration graph, product latency with budgets and baselines, history, artifacts, recording; the `layer` address parameter pins it to one run's result, labelled "This run" against "Latest combined evidence" | Drawer, Table, Badge, Alert, Stage |
|
|
352
390
|
| Specs | All six existing kinds, overview/index with title search and folder or declared-domain grouping, Gherkin rules/scenarios/examples with a coverage summary and Read / Watch / Inspect per behavior, Quint declarations/traces, DMN tables, source text | SpecHeader, Accordion, Tabs, Table, CodeHighlightTabs, ResultRow |
|
|
391
|
+
| Behaviors | Project-wide behavior index with search, coverage filters and a selected behavior's Read / Watch / Inspect; an expanded read-and-watch workspace with a scoped outcome, reliability and performance summary that restores scope, scroll, selection and focus on return | Container, Grid, SimpleGrid, NavLink, TextInput, Button, Paper, Accordion, Tabs |
|
|
353
392
|
| Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
|
|
354
393
|
| Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |
|
|
355
394
|
| Recordings | Pictures/video/timeline, actor selection, moments, failure jumps, timing, model-state differences | Tabs, Slider, Button, ScrollArea, Table, Progress |
|