@crvouga/atlas 1.10.0 → 1.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -1
- package/dist/reports/src/review-collection-store.d.ts +47 -0
- package/dist/reports/src/review-collection-store.js +214 -0
- package/dist/reports/src/review-collection.d.ts +88 -0
- package/dist/reports/src/review-collection.js +144 -0
- package/dist/reports/src/test-catalog.d.ts +2 -0
- package/dist/reports/src/test-catalog.js +1 -1
- package/dist/schema/src/api.d.ts +574 -0
- package/dist/schema/src/api.js +152 -1
- package/dist/server/src/app.js +4 -1
- package/dist/server/src/openapi.js +46 -1
- package/dist/server/src/review-collections.d.ts +39 -0
- package/dist/server/src/review-collections.js +179 -0
- package/docs/extending.md +8 -1
- package/docs/reference/http-api.md +122 -0
- package/docs/reference/reports.md +294 -0
- package/docs/reference/schema.md +131 -0
- package/docs/reference/server.md +44 -0
- package/docs/rich-evidence.md +77 -0
- package/docs/server.md +41 -0
- package/docs/ui.md +67 -0
- package/llms-full.txt +315 -1
- package/package.json +1 -1
- package/packages/schema/schemas/review-collection.schema.json +398 -0
- package/packages/schema/schemas/review-manifest.schema.json +475 -0
- package/packages/server/openapi.json +6729 -4084
- package/packages/visualizer/src/app/router.tsx +9 -1
- package/packages/visualizer/src/evidence/applicability-view.tsx +233 -0
- package/packages/visualizer/src/evidence/applicability.test.ts +131 -0
- package/packages/visualizer/src/evidence/applicability.ts +241 -0
- package/packages/visualizer/src/evidence/artifact-tags.tsx +2 -1
- package/packages/visualizer/src/evidence/artifacts.tsx +88 -42
- package/packages/visualizer/src/evidence/behavior-panel.tsx +55 -26
- package/packages/visualizer/src/evidence/feature-page.tsx +25 -4
- package/packages/visualizer/src/evidence/media-timing.test.ts +85 -0
- package/packages/visualizer/src/evidence/media-timing.ts +150 -0
- package/packages/visualizer/src/evidence/moments.tsx +457 -0
- package/packages/visualizer/src/evidence/outcome-summary.tsx +27 -0
- package/packages/visualizer/src/evidence/outcome.ts +7 -2
- package/packages/visualizer/src/explorer/ArtifactsView.tsx +19 -5
- package/packages/visualizer/src/explorer/Explorer.tsx +7 -2
- package/packages/visualizer/src/explorer/RecordingTextSearch.tsx +355 -0
- package/packages/visualizer/src/explorer/TestDetail.tsx +2 -2
- package/packages/visualizer/src/explorer/collection-model.test.ts +31 -0
- package/packages/visualizer/src/explorer/collection-model.ts +45 -0
- package/packages/visualizer/src/explorer/collections.tsx +513 -0
- package/packages/visualizer/src/explorer/playlist-entries.ts +29 -0
- package/packages/visualizer/src/explorer/playlist.tsx +164 -37
- package/packages/visualizer/src/explorer/recording-text.test.ts +149 -0
- package/packages/visualizer/src/explorer/recording-text.ts +490 -0
- package/packages/visualizer/src/explorer/search.ts +17 -3
package/llms-full.txt
CHANGED
|
@@ -1014,6 +1014,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
|
|
|
1014
1014
|
| `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
|
|
1015
1015
|
| `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
|
|
1016
1016
|
| `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
|
|
1017
|
+
| `extensions['atlas.media']` | attachment (video) | Where the check's steps are in the recording, so a click on a step seeks the video. See [Place steps in a recording](#place-steps-in-a-recording). |
|
|
1017
1018
|
| `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
|
|
1018
1019
|
| `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
|
|
1019
1020
|
| `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
|
|
@@ -1096,6 +1097,44 @@ Every field is optional, and every value except the booleans is an open string.
|
|
|
1096
1097
|
- **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
|
|
1097
1098
|
dropped, and the attachment, check and layer stay.
|
|
1098
1099
|
|
|
1100
|
+
## Place steps in a recording
|
|
1101
|
+
|
|
1102
|
+
A step says when it started on the check's clock (`offsetMs`, after the check started). A video has its own clock, which
|
|
1103
|
+
began when the recorder did, and the two are not assumed to agree: a recorder usually starts before the first step, and a
|
|
1104
|
+
paced demo may have delays put into the video that the check never waited for. To let a reviewer click a step and land on
|
|
1105
|
+
its moment in the video, say how the clocks relate in the `atlas.media` extension of the video attachment:
|
|
1106
|
+
|
|
1107
|
+
```ts
|
|
1108
|
+
{
|
|
1109
|
+
name: 'checkout.webm', role: 'video', durationMs: 9000,
|
|
1110
|
+
extensions: {
|
|
1111
|
+
'atlas.media': {
|
|
1112
|
+
checkStartMs: 1200, // the check started 1.2 s into the video
|
|
1113
|
+
pauses: [{ atMs: 900, durationMs: 2000, reason: 'title card' }], // 2 s put into the video before the step at +900 ms
|
|
1114
|
+
marks: [{ step: 'pay', mediaMs: 4100 }] // a step whose moment in the video you know exactly
|
|
1115
|
+
}
|
|
1116
|
+
}
|
|
1117
|
+
}
|
|
1118
|
+
```
|
|
1119
|
+
|
|
1120
|
+
Every field is optional and none of it decides a status, a slot or a winner. A step is placed at `checkStartMs` + its
|
|
1121
|
+
`offsetMs` + the `durationMs` of every pause whose `atMs` is at or before the step; a mark names the moment outright and
|
|
1122
|
+
wins. Anything else is unavailable and said so, never guessed.
|
|
1123
|
+
|
|
1124
|
+
- **Without timing, nothing is placed.** A video with no `atlas.media`, no `checkStartMs`, or a step with no `offsetMs` and
|
|
1125
|
+
no mark, is never assumed to start with the check, and a step is never placed from the wording of its name. Selecting
|
|
1126
|
+
such a step says why it cannot be shown, and moves nothing.
|
|
1127
|
+
- **Outside the video is not clamped.** A moment before the start, or after `durationMs` (or the length the player reports),
|
|
1128
|
+
is unavailable. A declaration with a field Atlas cannot read (a pause without a duration) is reported and places nothing.
|
|
1129
|
+
- **Pacing is not latency.** A video with `pauses`, or with a `scope` of `pacing: 'paced'`, is labelled as paced: the time
|
|
1130
|
+
between its steps includes deliberate pauses and is not application latency, and no measurement is changed by it.
|
|
1131
|
+
- **Each try has its own clock.** Put the video and the `atlas.media` of a retry on its `attempts[]` entry, with that
|
|
1132
|
+
try's `steps`. A try is aligned only by its own video, never by the check's or an earlier try's.
|
|
1133
|
+
- **Link to a moment.** The viewer copies a link that names the run (`layer`), the exact recording (`clip`, its digest or
|
|
1134
|
+
stored key) and the position (`clipAt`, milliseconds into the video). Opening it selects that recording of that run, even
|
|
1135
|
+
when a newer run exists, and goes to the position; a recording the run does not hold is reported, never replaced.
|
|
1136
|
+
- **Older layers.** A layer without `atlas.media` reads exactly as before; the video plays and its steps list as ever.
|
|
1137
|
+
|
|
1099
1138
|
## Product latency
|
|
1100
1139
|
|
|
1101
1140
|
A test's `durationMs` includes fixture setup, deliberate waits, demo pacing, retries and cleanup, so it says how long the
|
|
@@ -1170,6 +1209,44 @@ curl "$ATLAS_SERVER_URL/api/reports/artifacts?artifactTag=demo&type=video&domain
|
|
|
1170
1209
|
**live query**: opening it asks again and shows what matches the newest report. It is not a snapshot; an immutable
|
|
1171
1210
|
review collection is a separate feature. *Copy link* gives the address, which holds the whole query.
|
|
1172
1211
|
|
|
1212
|
+
## Words in recordings
|
|
1213
|
+
|
|
1214
|
+
A reviewer who remembers a phrase that was spoken or shown in a video can search for it on the **Artifacts** tab
|
|
1215
|
+
(**Search inside recordings**, the `inside` address parameter). Atlas never transcribes video or reads a screen: it
|
|
1216
|
+
searches text you attach, and nothing is read from a recording until words are typed. Attach it beside the video, at the
|
|
1217
|
+
same level (the check, or the retry attempt that recorded the video):
|
|
1218
|
+
|
|
1219
|
+
```ts
|
|
1220
|
+
attachments: [
|
|
1221
|
+
{ id: 'video', name: 'checkout.webm', contentType: 'video/webm', data: video },
|
|
1222
|
+
// Spoken words: WebVTT, SubRip, JSON segments or plain text.
|
|
1223
|
+
{ name: 'checkout.vtt', role: 'transcript', contentType: 'text/vtt', data: transcript,
|
|
1224
|
+
extensions: { 'atlas.text': { recording: 'video', generated: true, generator: 'my-speech-tool' } } },
|
|
1225
|
+
// Text shown on screen, with its own timing.
|
|
1226
|
+
{ name: 'screen.json', role: 'ocr', contentType: 'application/json', data: JSON.stringify({ segments: [{ start: 2, end: 4, text: 'Order total', confidence: 0.9 }] }) }
|
|
1227
|
+
]
|
|
1228
|
+
```
|
|
1229
|
+
|
|
1230
|
+
- **Which attachment is text.** One with `extensions["atlas.text"]`, a `role` of `transcript`, `ocr`, `screen-text`,
|
|
1231
|
+
`captions` or `subtitles`, or a WebVTT or SubRip type. `extensions["atlas.text"].kind` (`speech`, `screen`, `caption`)
|
|
1232
|
+
says what it is; otherwise the role does. It belongs to the video that lists its `id` in `captions`, or that
|
|
1233
|
+
`extensions["atlas.text"].recording` names by `id`, else to the only video at the same level.
|
|
1234
|
+
- **Segments.** WebVTT and SubRip cues; JSON as `{ "segments": [...] }` (or `cues`, `results`, `lines`, or a bare list) of
|
|
1235
|
+
`{ text, start, end }` in seconds or `{ text, startMs, endMs }`, with an optional `confidence` between 0 and 1. A
|
|
1236
|
+
segment whose time is missing or invalid is still searched and is shown with no time; the viewer offers the recording
|
|
1237
|
+
and says seeking is unavailable, and never guesses a moment.
|
|
1238
|
+
- **Honesty about the text.** `generated: true` (with `generator`) marks text a tool produced; results show it with its
|
|
1239
|
+
confidence and say it may contain errors. A result always shows the indexed words, never a reworded quotation. A
|
|
1240
|
+
match finds a moment; it is not a new assertion, and the result's own outcome and the retry attempt stay beside it.
|
|
1241
|
+
- **States are different things.** Searched with no match; not indexed (no text was supplied); unsupported (the video is
|
|
1242
|
+
silent by its `scope.audioCaptured: false`, or the text is in a form that is not read, or sits at a `uri`); failed
|
|
1243
|
+
(`extensions["atlas.text"].state: "failed"` with a `reason`, a file left out with `omitted`, or text that did not
|
|
1244
|
+
parse). The ones not searched are listed apart and never counted as "no match".
|
|
1245
|
+
- **Access.** The text is read through the same report API, and so the same access rules, that play the recording. A file
|
|
1246
|
+
the viewer may not read is not indexed and appears in no snippet or count. The index exists only in the open page: there
|
|
1247
|
+
is no stored or cross-project index. Sending evidence to a transcription or OCR service is the producer's explicit
|
|
1248
|
+
decision and happens outside Atlas.
|
|
1249
|
+
|
|
1173
1250
|
## Labels describe, parameters split
|
|
1174
1251
|
|
|
1175
1252
|
A result's **slot** is its producer, check id, part kind, spec, environment and **parameters**. The newest completed
|
|
@@ -1724,6 +1801,37 @@ edit, and the server's time. Nothing about the layer, its results or the report'
|
|
|
1724
1801
|
See [Tagging artifacts and demos](rich-evidence.md#tagging-artifacts-and-demos) for the tag rules. Reviewer tags are
|
|
1725
1802
|
shown in the viewer's evidence; the tests API (`artifactTag`) searches the producers' tags.
|
|
1726
1803
|
|
|
1804
|
+
## Review collections
|
|
1805
|
+
|
|
1806
|
+
A team can publish a named, curated review of a few behaviors: recordings from several explicit executions, notes on
|
|
1807
|
+
what each is for, and the gaps nobody recorded yet. A **review collection** is kept beside the report under
|
|
1808
|
+
`curation/review-collections/` (never as a layer, a check or a run), so it cannot pass or fail anything, decides no
|
|
1809
|
+
slot or winner, and never claims its entries are one execution: every entry keeps the layer, run, commit, environment and
|
|
1810
|
+
status it was recorded with.
|
|
1811
|
+
|
|
1812
|
+
- **Freeze.** `POST /api/reports/review-collections` (publish) takes a title, optional `purpose`, `audience` and `tags`
|
|
1813
|
+
(such as `demo`), the ordered `entries` (`layer` and `check` from `GET /api/reports/artifacts`, optionally `kind`
|
|
1814
|
+
and one `recording`, and a `caption`) and optional `gaps`. The viewer's **Freeze as review collection** button on the
|
|
1815
|
+
Artifacts tab sends the filtered set as it is listed. The server resolves each entry to the result it holds and
|
|
1816
|
+
records what the report says about it; an entry it cannot resolve refuses the whole request (`422`, naming the entry).
|
|
1817
|
+
The order is the curator's: a failed, retried or incomplete result stays beside a later passing replay, each with its
|
|
1818
|
+
own status, time and tries.
|
|
1819
|
+
- **Immutable.** A revision is never changed or deleted by the server. Later runs and changed tags do not alter it. To
|
|
1820
|
+
change a collection, publish a new revision with `basedOn: { id, revision }` naming the newest (`409` otherwise); older
|
|
1821
|
+
revisions stay readable (`?revision=`).
|
|
1822
|
+
- **Notes and marks** (`POST .../review-collections/reviews`, publish) are separate immutable records with the token's name
|
|
1823
|
+
and the time: a `note`, a `gap` note (optionally at a `timecodeMs` of the entry's recording), `reviewed` and `unreviewed`.
|
|
1824
|
+
They are curation: they change no outcome, layer or winner, and a reviewed mark is not an acceptance sign-off.
|
|
1825
|
+
- **When a source goes away** (`GET .../review-collections/detail`) the entry stays where it was with its frozen snapshot
|
|
1826
|
+
and an `availability` of `unavailable` (its layer is gone, for example collected) or `unverified` (held, but the report
|
|
1827
|
+
no longer lists it), with the reason. Nothing is removed or replaced.
|
|
1828
|
+
- **Sharing** adds no access. The share URL is the viewer's `?tab=collections&collection=<id>`; reading needs the same
|
|
1829
|
+
read access as the report, and a visitor can Read, Watch and Inspect only what the attachment routes already allow them.
|
|
1830
|
+
Nothing is public by default.
|
|
1831
|
+
- **Export.** `GET .../review-collections/export` is a portable manifest: the exact sources, each file with its recorded
|
|
1832
|
+
name, a safe name and path, its `sha256` when the report holds one (`null` is unknown, not verified), and every missing
|
|
1833
|
+
source or file. It holds no bytes. `capabilities.reviewCollections` on `/api/server` says a server has all this.
|
|
1834
|
+
|
|
1727
1835
|
## Streaming
|
|
1728
1836
|
|
|
1729
1837
|
`GET /api/events` is a server-sent-events stream:
|
|
@@ -1864,6 +1972,16 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
|
|
|
1864
1972
|
| `GET /api/projects/{project}/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
|
|
1865
1973
|
| `POST /api/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
|
|
1866
1974
|
| `POST /api/projects/{project}/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
|
|
1975
|
+
| `GET /api/reports/review-collections` | read | Review collections: curated sets of recordings, newest first. |
|
|
1976
|
+
| `GET /api/projects/{project}/reports/review-collections` | read | Review collections: curated sets of recordings, newest first. |
|
|
1977
|
+
| `POST /api/reports/review-collections` | publish | Publish an immutable review collection (or a new revision of one). |
|
|
1978
|
+
| `POST /api/projects/{project}/reports/review-collections` | publish | Publish an immutable review collection (or a new revision of one). |
|
|
1979
|
+
| `GET /api/reports/review-collections/detail` | read | One review collection revision with how its sources stand now. |
|
|
1980
|
+
| `GET /api/projects/{project}/reports/review-collections/detail` | read | One review collection revision with how its sources stand now. |
|
|
1981
|
+
| `POST /api/reports/review-collections/reviews` | publish | Add a note, a time-coded gap note or a reviewed mark to a collection entry. |
|
|
1982
|
+
| `POST /api/projects/{project}/reports/review-collections/reviews` | publish | Add a note, a time-coded gap note or a reviewed mark to a collection entry. |
|
|
1983
|
+
| `GET /api/reports/review-collections/export` | read | Portable index (manifest) of one review collection revision. |
|
|
1984
|
+
| `GET /api/projects/{project}/reports/review-collections/export` | read | Portable index (manifest) of one review collection revision. |
|
|
1867
1985
|
| `GET /api/reports/runs` | read | The runs that recorded the report. |
|
|
1868
1986
|
| `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
|
|
1869
1987
|
| `GET /api/reports/artifacts` | read | Recorded results that carry evidence, across the whole history. |
|
|
@@ -2887,7 +3005,7 @@ carry any of these. Every one is optional.
|
|
|
2887
3005
|
| `annotations[]` | Notes about the result: a warning, a deprecation, an accessibility finding, a coverage figure. |
|
|
2888
3006
|
| `attempts[]` | Every try, when the check was retried. |
|
|
2889
3007
|
| `tags[]` | Free labels to filter by: `smoke`, `payments`. |
|
|
2890
|
-
| `labels{}` | Facts about the result that are not its identity: `owner`, `severity`, `browser`, `build`. |
|
|
3008
|
+
| `labels{}` | Facts about the result that are not its identity: `owner`, `severity`, `browser`, `build`. The viewer reads `build`, `platform`, `applies-to` and `applies-to-by` to tell whether evidence applies to a release (see [the viewer conventions](ui.md#behaviors-read-watch-inspect)). |
|
|
2891
3009
|
| `scope{}` | What the evidence does and does not show: surface, device kind, synthetic data, scripted dependencies, audio, pacing, what was asserted and its limits. Unknown when absent. See [Evidence scope](rich-evidence.md#evidence-scope). |
|
|
2892
3010
|
| `links[]` | Where to read more: the CI run, the issue, the trace client. |
|
|
2893
3011
|
| `extensions{}` | Structured data of your own or a tool's, under a name you own. |
|
|
@@ -2978,6 +3096,13 @@ when you do not know, and the viewer says the cause is unknown. It is never infe
|
|
|
2978
3096
|
failed before the behavior started comes from the failed step's `kind`: `hook`, `setup`, `fixture`, `before`,
|
|
2979
3097
|
`beforeAll`, `beforeEach` or `given` mean preparation; `when`, `then`, `assertion` or `expect` mean the behavior.
|
|
2980
3098
|
|
|
3099
|
+
### Placing steps in a recording
|
|
3100
|
+
|
|
3101
|
+
A video may declare how its clock relates to the check's under the `atlas.media` extension (`checkStartMs`, `pauses`,
|
|
3102
|
+
`marks`), so the viewer can seek it from a step and mark the step playing. Without it the video plays as ever and no step
|
|
3103
|
+
is placed. The fields, the rules and the link to a moment are in
|
|
3104
|
+
[Place steps in a recording](rich-evidence.md#place-steps-in-a-recording).
|
|
3105
|
+
|
|
2981
3106
|
## Attach evidence
|
|
2982
3107
|
|
|
2983
3108
|
### From the command line
|
|
@@ -3992,6 +4117,15 @@ when Watch is on screen.
|
|
|
3992
4117
|
- The promise's revision (the spec's fingerprint now) and the evidence's revision (the fingerprint the layer recorded)
|
|
3993
4118
|
are shown separately. A known difference demotes a recording to "Recording is older" with an alert; a missing
|
|
3994
4119
|
fingerprint is "not recorded", never a match.
|
|
4120
|
+
- Whether the evidence applies is a separate reading (`evidence/applicability.ts`), shown on Watch and Inspect and as a
|
|
4121
|
+
row badge when it is stale (or unknown against a named release): `applies`, `stale` (an earlier promise or another
|
|
4122
|
+
build), `unknown`, or `none`. It never changes a status or a coverage state. Age is when the checks ran
|
|
4123
|
+
(`recordedAtMs`), never when the layer was uploaded or annotated. A release is named in the address (`release` build,
|
|
4124
|
+
`releaseSha`, `releasePlatform`); a branch name or a passing run never supplies one. Another platform, a missing
|
|
4125
|
+
build or commit, or a checkout with local changes (`freshness`) is `unknown`, never an exact match. Producers say
|
|
4126
|
+
what they know with labels: `build`, `platform`, and `applies-to` with `applies-to-by` (the policy that asserts the
|
|
4127
|
+
evidence still holds for those builds; without it the assertion is refused). The "Stale or unverified" filter
|
|
4128
|
+
(`evidence=stale`) lists every behavior whose evidence does not apply, including those with none.
|
|
3995
4129
|
- A recording is always labelled illustrative: it shows one run and does not approve a behavior for production.
|
|
3996
4130
|
`demo`/`representative` tags only add a "Representative demo" label.
|
|
3997
4131
|
- The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
|
|
@@ -4051,6 +4185,64 @@ over the same filtered set the page lists, in the same order (`explorer/playlist
|
|
|
4051
4185
|
(paced, scripted, silent, synthetic) and the sentence that the length is the recording's, not production latency. Developer
|
|
4052
4186
|
details opens the test pinned to that run's record and closes the playlist.
|
|
4053
4187
|
|
|
4188
|
+
## Steps and moments in a recording
|
|
4189
|
+
|
|
4190
|
+
Wherever a result's steps sit beside its recording (a result's Timeline under Inspect, the Watch tab of a behavior, an
|
|
4191
|
+
opened watch playlist), `evidence/moments.tsx` joins them. Placement is from the recording's own declared timing
|
|
4192
|
+
(`evidence/media-timing.ts`, the `atlas.media` extension; see [Place steps in a recording](rich-evidence.md#place-steps-in-a-recording)),
|
|
4193
|
+
never from an assumption that the recording starts with the check.
|
|
4194
|
+
|
|
4195
|
+
- A step has a "Recording m:ss.s" button that seeks the selected recording; the step's test-clock offset stays visible beside it.
|
|
4196
|
+
A step that cannot be placed (no timing, no origin, no offset, outside the recording, unreadable declaration) is "Not aligned"
|
|
4197
|
+
and selecting it states the reason in a status alert. Nothing is guessed and the video is not moved.
|
|
4198
|
+
- While playing, the step under the playhead gets `aria-current="step"` and "Playing now". It never moves focus, scrolls
|
|
4199
|
+
the page or announces itself. Seeking only scrolls the player into view.
|
|
4200
|
+
- A paced recording (pauses declared, or a `pacing: paced` scope) says that gaps between steps include deliberate pauses and are
|
|
4201
|
+
not application latency.
|
|
4202
|
+
- A failed step has "Jump to the failing moment" in the bar above the timeline, and the failure summary has the same action
|
|
4203
|
+
(`requestFailureMoment`). If the recording is not on screen, or the step cannot be placed, it says so.
|
|
4204
|
+
- "Copy link to this moment" names the run (`layer`), the exact recording (`clip`) and the position (`clipAt`, ms), as the
|
|
4205
|
+
explorer address `/?test=…&layer=…&clip=…&clipAt=…` (the playlist links to the result it is playing). Opening it selects that
|
|
4206
|
+
recording of that exact run and seeks once the player knows its length; a position past the end or a recording that the run
|
|
4207
|
+
does not hold is reported and nothing else is played in its place. If the browser refuses to copy, the link is shown to copy.
|
|
4208
|
+
- One provider serves one holder: the check, or one try. A try's steps are placed by that try's recording and timing only.
|
|
4209
|
+
|
|
4210
|
+
## Review collections
|
|
4211
|
+
|
|
4212
|
+
The **Collections** tab (`explorer/collections.tsx`, `?tab=collections&collection=<id>&rev=<n>`) lists and opens review
|
|
4213
|
+
collections, and the Artifacts tab's **Freeze as review collection** button (a Mantine `Modal`) freezes the filtered set as it is
|
|
4214
|
+
listed. A collection is curation, not a test run (`@crvouga/atlas/reports/review-collection`, `docs/server.md#review-collections`).
|
|
4215
|
+
|
|
4216
|
+
- The page never shows one status for the whole collection. Every entry shows the status it was **recorded** with ("Failed as
|
|
4217
|
+
recorded"), its time, run, commit, layer and tries in order, so entries from several runs are never read as one execution; the
|
|
4218
|
+
note at the top says how many runs they come from. The order is the curator's: a failed or incomplete take stays beside a
|
|
4219
|
+
later passing replay and is never collapsed to the newest pass.
|
|
4220
|
+
- A revision is frozen. Opening an older revision says a newer one exists; a change is a new revision, never an edit.
|
|
4221
|
+
- An entry whose source is gone (`unavailable`) or cannot be checked (`unverified`) stays in its place with the reason and
|
|
4222
|
+
what it was when frozen, and its recording says it is not available. Nothing is removed or replaced.
|
|
4223
|
+
- Notes, time-coded gap notes (`m:ss`, or seconds; a mistake is said, not guessed) and the **Reviewed** switch are separate
|
|
4224
|
+
records with who and when. They change no outcome and the page says a mark is not an approval. Writing needs a token that
|
|
4225
|
+
may publish, held only in this tab; a read-only visitor is refused with the server's reason and can still Read, Watch and
|
|
4226
|
+
Inspect. **Download manifest** is the portable index; the share link adds no access.
|
|
4227
|
+
|
|
4228
|
+
## Search inside recordings
|
|
4229
|
+
|
|
4230
|
+
The Artifacts tab has a **Search inside recordings** section (`explorer/RecordingTextSearch.tsx`; words in `inside`). It
|
|
4231
|
+
applies the shared matcher (`find/`) to the transcript, caption and screen-text segments producers attached
|
|
4232
|
+
(`explorer/recording-text.ts`; format in [rich evidence](rich-evidence.md#words-in-recordings)), so phrases, close
|
|
4233
|
+
spellings, meaning and typed filters (`status:failed`) behave as in every search box.
|
|
4234
|
+
|
|
4235
|
+
- Nothing is read until words are typed; then each listed result's evidence and text files are read a few at a time
|
|
4236
|
+
through the report API (the same access rules as playing the recording). The video is never read. A file the viewer
|
|
4237
|
+
may not read is not indexed and is reported as not searched, never counted or quoted. There is no stored index.
|
|
4238
|
+
- A result shows the indexed words exactly as indexed, why it matched (exact, close spelling, meaning), the source (spoken,
|
|
4239
|
+
on screen, caption), generated text with its generator, confidence and a warning, the time range, the owning result and
|
|
4240
|
+
retry attempt, and the result's own outcome. A match is never described as the behavior passing.
|
|
4241
|
+
- "No match" is only said about searched recordings. Recordings with no text (`Not indexed`), silent or unsupported ones
|
|
4242
|
+
and unreadable text (`Failed`) are counted separately and listed under **Recordings that were not searched**.
|
|
4243
|
+
- A result opens the playlist on that result, that attempt's video and, when its time is usable and reachable, that time.
|
|
4244
|
+
With no usable time the recording opens at its start and a note says seeking is unavailable.
|
|
4245
|
+
|
|
4054
4246
|
## Icons and notation marks
|
|
4055
4247
|
|
|
4056
4248
|
Use named imports from `@tabler/icons-react` for all UI glyphs. The icon package supplies SVG paths;
|
|
@@ -5349,6 +5541,128 @@ Scoped to one project. The same project may be selected on the unprefixed API us
|
|
|
5349
5541
|
|
|
5350
5542
|
Responses: `201` The recorded entry.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` The token is known but its role may not do this.; `413` Request too large.; `422` Not a tag edit: a bad target, an empty or over-long tag, or nothing to change.
|
|
5351
5543
|
|
|
5544
|
+
### Review collections: curated sets of recordings, newest first
|
|
5545
|
+
|
|
5546
|
+
`GET /api/reports/review-collections` · role: read
|
|
5547
|
+
|
|
5548
|
+
The newest revision of every review collection, in brief: id, revision and how many exist, title, purpose, audience, tags, how many entries and gaps it has, who published it and when. A collection is curation kept beside the report; it is not a layer, a check or a run, and it has no status. Open one with `/api/reports/review-collections/detail?id=`. Its share URL is the viewer's `?tab=collections&collection=<id>` and needs the same read access as the rest of the report.
|
|
5549
|
+
|
|
5550
|
+
- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5551
|
+
|
|
5552
|
+
Responses: `200` The collections.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.
|
|
5553
|
+
|
|
5554
|
+
### Review collections: curated sets of recordings, newest first
|
|
5555
|
+
|
|
5556
|
+
`GET /api/projects/{project}/reports/review-collections` · role: read
|
|
5557
|
+
|
|
5558
|
+
The newest revision of every review collection, in brief: id, revision and how many exist, title, purpose, audience, tags, how many entries and gaps it has, who published it and when. A collection is curation kept beside the report; it is not a layer, a check or a run, and it has no status. Open one with `/api/reports/review-collections/detail?id=`. Its share URL is the viewer's `?tab=collections&collection=<id>` and needs the same read access as the rest of the report.
|
|
5559
|
+
|
|
5560
|
+
Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
|
|
5561
|
+
|
|
5562
|
+
- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5563
|
+
|
|
5564
|
+
Responses: `200` The collections.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.
|
|
5565
|
+
|
|
5566
|
+
### Publish an immutable review collection (or a new revision of one)
|
|
5567
|
+
|
|
5568
|
+
`POST /api/reports/review-collections` · role: publish
|
|
5569
|
+
|
|
5570
|
+
Freezes an ordered set of references to recorded results as one named collection, recorded under the name of the publish (or admin) token that sends it. Take each entry's `layer` and `check` (and `kind`) from `GET /api/reports/artifacts`, and optionally one of its recordings (`videos[].recording`). The server resolves every entry to the result the report holds and records what the report says about it (status, time, run, commit, environment, tries, videos): the request does not claim them. An entry it cannot resolve refuses the whole request (`422`, naming the entry). The order is yours and is never re-sorted; failed, rejected and incomplete results stay in it beside a later passing replay.
|
|
5571
|
+
|
|
5572
|
+
Entries from different runs are never one execution: each keeps its own layer, run and status. A collection has no status of its own, adds no check and cannot change any result or winner. A revision is immutable. To change a collection, publish a new revision with `basedOn: { id, revision }` naming the newest revision (`409` otherwise); the earlier ones stay readable. Notes and reviewed marks are separate (`/reviews`). A reader-only token, a browser sign-in box and an anonymous reader are refused; nothing is shared publicly by default.
|
|
5573
|
+
|
|
5574
|
+
- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5575
|
+
|
|
5576
|
+
Responses: `201` The recorded revision.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` The token is known but its role may not do this.; `404` `basedOn` names no collection.; `409` `basedOn.revision` is not the newest.; `413` Request too large.; `422` Not a collection, or an entry the report does not hold.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
|
|
5577
|
+
|
|
5578
|
+
### Publish an immutable review collection (or a new revision of one)
|
|
5579
|
+
|
|
5580
|
+
`POST /api/projects/{project}/reports/review-collections` · role: publish
|
|
5581
|
+
|
|
5582
|
+
Freezes an ordered set of references to recorded results as one named collection, recorded under the name of the publish (or admin) token that sends it. Take each entry's `layer` and `check` (and `kind`) from `GET /api/reports/artifacts`, and optionally one of its recordings (`videos[].recording`). The server resolves every entry to the result the report holds and records what the report says about it (status, time, run, commit, environment, tries, videos): the request does not claim them. An entry it cannot resolve refuses the whole request (`422`, naming the entry). The order is yours and is never re-sorted; failed, rejected and incomplete results stay in it beside a later passing replay.
|
|
5583
|
+
|
|
5584
|
+
Entries from different runs are never one execution: each keeps its own layer, run and status. A collection has no status of its own, adds no check and cannot change any result or winner. A revision is immutable. To change a collection, publish a new revision with `basedOn: { id, revision }` naming the newest revision (`409` otherwise); the earlier ones stay readable. Notes and reviewed marks are separate (`/reviews`). A reader-only token, a browser sign-in box and an anonymous reader are refused; nothing is shared publicly by default.
|
|
5585
|
+
|
|
5586
|
+
Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
|
|
5587
|
+
|
|
5588
|
+
- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5589
|
+
|
|
5590
|
+
Responses: `201` The recorded revision.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` The token is known but its role may not do this.; `404` `basedOn` names no collection.; `409` `basedOn.revision` is not the newest.; `413` Request too large.; `422` Not a collection, or an entry the report does not hold.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
|
|
5591
|
+
|
|
5592
|
+
### One review collection revision with how its sources stand now
|
|
5593
|
+
|
|
5594
|
+
`GET /api/reports/review-collections/detail` · role: read
|
|
5595
|
+
|
|
5596
|
+
The revision (the newest without `revision`), every revision number, the review notes and reviewed marks of that revision, and for each entry `availability`: `available` (its layer is held and the report lists the result), `unverified` (the layer is held but the report no longer lists the result or recording) or `unavailable` (the layer is gone, for example deleted by garbage collection), with the reason. An entry whose source is not available stays in its place with its frozen snapshot; it is never removed or replaced. Entries keep the status each layer recorded, with their own times, in the order the curator chose. A reviewed mark is curation: it changes no outcome and is not an approval.
|
|
5597
|
+
|
|
5598
|
+
- `id` (query, required): A collection id from the list.
|
|
5599
|
+
- `revision` (query): A revision number. Default: the newest.
|
|
5600
|
+
- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5601
|
+
|
|
5602
|
+
Responses: `200` The revision.; `400` No `id`, or a bad `revision`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No such collection or revision.
|
|
5603
|
+
|
|
5604
|
+
### One review collection revision with how its sources stand now
|
|
5605
|
+
|
|
5606
|
+
`GET /api/projects/{project}/reports/review-collections/detail` · role: read
|
|
5607
|
+
|
|
5608
|
+
The revision (the newest without `revision`), every revision number, the review notes and reviewed marks of that revision, and for each entry `availability`: `available` (its layer is held and the report lists the result), `unverified` (the layer is held but the report no longer lists the result or recording) or `unavailable` (the layer is gone, for example deleted by garbage collection), with the reason. An entry whose source is not available stays in its place with its frozen snapshot; it is never removed or replaced. Entries keep the status each layer recorded, with their own times, in the order the curator chose. A reviewed mark is curation: it changes no outcome and is not an approval.
|
|
5609
|
+
|
|
5610
|
+
Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
|
|
5611
|
+
|
|
5612
|
+
- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5613
|
+
- `id` (query, required): A collection id from the list.
|
|
5614
|
+
- `revision` (query): A revision number. Default: the newest.
|
|
5615
|
+
|
|
5616
|
+
Responses: `200` The revision.; `400` No `id`, or a bad `revision`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No such collection or revision.
|
|
5617
|
+
|
|
5618
|
+
### Add a note, a time-coded gap note or a reviewed mark to a collection entry
|
|
5619
|
+
|
|
5620
|
+
`POST /api/reports/review-collections/reviews` · role: publish
|
|
5621
|
+
|
|
5622
|
+
Records one immutable entry, by the name of the publish (or admin) token and the server's time: a `note` or a `gap` (text required; with `entry` and optionally `timecodeMs`, a moment in that entry's recording), or a `reviewed` / `unreviewed` mark for one entry. It changes no check outcome, no layer, no collection revision and no report winner; a mark is not an acceptance sign-off. The notes of a revision are in its `detail`.
|
|
5623
|
+
|
|
5624
|
+
- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5625
|
+
|
|
5626
|
+
Responses: `201` The recorded note.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` The token is known but its role may not do this.; `404` No such collection or revision.; `413` Request too large.; `422` Not a note: missing text, an unknown entry, or a mark without an entry.
|
|
5627
|
+
|
|
5628
|
+
### Add a note, a time-coded gap note or a reviewed mark to a collection entry
|
|
5629
|
+
|
|
5630
|
+
`POST /api/projects/{project}/reports/review-collections/reviews` · role: publish
|
|
5631
|
+
|
|
5632
|
+
Records one immutable entry, by the name of the publish (or admin) token and the server's time: a `note` or a `gap` (text required; with `entry` and optionally `timecodeMs`, a moment in that entry's recording), or a `reviewed` / `unreviewed` mark for one entry. It changes no check outcome, no layer, no collection revision and no report winner; a mark is not an acceptance sign-off. The notes of a revision are in its `detail`.
|
|
5633
|
+
|
|
5634
|
+
Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
|
|
5635
|
+
|
|
5636
|
+
- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5637
|
+
|
|
5638
|
+
Responses: `201` The recorded note.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` The token is known but its role may not do this.; `404` No such collection or revision.; `413` Request too large.; `422` Not a note: missing text, an unknown entry, or a mark without an entry.
|
|
5639
|
+
|
|
5640
|
+
### Portable index (manifest) of one review collection revision
|
|
5641
|
+
|
|
5642
|
+
`GET /api/reports/review-collections/export` · role: read
|
|
5643
|
+
|
|
5644
|
+
A manifest an archive can be built from: the collection, each entry with its exact source (layer, check, kind, recording), the status and run it was recorded with, how it stands now, whether it is marked reviewed, and the files it references with their name as recorded, a safe name and path (`files/<position>-<safeName>`), and the recorded `sha256` when the report holds one (`null` is unknown, not verified). A source that cannot be read now is listed under `missing` and its files are marked `missing`; nothing is left out silently. The manifest contains no bytes; recordings are fetched from the attachment routes, which keep their own access control. Sent as an attachment.
|
|
5645
|
+
|
|
5646
|
+
- `id` (query, required): A collection id.
|
|
5647
|
+
- `revision` (query): A revision number. Default: the newest.
|
|
5648
|
+
- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5649
|
+
|
|
5650
|
+
Responses: `200` The manifest.; `400` No `id`, or a bad `revision`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No such collection or revision.
|
|
5651
|
+
|
|
5652
|
+
### Portable index (manifest) of one review collection revision
|
|
5653
|
+
|
|
5654
|
+
`GET /api/projects/{project}/reports/review-collections/export` · role: read
|
|
5655
|
+
|
|
5656
|
+
A manifest an archive can be built from: the collection, each entry with its exact source (layer, check, kind, recording), the status and run it was recorded with, how it stands now, whether it is marked reviewed, and the files it references with their name as recorded, a safe name and path (`files/<position>-<safeName>`), and the recorded `sha256` when the report holds one (`null` is unknown, not verified). A source that cannot be read now is listed under `missing` and its files are marked `missing`; nothing is left out silently. The manifest contains no bytes; recordings are fetched from the attachment routes, which keep their own access control. Sent as an attachment.
|
|
5657
|
+
|
|
5658
|
+
Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
|
|
5659
|
+
|
|
5660
|
+
- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
|
|
5661
|
+
- `id` (query, required): A collection id.
|
|
5662
|
+
- `revision` (query): A revision number. Default: the newest.
|
|
5663
|
+
|
|
5664
|
+
Responses: `200` The manifest.; `400` No `id`, or a bad `revision`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No such collection or revision.
|
|
5665
|
+
|
|
5352
5666
|
### The runs that recorded the report
|
|
5353
5667
|
|
|
5354
5668
|
`GET /api/reports/runs` · role: read
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@crvouga/atlas",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.12.0",
|
|
4
4
|
"description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|