@crvouga/atlas 1.10.0 → 1.12.0

package/docs/rich-evidence.md

@@ -27,6 +27,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
 | `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
 | `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
 | `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
+| `extensions['atlas.media']` | attachment (video) | Where the check's steps are in the recording, so a click on a step seeks the video. See [Place steps in a recording](#place-steps-in-a-recording). |
 | `scope` | attachment, attempt, check, part, layer | What a recording does and does not show: where it ran, whether data and dependencies were real, whether audio was captured. See [Evidence scope](#evidence-scope). |
 | `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
 | `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
@@ -109,6 +110,44 @@ Every field is optional, and every value except the booleans is an open string.
 - **It decides nothing.** A scope never changes a status, a slot or a winner. A scope an older Atlas cannot read is
   dropped, and the attachment, check and layer stay.
 
+## Place steps in a recording
+
+A step says when it started on the check's clock (`offsetMs`, after the check started). A video has its own clock, which
+began when the recorder did, and the two are not assumed to agree: a recorder usually starts before the first step, and a
+paced demo may have delays put into the video that the check never waited for. To let a reviewer click a step and land on
+its moment in the video, say how the clocks relate in the `atlas.media` extension of the video attachment:
+
+```ts
+{
+  name: 'checkout.webm', role: 'video', durationMs: 9000,
+  extensions: {
+    'atlas.media': {
+      checkStartMs: 1200,                                           // the check started 1.2 s into the video
+      pauses: [{ atMs: 900, durationMs: 2000, reason: 'title card' }], // 2 s put into the video before the step at +900 ms
+      marks: [{ step: 'pay', mediaMs: 4100 }]                        // a step whose moment in the video you know exactly
+    }
+  }
+}
+```
+
+Every field is optional and none of it decides a status, a slot or a winner. A step is placed at `checkStartMs` + its
+`offsetMs` + the `durationMs` of every pause whose `atMs` is at or before the step; a mark names the moment outright and
+wins. Anything else is unavailable and said so, never guessed.
+
+- **Without timing, nothing is placed.** A video with no `atlas.media`, no `checkStartMs`, or a step with no `offsetMs` and
+  no mark, is never assumed to start with the check, and a step is never placed from the wording of its name. Selecting
+  such a step says why it cannot be shown, and moves nothing.
+- **Outside the video is not clamped.** A moment before the start, or after `durationMs` (or the length the player reports),
+  is unavailable. A declaration with a field Atlas cannot read (a pause without a duration) is reported and places nothing.
+- **Pacing is not latency.** A video with `pauses`, or with a `scope` of `pacing: 'paced'`, is labelled as paced: the time
+  between its steps includes deliberate pauses and is not application latency, and no measurement is changed by it.
+- **Each try has its own clock.** Put the video and the `atlas.media` of a retry on its `attempts[]` entry, with that
+  try's `steps`. A try is aligned only by its own video, never by the check's or an earlier try's.
+- **Link to a moment.** The viewer copies a link that names the run (`layer`), the exact recording (`clip`, its digest or
+  stored key) and the position (`clipAt`, milliseconds into the video). Opening it selects that recording of that run, even
+  when a newer run exists, and goes to the position; a recording the run does not hold is reported, never replaced.
+- **Older layers.** A layer without `atlas.media` reads exactly as before; the video plays and its steps list as ever.
+
 ## Product latency
 
 A test's `durationMs` includes fixture setup, deliberate waits, demo pacing, retries and cleanup, so it says how long the
@@ -183,6 +222,44 @@ curl "$ATLAS_SERVER_URL/api/reports/artifacts?artifactTag=demo&type=video&domain
   **live query**: opening it asks again and shows what matches the newest report. It is not a snapshot; an immutable
   review collection is a separate feature. *Copy link* gives the address, which holds the whole query.
 
+## Words in recordings
+
+A reviewer who remembers a phrase that was spoken or shown in a video can search for it on the **Artifacts** tab
+(**Search inside recordings**, the `inside` address parameter). Atlas never transcribes video or reads a screen: it
+searches text you attach, and nothing is read from a recording until words are typed. Attach it beside the video, at the
+same level (the check, or the retry attempt that recorded the video):
+
+```ts
+attachments: [
+  { id: 'video', name: 'checkout.webm', contentType: 'video/webm', data: video },
+  // Spoken words: WebVTT, SubRip, JSON segments or plain text.
+  { name: 'checkout.vtt', role: 'transcript', contentType: 'text/vtt', data: transcript,
+    extensions: { 'atlas.text': { recording: 'video', generated: true, generator: 'my-speech-tool' } } },
+  // Text shown on screen, with its own timing.
+  { name: 'screen.json', role: 'ocr', contentType: 'application/json', data: JSON.stringify({ segments: [{ start: 2, end: 4, text: 'Order total', confidence: 0.9 }] }) }
+]
+```
+
+- **Which attachment is text.** One with `extensions["atlas.text"]`, a `role` of `transcript`, `ocr`, `screen-text`,
+  `captions` or `subtitles`, or a WebVTT or SubRip type. `extensions["atlas.text"].kind` (`speech`, `screen`, `caption`)
+  says what it is; otherwise the role does. It belongs to the video that lists its `id` in `captions`, or that
+  `extensions["atlas.text"].recording` names by `id`, else to the only video at the same level.
+- **Segments.** WebVTT and SubRip cues; JSON as `{ "segments": [...] }` (or `cues`, `results`, `lines`, or a bare list) of
+  `{ text, start, end }` in seconds or `{ text, startMs, endMs }`, with an optional `confidence` between 0 and 1. A
+  segment whose time is missing or invalid is still searched and is shown with no time; the viewer offers the recording
+  and says seeking is unavailable, and never guesses a moment.
+- **Honesty about the text.** `generated: true` (with `generator`) marks text a tool produced; results show it with its
+  confidence and say it may contain errors. A result always shows the indexed words, never a reworded quotation. A
+  match finds a moment; it is not a new assertion, and the result's own outcome and the retry attempt stay beside it.
+- **States are different things.** Searched with no match; not indexed (no text was supplied); unsupported (the video is
+  silent by its `scope.audioCaptured: false`, or the text is in a form that is not read, or sits at a `uri`); failed
+  (`extensions["atlas.text"].state: "failed"` with a `reason`, a file left out with `omitted`, or text that did not
+  parse). The ones not searched are listed apart and never counted as "no match".
+- **Access.** The text is read through the same report API, and so the same access rules, that play the recording. A file
+  the viewer may not read is not indexed and appears in no snippet or count. The index exists only in the open page: there
+  is no stored or cross-project index. Sending evidence to a transcription or OCR service is the producer's explicit
+  decision and happens outside Atlas.
+
 ## Labels describe, parameters split
 
 A result's **slot** is its producer, check id, part kind, spec, environment and **parameters**. The newest completed

package/docs/server.md

@@ -335,6 +335,37 @@ edit, and the server's time. Nothing about the layer, its results or the report'
 See [Tagging artifacts and demos](rich-evidence.md#tagging-artifacts-and-demos) for the tag rules. Reviewer tags are
 shown in the viewer's evidence; the tests API (`artifactTag`) searches the producers' tags.
 
+## Review collections
+
+A team can publish a named, curated review of a few behaviors: recordings from several explicit executions, notes on
+what each is for, and the gaps nobody recorded yet. A **review collection** is kept beside the report under
+`curation/review-collections/` (never as a layer, a check or a run), so it cannot pass or fail anything, decides no
+slot or winner, and never claims its entries are one execution: every entry keeps the layer, run, commit, environment and
+status it was recorded with.
+
+- **Freeze.** `POST /api/reports/review-collections` (publish) takes a title, optional `purpose`, `audience` and `tags`
+  (such as `demo`), the ordered `entries` (`layer` and `check` from `GET /api/reports/artifacts`, optionally `kind`
+  and one `recording`, and a `caption`) and optional `gaps`. The viewer's **Freeze as review collection** button on the
+  Artifacts tab sends the filtered set as it is listed. The server resolves each entry to the result it holds and
+  records what the report says about it; an entry it cannot resolve refuses the whole request (`422`, naming the entry).
+  The order is the curator's: a failed, retried or incomplete result stays beside a later passing replay, each with its
+  own status, time and tries.
+- **Immutable.** A revision is never changed or deleted by the server. Later runs and changed tags do not alter it. To
+  change a collection, publish a new revision with `basedOn: { id, revision }` naming the newest (`409` otherwise); older
+  revisions stay readable (`?revision=`).
+- **Notes and marks** (`POST .../review-collections/reviews`, publish) are separate immutable records with the token's name
+  and the time: a `note`, a `gap` note (optionally at a `timecodeMs` of the entry's recording), `reviewed` and `unreviewed`.
+  They are curation: they change no outcome, layer or winner, and a reviewed mark is not an acceptance sign-off.
+- **When a source goes away** (`GET .../review-collections/detail`) the entry stays where it was with its frozen snapshot
+  and an `availability` of `unavailable` (its layer is gone, for example collected) or `unverified` (held, but the report
+  no longer lists it), with the reason. Nothing is removed or replaced.
+- **Sharing** adds no access. The share URL is the viewer's `?tab=collections&collection=<id>`; reading needs the same
+  read access as the report, and a visitor can Read, Watch and Inspect only what the attachment routes already allow them.
+  Nothing is public by default.
+- **Export.** `GET .../review-collections/export` is a portable manifest: the exact sources, each file with its recorded
+  name, a safe name and path, its `sha256` when the report holds one (`null` is unknown, not verified), and every missing
+  source or file. It holds no bytes. `capabilities.reviewCollections` on `/api/server` says a server has all this.
+
 ## Streaming
 
 `GET /api/events` is a server-sent-events stream:
@@ -475,6 +506,16 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
 | `POST /api/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
 | `POST /api/projects/{project}/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
+| `GET /api/reports/review-collections` | read | Review collections: curated sets of recordings, newest first. |
+| `GET /api/projects/{project}/reports/review-collections` | read | Review collections: curated sets of recordings, newest first. |
+| `POST /api/reports/review-collections` | publish | Publish an immutable review collection (or a new revision of one). |
+| `POST /api/projects/{project}/reports/review-collections` | publish | Publish an immutable review collection (or a new revision of one). |
+| `GET /api/reports/review-collections/detail` | read | One review collection revision with how its sources stand now. |
+| `GET /api/projects/{project}/reports/review-collections/detail` | read | One review collection revision with how its sources stand now. |
+| `POST /api/reports/review-collections/reviews` | publish | Add a note, a time-coded gap note or a reviewed mark to a collection entry. |
+| `POST /api/projects/{project}/reports/review-collections/reviews` | publish | Add a note, a time-coded gap note or a reviewed mark to a collection entry. |
+| `GET /api/reports/review-collections/export` | read | Portable index (manifest) of one review collection revision. |
+| `GET /api/projects/{project}/reports/review-collections/export` | read | Portable index (manifest) of one review collection revision. |
 | `GET /api/reports/runs` | read | The runs that recorded the report. |
 | `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
 | `GET /api/reports/artifacts` | read | Recorded results that carry evidence, across the whole history. |

package/docs/ui.md

@@ -82,6 +82,15 @@ when Watch is on screen.
 - The promise's revision (the spec's fingerprint now) and the evidence's revision (the fingerprint the layer recorded)
   are shown separately. A known difference demotes a recording to "Recording is older" with an alert; a missing
   fingerprint is "not recorded", never a match.
+- Whether the evidence applies is a separate reading (`evidence/applicability.ts`), shown on Watch and Inspect and as a
+  row badge when it is stale (or unknown against a named release): `applies`, `stale` (an earlier promise or another
+  build), `unknown`, or `none`. It never changes a status or a coverage state. Age is when the checks ran
+  (`recordedAtMs`), never when the layer was uploaded or annotated. A release is named in the address (`release` build,
+  `releaseSha`, `releasePlatform`); a branch name or a passing run never supplies one. Another platform, a missing
+  build or commit, or a checkout with local changes (`freshness`) is `unknown`, never an exact match. Producers say
+  what they know with labels: `build`, `platform`, and `applies-to` with `applies-to-by` (the policy that asserts the
+  evidence still holds for those builds; without it the assertion is refused). The "Stale or unverified" filter
+  (`evidence=stale`) lists every behavior whose evidence does not apply, including those with none.
 - A recording is always labelled illustrative: it shows one run and does not approve a behavior for production.
   `demo`/`representative` tags only add a "Representative demo" label.
 - The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
@@ -141,6 +150,64 @@ over the same filtered set the page lists, in the same order (`explorer/playlist
   (paced, scripted, silent, synthetic) and the sentence that the length is the recording's, not production latency. Developer
   details opens the test pinned to that run's record and closes the playlist.
 
+## Steps and moments in a recording
+
+Wherever a result's steps sit beside its recording (a result's Timeline under Inspect, the Watch tab of a behavior, an
+opened watch playlist), `evidence/moments.tsx` joins them. Placement is from the recording's own declared timing
+(`evidence/media-timing.ts`, the `atlas.media` extension; see [Place steps in a recording](rich-evidence.md#place-steps-in-a-recording)),
+never from an assumption that the recording starts with the check.
+
+- A step has a "Recording m:ss.s" button that seeks the selected recording; the step's test-clock offset stays visible beside it.
+  A step that cannot be placed (no timing, no origin, no offset, outside the recording, unreadable declaration) is "Not aligned"
+  and selecting it states the reason in a status alert. Nothing is guessed and the video is not moved.
+- While playing, the step under the playhead gets `aria-current="step"` and "Playing now". It never moves focus, scrolls
+  the page or announces itself. Seeking only scrolls the player into view.
+- A paced recording (pauses declared, or a `pacing: paced` scope) says that gaps between steps include deliberate pauses and are
+  not application latency.
+- A failed step has "Jump to the failing moment" in the bar above the timeline, and the failure summary has the same action
+  (`requestFailureMoment`). If the recording is not on screen, or the step cannot be placed, it says so.
+- "Copy link to this moment" names the run (`layer`), the exact recording (`clip`) and the position (`clipAt`, ms), as the
+  explorer address `/?test=…&layer=…&clip=…&clipAt=…` (the playlist links to the result it is playing). Opening it selects that
+  recording of that exact run and seeks once the player knows its length; a position past the end or a recording that the run
+  does not hold is reported and nothing else is played in its place. If the browser refuses to copy, the link is shown to copy.
+- One provider serves one holder: the check, or one try. A try's steps are placed by that try's recording and timing only.
+
+## Review collections
+
+The **Collections** tab (`explorer/collections.tsx`, `?tab=collections&collection=<id>&rev=<n>`) lists and opens review
+collections, and the Artifacts tab's **Freeze as review collection** button (a Mantine `Modal`) freezes the filtered set as it is
+listed. A collection is curation, not a test run (`@crvouga/atlas/reports/review-collection`, `docs/server.md#review-collections`).
+
+- The page never shows one status for the whole collection. Every entry shows the status it was **recorded** with ("Failed as
+  recorded"), its time, run, commit, layer and tries in order, so entries from several runs are never read as one execution; the
+  note at the top says how many runs they come from. The order is the curator's: a failed or incomplete take stays beside a
+  later passing replay and is never collapsed to the newest pass.
+- A revision is frozen. Opening an older revision says a newer one exists; a change is a new revision, never an edit.
+- An entry whose source is gone (`unavailable`) or cannot be checked (`unverified`) stays in its place with the reason and
+  what it was when frozen, and its recording says it is not available. Nothing is removed or replaced.
+- Notes, time-coded gap notes (`m:ss`, or seconds; a mistake is said, not guessed) and the **Reviewed** switch are separate
+  records with who and when. They change no outcome and the page says a mark is not an approval. Writing needs a token that
+  may publish, held only in this tab; a read-only visitor is refused with the server's reason and can still Read, Watch and
+  Inspect. **Download manifest** is the portable index; the share link adds no access.
+
+## Search inside recordings
+
+The Artifacts tab has a **Search inside recordings** section (`explorer/RecordingTextSearch.tsx`; words in `inside`). It
+applies the shared matcher (`find/`) to the transcript, caption and screen-text segments producers attached
+(`explorer/recording-text.ts`; format in [rich evidence](rich-evidence.md#words-in-recordings)), so phrases, close
+spellings, meaning and typed filters (`status:failed`) behave as in every search box.
+
+- Nothing is read until words are typed; then each listed result's evidence and text files are read a few at a time
+  through the report API (the same access rules as playing the recording). The video is never read. A file the viewer
+  may not read is not indexed and is reported as not searched, never counted or quoted. There is no stored index.
+- A result shows the indexed words exactly as indexed, why it matched (exact, close spelling, meaning), the source (spoken,
+  on screen, caption), generated text with its generator, confidence and a warning, the time range, the owning result and
+  retry attempt, and the result's own outcome. A match is never described as the behavior passing.
+- "No match" is only said about searched recordings. Recordings with no text (`Not indexed`), silent or unsupported ones
+  and unreadable text (`Failed`) are counted separately and listed under **Recordings that were not searched**.
+- A result opens the playlist on that result, that attempt's video and, when its time is usable and reachable, that time.
+  With no usable time the recording opens at its start and a note says seeking is unavailable.
+
 ## Icons and notation marks
 
 Use named imports from `@tabler/icons-react` for all UI glyphs. The icon package supplies SVG paths;