@crvouga/atlas 0.14.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/dist/reports/src/artifact-tag-store.d.ts +20 -0
- package/dist/reports/src/artifact-tag-store.js +70 -0
- package/dist/reports/src/artifact-tags.d.ts +96 -0
- package/dist/reports/src/artifact-tags.js +96 -0
- package/dist/reports/src/evidence-detail.d.ts +20 -3
- package/dist/reports/src/evidence-detail.js +66 -13
- package/dist/reports/src/evidence-types.d.ts +33 -0
- package/dist/reports/src/layer-merge.js +2 -2
- package/dist/reports/src/reports-explorer-route.d.ts +8 -0
- package/dist/reports/src/reports-route.d.ts +8 -0
- package/dist/reports/src/test-catalog.d.ts +35 -26
- package/dist/reports/src/test-catalog.js +79 -15
- package/dist/schema/src/api.d.ts +246 -0
- package/dist/schema/src/api.js +59 -5
- package/dist/schema/src/layer.d.ts +33 -0
- package/dist/schema/src/layer.js +11 -1
- package/dist/server/src/app.js +35 -3
- package/dist/server/src/llms.js +2 -1
- package/dist/server/src/openapi.d.ts +1 -1
- package/dist/server/src/openapi.js +20 -3
- package/docs/extending.md +3 -0
- package/docs/reference/http-api.md +52 -2
- package/docs/reference/reports.md +327 -5
- package/docs/reference/schema.md +48 -0
- package/docs/rich-evidence.md +28 -0
- package/docs/server.md +25 -1
- package/llms-full.txt +108 -3
- package/package.json +1 -1
- package/packages/schema/schemas/check-detail.schema.json +28 -0
- package/packages/schema/schemas/effective-delta.schema.json +96 -0
- package/packages/schema/schemas/effective-evidence.schema.json +96 -0
- package/packages/schema/schemas/evidence-page.schema.json +48 -0
- package/packages/schema/schemas/layer-rows.schema.json +48 -0
- package/packages/schema/schemas/layer.schema.json +70 -0
- package/packages/schema/schemas/publish-checks.schema.json +35 -0
- package/packages/schema/schemas/report-run.schema.json +71 -0
- package/packages/schema/schemas/report-runs.schema.json +38 -1
- package/packages/schema/schemas/test-catalog.schema.json +83 -0
- package/packages/schema/schemas/test-history.schema.json +110 -0
- package/packages/server/openapi.json +1259 -33
- package/packages/visualizer/src/evidence/artifact-tags.tsx +285 -0
- package/packages/visualizer/src/evidence/artifacts.tsx +18 -12
- package/packages/visualizer/src/explorer/RunsView.tsx +128 -10
- package/packages/visualizer/src/explorer/catalog.tsx +16 -6
- package/packages/visualizer/src/explorer/search.ts +4 -1
|
@@ -4,6 +4,209 @@ Layered evidence for any kind of spec: publishing layers, stores (folder, S3, an
|
|
|
4
4
|
|
|
5
5
|
> Generated from the doc comments of `packages/reports/src` by `bun run docs`. Do not edit: change the comment next to the code.
|
|
6
6
|
|
|
7
|
+
### Reviewer tags in a store
|
|
8
|
+
|
|
9
|
+
Source: `packages/reports/src/artifact-tag-store.ts`. Import: `@crvouga/atlas/reports/artifact-tag-store`
|
|
10
|
+
|
|
11
|
+
Where a reviewer's tags for attachments are kept: one small immutable object per edit under
|
|
12
|
+
`curation/artifact-tags/`, named by time and a random id so editors never overwrite one another, and never changed
|
|
13
|
+
or deleted afterwards. The layers they are about are not touched. The list of edits is the audit trail: who changed
|
|
14
|
+
which tags of which attachment, and when. `foldArtifactTags` turns it into the tags that stand now.
|
|
15
|
+
|
|
16
|
+
See docs/server.md#tagging-artifacts
|
|
17
|
+
|
|
18
|
+
#### `ARTIFACT_TAGS_PREFIX`
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
export const ARTIFACT_TAGS_PREFIX = …
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
The prefix of every recorded edit in a store.
|
|
25
|
+
|
|
26
|
+
#### `ArtifactTagError`
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
export class ArtifactTagError extends Error
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The edit was not acceptable (`status` is the HTTP status to answer).
|
|
33
|
+
|
|
34
|
+
#### `recordArtifactTags`
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
export async function recordArtifactTags(store: ObjectStore, input: unknown, by: string, atMs: number): Promise<ArtifactTagEntry>
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Validates an edit (`ArtifactTagEditSchema`), normalizes its tags and records it as a new object in `store`, by
|
|
41
|
+
`by` at `atMs`. Throws {@link ArtifactTagError} for a body that is not an edit, a tag that is not a tag (empty, or
|
|
42
|
+
over 64 characters) or an edit that changes nothing. Recording the same edit twice records two entries.
|
|
43
|
+
|
|
44
|
+
#### `readArtifactTagEntries`
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
export async function readArtifactTagEntries(store: ObjectStore): Promise<ArtifactTagEntry[]>
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Every recorded edit in `store`, oldest first. An object that cannot be read as an edit is skipped (unknown, never
|
|
51
|
+
guessed at). A store with none gives an empty list. Transport errors throw.
|
|
52
|
+
|
|
53
|
+
### Artifact tags
|
|
54
|
+
|
|
55
|
+
Source: `packages/reports/src/artifact-tags.ts`. Import: `@crvouga/atlas/reports/artifact-tags`
|
|
56
|
+
|
|
57
|
+
The tags of one attachment (a recording, a screenshot, a trace): how a tag is normalized so `demo` and `#demo`
|
|
58
|
+
are the same tag, which tags an attachment shows (its producer's, its reviewers', and those it inherits from its
|
|
59
|
+
check and run), and how a reviewer's edits fold into the tags that stand now. Pure functions, safe in a browser.
|
|
60
|
+
|
|
61
|
+
Tags describe evidence; they never decide a status, a slot or a winner. A producer writes `tags` on the attachment
|
|
62
|
+
inside an immutable layer. A reviewer's tags live apart, as an append-only list of edits that the server records
|
|
63
|
+
with who made each and when (`artifact-tag-store.ts`); the layer is never rewritten.
|
|
64
|
+
|
|
65
|
+
See docs/rich-evidence.md#tagging-artifacts-and-demos
|
|
66
|
+
|
|
67
|
+
#### `MAX_TAG_LENGTH`
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
export const MAX_TAG_LENGTH = …
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
The longest a normalized tag may be, in characters.
|
|
74
|
+
|
|
75
|
+
#### `normalizeTag`
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
export function normalizeTag(raw: string): string | null
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Normalizes a tag so equal tags compare equal: Unicode NFC, any leading `#` removed, surrounding whitespace
|
|
82
|
+
trimmed, letters lower-cased and each run of whitespace inside turned into one `-`. Every other character is kept,
|
|
83
|
+
so `Smoke`, `#smoke` and ` smoke ` are `smoke`, and `team/payments` or `release=2` stay as they are. Returns
|
|
84
|
+
`null` for a tag that is empty after that (`''`, `'#'`, `' '`) or longer than {@link MAX_TAG_LENGTH}.
|
|
85
|
+
|
|
86
|
+
**Example**
|
|
87
|
+
|
|
88
|
+
```ts
|
|
89
|
+
normalizeTag('#Demo'); // 'demo'
|
|
90
|
+
normalizeTag('release candidate'); // 'release-candidate'
|
|
91
|
+
normalizeTag('#'); // null
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
#### `normalizeTags`
|
|
95
|
+
|
|
96
|
+
```ts
|
|
97
|
+
export function normalizeTags(raw: Iterable<string> | undefined): string[]
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Normalizes every tag with {@link normalizeTag}, dropping the ones that are not tags and the repeats; the first of each keeps its place.
|
|
101
|
+
|
|
102
|
+
#### `displayTag`
|
|
103
|
+
|
|
104
|
+
```ts
|
|
105
|
+
export const displayTag = …
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
The tag as a person sees it: `#demo`.
|
|
109
|
+
|
|
110
|
+
#### `artifactRef`
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
export function artifactRef(attachment: Pick<Attachment, 'id' | 'bundleKey' | 'name'>): string
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
How an edit names an attachment: `id:<id>` when it has an `id`, else `key:<bundleKey>` (its bytes), else
|
|
117
|
+
`name:<name>`. It is stable for a layer, which never changes. Two attachments in one check with the same bytes and
|
|
118
|
+
no `id` share a reference, and so share reviewer tags; give them an `id` to tag them apart.
|
|
119
|
+
|
|
120
|
+
#### `ArtifactTarget`
|
|
121
|
+
|
|
122
|
+
```ts
|
|
123
|
+
export type ArtifactTarget = {
|
|
124
|
+
readonly layer: string;
|
|
125
|
+
readonly check: string;
|
|
126
|
+
readonly kind?: string | undefined;
|
|
127
|
+
readonly artifact: string;
|
|
128
|
+
};
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Where a reviewer's tags are about: one attachment of one check of one archived layer.
|
|
132
|
+
|
|
133
|
+
#### `ArtifactTagEntry`
|
|
134
|
+
|
|
135
|
+
```ts
|
|
136
|
+
export type ArtifactTagEntry = {
|
|
137
|
+
readonly id: string;
|
|
138
|
+
readonly target: ArtifactTarget;
|
|
139
|
+
readonly add: readonly string[];
|
|
140
|
+
readonly remove: readonly string[];
|
|
141
|
+
readonly by: string;
|
|
142
|
+
readonly atMs: number;
|
|
143
|
+
};
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
One recorded edit. Entries are never changed or removed.
|
|
147
|
+
|
|
148
|
+
#### `targetKey`
|
|
149
|
+
|
|
150
|
+
```ts
|
|
151
|
+
export const targetKey = …
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
A string that is equal for equal targets (the part `kind` is optional and then does not matter).
|
|
155
|
+
|
|
156
|
+
#### `ReviewerTag`
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
export type ReviewerTag = { readonly tag: string; readonly by: string; readonly atMs: number };
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
A tag a reviewer added: who added it last, and when.
|
|
163
|
+
|
|
164
|
+
#### `foldArtifactTags`
|
|
165
|
+
|
|
166
|
+
```ts
|
|
167
|
+
export function foldArtifactTags(entries: readonly ArtifactTagEntry[]): Map<string, ReviewerTag[]>
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Folds the recorded edits (in any order) into the reviewer tags that stand now, by target (see {@link targetKey}).
|
|
171
|
+
Edits apply oldest first, ties by `id`; a tag added, removed and added again stands, with the time and author of
|
|
172
|
+
the last add. A target whose tags were all removed is absent. Entries that are not understood are ignored.
|
|
173
|
+
|
|
174
|
+
#### `TagSource`
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
export type TagSource = 'artifact' | 'reviewer' | 'inherited';
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Where a shown tag comes from: the attachment's own producer (`artifact`), a reviewer after the upload (`reviewer`), or the check or run around it (`inherited`).
|
|
181
|
+
|
|
182
|
+
#### `ArtifactTag`
|
|
183
|
+
|
|
184
|
+
```ts
|
|
185
|
+
export type ArtifactTag = { readonly tag: string; readonly source: TagSource; readonly by?: string; readonly atMs?: number };
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
One tag an attachment shows, with its source and, for a reviewer's tag, who added it and when.
|
|
189
|
+
|
|
190
|
+
#### `artifactTagsOf`
|
|
191
|
+
|
|
192
|
+
```ts
|
|
193
|
+
export function artifactTagsOf(attachment: Pick<Attachment, 'tags'>, options: { readonly reviewer?: readonly ReviewerTag[]; readonly inherited?: Iterable<string> } = {}): ArtifactTag[]
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The tags one attachment shows, normalized and without repeats: its producer's own (`artifact`), then the
|
|
197
|
+
reviewers' (`reviewer`), then those of its check and run that it does not have itself (`inherited`). A tag it has
|
|
198
|
+
more than once is shown once, by the first source above.
|
|
199
|
+
|
|
200
|
+
#### `hasArtifactTag`
|
|
201
|
+
|
|
202
|
+
```ts
|
|
203
|
+
export function hasArtifactTag(tags: readonly ArtifactTag[], wanted: string, options: { readonly inherited?: boolean } = {}): boolean
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Does an attachment match a tag (either spelling)? An exact query (`inherited: false`, the default) matches only the
|
|
207
|
+
tags the attachment has itself, from its producer or a reviewer; an inherited query also matches the tags it
|
|
208
|
+
inherits from its check and run. The two differ for a recording that is untagged under a tagged run.
|
|
209
|
+
|
|
7
210
|
### Reading the files attachments name
|
|
8
211
|
|
|
9
212
|
Source: `packages/reports/src/attachment-files.ts`. Import: `@crvouga/atlas/reports/attachment-files`
|
|
@@ -1222,8 +1425,37 @@ read when someone asks for it. Undefined when there is none.
|
|
|
1222
1425
|
|
|
1223
1426
|
Several holders (for example a check and its layer) are summed: `attachments` also counts those on steps and
|
|
1224
1427
|
attempts, `steps` counts only each holder's own top-level steps, tags are de-duplicated, and for a repeated
|
|
1225
|
-
label the later holder wins. `roles` are the attachments' roles (or, without one, what their media type or file
|
|
1226
|
-
name implies), sorted and distinct.
|
|
1428
|
+
label the later holder wins. `artifactTags` are the attachments' own normalized tags and are never inherited. `roles` are the attachments' roles (or, without one, what their media type or file
|
|
1429
|
+
name implies), sorted and distinct. `videos` lists every attachment whose effective media type is video, with the
|
|
1430
|
+
check, retry attempt or step that holds it (see {@link isVideoAttachment}).
|
|
1431
|
+
|
|
1432
|
+
#### `summarizeLayerEvidence`
|
|
1433
|
+
|
|
1434
|
+
```ts
|
|
1435
|
+
export function summarizeLayerEvidence(...holders: readonly EvidenceHolder[]): EvidenceSummary | undefined
|
|
1436
|
+
```
|
|
1437
|
+
|
|
1438
|
+
Like {@link summarizeEvidence} for a layer and its parts: the same summary, but a video attached to the layer or a
|
|
1439
|
+
part is owned by the `layer`, not by a check.
|
|
1440
|
+
|
|
1441
|
+
#### `isVideoAttachment`
|
|
1442
|
+
|
|
1443
|
+
```ts
|
|
1444
|
+
export function isVideoAttachment(attachment: Pick<Attachment, 'name' | 'role' | 'contentType'>): boolean
|
|
1445
|
+
```
|
|
1446
|
+
|
|
1447
|
+
Whether an attachment is a video, by its effective media type: the producer's `contentType`, else the one its file
|
|
1448
|
+
name implies. A role never decides this on its own (`demo` on a PNG is not a video), except that a `video` role
|
|
1449
|
+
settles a file whose type nothing says (`application/octet-stream`).
|
|
1450
|
+
|
|
1451
|
+
#### `videoCounts`
|
|
1452
|
+
|
|
1453
|
+
```ts
|
|
1454
|
+
export function videoCounts(videos: readonly VideoRef[] | undefined): VideoCounts | undefined
|
|
1455
|
+
```
|
|
1456
|
+
|
|
1457
|
+
Counts references, distinct playable recordings and unplayable references. Undefined when there is no reference, so
|
|
1458
|
+
"no video" and "nothing known about video" look the same on the wire.
|
|
1227
1459
|
|
|
1228
1460
|
#### `roleOfType`
|
|
1229
1461
|
|
|
@@ -1406,6 +1638,51 @@ export type StatusCounts = Readonly<Record<EvidenceStatus, number>>;
|
|
|
1406
1638
|
|
|
1407
1639
|
How many results have each status (`passed`, `failed`, `running`, `not-run`).
|
|
1408
1640
|
|
|
1641
|
+
#### `VideoOwner`
|
|
1642
|
+
|
|
1643
|
+
```ts
|
|
1644
|
+
export type VideoOwner = 'layer' | 'check' | 'attempt' | 'step';
|
|
1645
|
+
```
|
|
1646
|
+
|
|
1647
|
+
Who holds a recording: the whole layer (or one of its parts), one check, one retry attempt of a check, or one step.
|
|
1648
|
+
|
|
1649
|
+
#### `VideoRef`
|
|
1650
|
+
|
|
1651
|
+
```ts
|
|
1652
|
+
export type VideoRef = {
|
|
1653
|
+
/** The attachment's name, for a person. */
|
|
1654
|
+
readonly name: string;
|
|
1655
|
+
readonly owner: VideoOwner;
|
|
1656
|
+
/** The try that holds it, when `owner` is `attempt` or the step is inside one. */
|
|
1657
|
+
readonly attempt?: number;
|
|
1658
|
+
/** The name of the step that holds it, when `owner` is `step`. */
|
|
1659
|
+
readonly step?: string;
|
|
1660
|
+
/** What identifies the bytes (digest, bundle key or address); references with the same one are the same recording. Absent when `available` is false. */
|
|
1661
|
+
readonly recording?: string;
|
|
1662
|
+
/** False when the reference names no bytes at all (no bundle key, address or inline text), so it can never be played. */
|
|
1663
|
+
readonly available: boolean;
|
|
1664
|
+
};
|
|
1665
|
+
```
|
|
1666
|
+
|
|
1667
|
+
One reference to a video, found by its effective media type (the producer's `contentType`, else the one the file
|
|
1668
|
+
name implies; a `video` role alone counts only when nothing else says what the file is). A reference is not a
|
|
1669
|
+
recording: the same bytes referenced from three checks are three references and one recording.
|
|
1670
|
+
|
|
1671
|
+
#### `VideoCounts`
|
|
1672
|
+
|
|
1673
|
+
```ts
|
|
1674
|
+
export type VideoCounts = {
|
|
1675
|
+
/** Every reference to a video, playable or not. */
|
|
1676
|
+
readonly references: number;
|
|
1677
|
+
/** Distinct playable recordings: identical bytes referenced twice are one. */
|
|
1678
|
+
readonly recordings: number;
|
|
1679
|
+
/** References that name no bytes. A report that predates availability metadata has none of these and none of the others. */
|
|
1680
|
+
readonly unavailable: number;
|
|
1681
|
+
};
|
|
1682
|
+
```
|
|
1683
|
+
|
|
1684
|
+
Video availability as counts: references, distinct playable recordings (deduplicated), and references that cannot be played.
|
|
1685
|
+
|
|
1409
1686
|
#### `EvidenceSummary`
|
|
1410
1687
|
|
|
1411
1688
|
```ts
|
|
@@ -1418,9 +1695,13 @@ export type EvidenceSummary = {
|
|
|
1418
1695
|
readonly attempts: number;
|
|
1419
1696
|
readonly links: number;
|
|
1420
1697
|
readonly tags: readonly string[];
|
|
1698
|
+
/** The normalized tags of the attachments themselves (`#demo` is `demo`), sorted and distinct. Absent when no attachment is tagged. Unlike `tags` they are never inherited from a check or run. */
|
|
1699
|
+
readonly artifactTags?: readonly string[];
|
|
1421
1700
|
readonly labels: Readonly<Record<string, string>>;
|
|
1422
1701
|
/** The distinct roles of its attachments (or what their media types say), to show what kind of evidence there is. */
|
|
1423
1702
|
readonly roles: readonly string[];
|
|
1703
|
+
/** Its references to videos, wherever they are held (layer, check, attempt, step). Only there when it has any. */
|
|
1704
|
+
readonly videos?: readonly VideoRef[];
|
|
1424
1705
|
};
|
|
1425
1706
|
```
|
|
1426
1707
|
|
|
@@ -7514,6 +7795,10 @@ export type TestSummary = {
|
|
|
7514
7795
|
readonly outcomes: readonly TestOutcome[];
|
|
7515
7796
|
readonly stats: TestStats;
|
|
7516
7797
|
readonly tags: readonly string[];
|
|
7798
|
+
/** The normalized tags of the attachments of its current results (`demo`, shown as `#demo`). Never inherited from a check or run; the `tags` above are. Absent in documents of older servers. */
|
|
7799
|
+
readonly artifactTags?: readonly string[];
|
|
7800
|
+
/** The normalized tags its attachments inherit from their check and run (those of the results that have attachments), apart from `artifactTags`. What an inherited artifact query also matches. */
|
|
7801
|
+
readonly inheritedArtifactTags?: readonly string[];
|
|
7517
7802
|
readonly labels: Readonly<Record<string, string>>;
|
|
7518
7803
|
/** What its current results carry: attachments, steps, retries, and the kinds of attachment. */
|
|
7519
7804
|
readonly evidence: { readonly attachments: number; readonly steps: number; readonly attempts: number; readonly roles: readonly string[] };
|
|
@@ -7604,6 +7889,8 @@ export type CatalogFacets = {
|
|
|
7604
7889
|
readonly domains: readonly FacetValue[];
|
|
7605
7890
|
readonly specs: readonly (CatalogSpec & { readonly tests: number; readonly counts: StatusCounts })[];
|
|
7606
7891
|
readonly tags: readonly FacetValue[];
|
|
7892
|
+
/** The tags of attachments, normalized. Absent in documents of older servers. */
|
|
7893
|
+
readonly artifactTags?: readonly FacetValue[];
|
|
7607
7894
|
readonly branches: readonly FacetValue[];
|
|
7608
7895
|
readonly labels: Readonly<Record<string, readonly FacetValue[]>>;
|
|
7609
7896
|
};
|
|
@@ -7641,6 +7928,10 @@ export type TestQuery = {
|
|
|
7641
7928
|
/** Spec paths (`specs/...`). */
|
|
7642
7929
|
readonly spec?: readonly string[];
|
|
7643
7930
|
readonly tag?: readonly string[];
|
|
7931
|
+
/** Only tests with an attachment carrying this tag (`demo` and `#demo` are the same), not counting tags inherited from a check or run: the exact artifact query. `tag` is the inherited one. */
|
|
7932
|
+
readonly artifactTag?: readonly string[];
|
|
7933
|
+
/** With `artifactTag`: also match the tags an attachment inherits from its check and run (a recording without a tag of its own under a run tagged `demo`). Default false: the exact query. */
|
|
7934
|
+
readonly inherited?: boolean;
|
|
7644
7935
|
/** Only tests recorded in this run (a key from `runList`). */
|
|
7645
7936
|
readonly run?: string;
|
|
7646
7937
|
readonly sort?: TestSort;
|
|
@@ -7715,20 +8006,39 @@ export type RunSummary = {
|
|
|
7715
8006
|
readonly totalDurationMs: number | null;
|
|
7716
8007
|
/** Up to twenty tests that failed in it. */
|
|
7717
8008
|
readonly failing: readonly { readonly key: string; readonly title: string; readonly environment: string | null; readonly detail: string }[];
|
|
8009
|
+
/**
|
|
8010
|
+
* Its videos, from the run's layers and from the last result of each test: references, distinct playable recordings
|
|
8011
|
+
* and references that cannot be played. Absent when it has none, and in a report that carries no video information.
|
|
8012
|
+
*/
|
|
8013
|
+
readonly videos?: VideoCounts;
|
|
7718
8014
|
};
|
|
7719
8015
|
```
|
|
7720
8016
|
|
|
7721
8017
|
A run: the layers that name the same execution (`run.id`), else share a CI run URL, else a branch and commit. When,
|
|
7722
8018
|
where, and how its tests went.
|
|
7723
8019
|
|
|
8020
|
+
#### `RunQuery`
|
|
8021
|
+
|
|
8022
|
+
```ts
|
|
8023
|
+
export type RunQuery = {
|
|
8024
|
+
/** Only runs that have at least one playable video recording. `total` then counts only those. */
|
|
8025
|
+
readonly video?: boolean;
|
|
8026
|
+
};
|
|
8027
|
+
```
|
|
8028
|
+
|
|
8029
|
+
What `runList` can be asked.
|
|
8030
|
+
|
|
7724
8031
|
#### `RunList`
|
|
7725
8032
|
|
|
7726
8033
|
```ts
|
|
7727
8034
|
export type RunList = {
|
|
7728
8035
|
readonly schemaVersion: typeof TEST_CATALOG_VERSION;
|
|
7729
8036
|
readonly version: string;
|
|
8037
|
+
/** Runs that matched, before `limit`. */
|
|
7730
8038
|
readonly total: number;
|
|
7731
8039
|
readonly runs: readonly RunSummary[];
|
|
8040
|
+
/** How many of all runs (whatever was asked, before `limit`) have a playable video recording. */
|
|
8041
|
+
readonly withVideo?: number;
|
|
7732
8042
|
};
|
|
7733
8043
|
```
|
|
7734
8044
|
|
|
@@ -7749,6 +8059,8 @@ export type RunResult = {
|
|
|
7749
8059
|
readonly at: number;
|
|
7750
8060
|
readonly layer: string;
|
|
7751
8061
|
readonly detail: string;
|
|
8062
|
+
/** Every video its evidence references, with who holds it (check, retry attempt or step). Absent when there is none. */
|
|
8063
|
+
readonly videos?: readonly VideoRef[];
|
|
7752
8064
|
};
|
|
7753
8065
|
```
|
|
7754
8066
|
|
|
@@ -7899,13 +8211,23 @@ export function testHistory(model: CatalogModel, key: string): TestHistoryDocume
|
|
|
7899
8211
|
|
|
7900
8212
|
One test and its recorded results, or null when the report has no test with that key.
|
|
7901
8213
|
|
|
8214
|
+
#### `hasVideo`
|
|
8215
|
+
|
|
8216
|
+
```ts
|
|
8217
|
+
export function hasVideo(run: Pick<RunSummary, 'videos'>): boolean
|
|
8218
|
+
```
|
|
8219
|
+
|
|
8220
|
+
Whether a run has at least one playable video recording.
|
|
8221
|
+
|
|
7902
8222
|
#### `runList`
|
|
7903
8223
|
|
|
7904
8224
|
```ts
|
|
7905
|
-
export function runList(history: TestHistory, version: string, limit = 100): RunList
|
|
8225
|
+
export function runList(history: TestHistory, version: string, limit = 100, query: RunQuery = {}): RunList
|
|
7906
8226
|
```
|
|
7907
8227
|
|
|
7908
|
-
Every run the history knows of, newest first, `limit` at most (default 100).
|
|
8228
|
+
Every run the history knows of, newest first, `limit` at most (default 100). With `query.video` only the runs that
|
|
8229
|
+
have a playable video recording: the filter and `total` apply to every run before `limit`, and nothing is read from
|
|
8230
|
+
any recording's bytes. `withVideo` always counts every run that has one.
|
|
7909
8231
|
|
|
7910
8232
|
#### `runDetail`
|
|
7911
8233
|
|
|
@@ -7921,7 +8243,7 @@ One run with every result it recorded, or null when no run has that key.
|
|
|
7921
8243
|
export function parseTestQuery(params: URLSearchParams): TestQuery
|
|
7922
8244
|
```
|
|
7923
8245
|
|
|
7924
|
-
Reads a `TestQuery` from URL search parameters: `q`, `sort`, `offset`, `limit`, `run`, and repeatable or comma-separated `status`, `environment`, `producer`, `domain`, `spec`, `tag`.
|
|
8246
|
+
Reads a `TestQuery` from URL search parameters: `q`, `sort`, `offset`, `limit`, `run`, and repeatable or comma-separated `status`, `environment`, `producer`, `domain`, `spec`, `tag`, `artifactTag`, and `inherited=1`.
|
|
7925
8247
|
|
|
7926
8248
|
#### `formatMs`
|
|
7927
8249
|
|
package/docs/reference/schema.md
CHANGED
|
@@ -69,6 +69,22 @@ export const BackendInfoSchema = …
|
|
|
69
69
|
|
|
70
70
|
A backend as the report shows it: where results came from, and whether it could be read.
|
|
71
71
|
|
|
72
|
+
#### `VideoRefSchema`
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
export const VideoRefSchema = …
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
One reference to a video held by a layer, check, retry attempt or step, found by the attachment's effective media type.
|
|
79
|
+
|
|
80
|
+
#### `VideoCountsSchema`
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
export const VideoCountsSchema = …
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Video references as counts: how many are referenced, how many distinct recordings can be played, how many cannot.
|
|
87
|
+
|
|
72
88
|
#### `EvidenceSummarySchema`
|
|
73
89
|
|
|
74
90
|
```ts
|
|
@@ -185,6 +201,38 @@ export const ChecksResultSchema = …
|
|
|
185
201
|
|
|
186
202
|
`POST /api/checks` answered: the layer that was written.
|
|
187
203
|
|
|
204
|
+
#### `ArtifactTargetSchema`
|
|
205
|
+
|
|
206
|
+
```ts
|
|
207
|
+
export const ArtifactTargetSchema = …
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Which attachment a reviewer's tags belong to: one attachment of one check of one archived layer.
|
|
211
|
+
|
|
212
|
+
#### `ArtifactTagEditSchema`
|
|
213
|
+
|
|
214
|
+
```ts
|
|
215
|
+
export const ArtifactTagEditSchema = …
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
The body of `POST /api/reports/artifact-tags`: tags to add to and to remove from one attachment. At least one tag overall.
|
|
219
|
+
|
|
220
|
+
#### `ArtifactTagEntrySchema`
|
|
221
|
+
|
|
222
|
+
```ts
|
|
223
|
+
export const ArtifactTagEntrySchema = …
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
One recorded edit of an attachment's reviewer tags. Entries are never changed or removed: they are the audit trail.
|
|
227
|
+
|
|
228
|
+
#### `ArtifactTagsSchema`
|
|
229
|
+
|
|
230
|
+
```ts
|
|
231
|
+
export const ArtifactTagsSchema = …
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
`GET /api/reports/artifact-tags`: every recorded edit, oldest first. Fold them with `foldArtifactTags`.
|
|
235
|
+
|
|
188
236
|
#### `HelloEventSchema`
|
|
189
237
|
|
|
190
238
|
```ts
|
package/docs/rich-evidence.md
CHANGED
|
@@ -26,6 +26,7 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
|
|
|
26
26
|
| `steps` (`parentId`, `offsetMs`, `durationMs`, `status`, `kind`) | check, attempt | Draws the check's timeline: nested steps, when each started, how long it took, which one failed. |
|
|
27
27
|
| `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
|
|
28
28
|
| `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
|
|
29
|
+
| `tags` | attachment | Marks one recording or screenshot, for example `demo`: shown as a `#demo` chip, and searchable. See [Tagging artifacts and demos](#tagging-artifacts-and-demos). |
|
|
29
30
|
| `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
|
|
30
31
|
| `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
|
|
31
32
|
| `specs[]` (`path`, `domain`, `kind`, `title`) | layer | Names each spec and its group (`domain`), so a server can group checks by spec with a title even when no [spec set](integrating.md#tell-atlas-which-spec-a-check-belongs-to) has been published. |
|
|
@@ -34,6 +35,33 @@ decided exactly as before ([the report contract](report-contract.md)). Evidence
|
|
|
34
35
|
Send `sha`, `branch` and `runUrl` on every layer too. They never change a result, but they tie results to a commit
|
|
35
36
|
and to the CI run that produced them.
|
|
36
37
|
|
|
38
|
+
## Tagging artifacts and demos
|
|
39
|
+
|
|
40
|
+
To mark one recording as a demo, tag the attachment itself. A tag on a check says something about the result; a tag on
|
|
41
|
+
an attachment says something about that file, and no Gherkin tag, label, environment or producer name is needed:
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
{ name: 'checkout.webm', role: 'video', tags: ['demo'], labels: { audience: 'business' } }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
The viewer shows it as a `#demo` chip on the artifact card. Selecting the chip filters the result's evidence to the
|
|
48
|
+
attachments with that tag, and says how many match. Tags describe evidence; they never change a status or a slot.
|
|
49
|
+
|
|
50
|
+
- **One spelling.** A tag is normalized before it is compared or stored: a leading `#` is dropped, letters are
|
|
51
|
+
lower-cased, each run of whitespace becomes `-`, and surrounding space is trimmed, so `demo`, `#demo` and `#Demo ` are
|
|
52
|
+
`demo`. Empty tags and tags over 64 characters are not tags and are ignored (or refused, when a reviewer enters them).
|
|
53
|
+
Everything else is a valid tag (`team/payments`, `release=2`), and repeats are shown once.
|
|
54
|
+
- **Inherited.** An attachment is also searchable by the tags of its check and its run. The viewer says `inherited` on
|
|
55
|
+
such a chip, and a switch decides whether the filter counts them: a recording without a tag of its own under a run
|
|
56
|
+
tagged `demo` is found by the inherited query and not by the exact one. In the API,
|
|
57
|
+
`GET /api/reports/tests?artifactTag=demo` is exact and adding `&inherited=1` includes inherited tags.
|
|
58
|
+
- **Reviewers.** After upload, a person with a publish token can add tags to an attachment (the viewer's *Add tag*
|
|
59
|
+
button) and remove the ones a reviewer added. Those tags are not written into the layer, which is immutable: the
|
|
60
|
+
server keeps them as an audit trail (who, when, which tags) and the viewer shows them as `added by <name>`, apart
|
|
61
|
+
from the producer's tags. See [Tagging artifacts](server.md#tagging-artifacts). A producer's own tags stay as uploaded.
|
|
62
|
+
- **Older layers.** A layer without attachment tags reads exactly as before. Tags on an attachment that cannot be read
|
|
63
|
+
(not a list of strings) are dropped, and the attachment, check and layer stay.
|
|
64
|
+
|
|
37
65
|
## Labels describe, parameters split
|
|
38
66
|
|
|
39
67
|
A result's **slot** is its producer, check id, part kind, spec, environment and **parameters**. The newest completed
|
package/docs/server.md
CHANGED
|
@@ -216,6 +216,26 @@ against a server unchanged. A client uploads **in dependency order**: blobs, the
|
|
|
216
216
|
entry that makes it visible. The server checks each step and says no to a catalog entry whose manifest is not
|
|
217
217
|
there (`409`), so nothing half-uploaded is ever shown.
|
|
218
218
|
|
|
219
|
+
## Tagging artifacts
|
|
220
|
+
|
|
221
|
+
A reviewer can mark a recording `#demo` after it was uploaded without uploading it again. The layer is immutable, so the
|
|
222
|
+
tag is kept beside it: each edit is one small object under `curation/artifact-tags/` in the write backend (or the live
|
|
223
|
+
store of a relay), named by time, never changed or removed. It records the attachment (layer, check, and the attachment's
|
|
224
|
+
`id`, else its bytes' key, else its name), the normalized tags added and removed, the name of the token that made the
|
|
225
|
+
edit, and the server's time. Nothing about the layer, its results or the report's winners changes.
|
|
226
|
+
|
|
227
|
+
- `GET /api/reports/artifact-tags` (read) returns every edit, oldest first. Fold them to get the tags that stand now: the
|
|
228
|
+
last edit of a tag wins.
|
|
229
|
+
- `POST /api/reports/artifact-tags` (publish, `Authorization: Bearer`) records an edit. A read token, an anonymous reader
|
|
230
|
+
and the browser's sign-in box are refused (`403`/`401`) like any upload, so a read-only visitor can see the tags and
|
|
231
|
+
not change them. The viewer asks for a token when a reviewer adds a tag and keeps it only in that tab.
|
|
232
|
+
- Producer tags (`tags` on the attachment) and reviewer tags are kept apart everywhere: the viewer says who added a
|
|
233
|
+
reviewer tag and when, and a reviewer can remove only reviewer tags. `capabilities.artifactTags` on `/api/server` says
|
|
234
|
+
a server has this.
|
|
235
|
+
|
|
236
|
+
See [Tagging artifacts and demos](rich-evidence.md#tagging-artifacts-and-demos) for the tag rules. Reviewer tags are
|
|
237
|
+
shown in the viewer's evidence; the tests API (`artifactTag`) searches the producers' tags.
|
|
238
|
+
|
|
219
239
|
## Streaming
|
|
220
240
|
|
|
221
241
|
`GET /api/events` is a server-sent-events stream:
|
|
@@ -256,7 +276,7 @@ with `format=markdown` (or `Accept: text/markdown`), and takes the project prefi
|
|
|
256
276
|
| Only some: `q` (all words), `status` (or `flaky`), `environment`, `producer`, `domain`, `spec`, `tag`, `run`; `sort`, `offset`, `limit` | `GET /api/reports/tests?status=failed&environment=ios` |
|
|
257
277
|
| One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
|
|
258
278
|
| Why Specs, domains and counts disagree: ambiguous spec paths, shared domains, gaps, a reconciliation of the totals | `GET /api/reports/tests/health` |
|
|
259
|
-
| The runs that recorded the report (by CI run URL, else branch and commit) | `GET /api/reports/runs` |
|
|
279
|
+
| The runs that recorded the report (by CI run URL, else branch and commit); `?video=true` keeps those with a playable video | `GET /api/reports/runs` |
|
|
260
280
|
| Every test one run recorded | `GET /api/reports/runs/detail?key=<run key>` |
|
|
261
281
|
|
|
262
282
|
```sh
|
|
@@ -330,6 +350,10 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
|
|
|
330
350
|
| `GET /api/projects/{project}/reports/tests/health` | read | Why specs, domains and counts disagree. |
|
|
331
351
|
| `GET /api/reports/tests/history` | read | One test and every result recorded for it. |
|
|
332
352
|
| `GET /api/projects/{project}/reports/tests/history` | read | One test and every result recorded for it. |
|
|
353
|
+
| `GET /api/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
|
|
354
|
+
| `GET /api/projects/{project}/reports/artifact-tags` | read | Reviewer tags of attachments: every recorded edit. |
|
|
355
|
+
| `POST /api/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
|
|
356
|
+
| `POST /api/projects/{project}/reports/artifact-tags` | publish | Tag or untag one attachment after upload. |
|
|
333
357
|
| `GET /api/reports/runs` | read | The runs that recorded the report. |
|
|
334
358
|
| `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
|
|
335
359
|
| `GET /api/reports/runs/detail` | read | Every test one run recorded. |
|