@crvouga/atlas 1.5.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -1
- package/docs/extending.md +17 -0
- package/docs/ui.md +25 -1
- package/llms-full.txt +42 -1
- package/package.json +1 -1
- package/packages/visualizer/README.md +28 -0
- package/packages/visualizer/src/evidence/behavior-panel.tsx +10 -16
- package/packages/visualizer/src/evidence/check-detail.tsx +50 -15
- package/packages/visualizer/src/evidence/outcome-summary.tsx +208 -0
- package/packages/visualizer/src/evidence/outcome.test.ts +129 -0
- package/packages/visualizer/src/evidence/outcome.ts +309 -0
- package/packages/visualizer/src/explorer/Explorer.tsx +105 -25
- package/packages/visualizer/src/explorer/RunsView.tsx +173 -35
- package/packages/visualizer/src/explorer/TestDetail.tsx +23 -1
- package/packages/visualizer/src/explorer/catalog.tsx +9 -5
- package/packages/visualizer/src/explorer/search-notes.tsx +74 -0
- package/packages/visualizer/src/explorer/search.ts +19 -6
- package/packages/visualizer/src/find/Finder.tsx +127 -81
- package/packages/visualizer/src/find/behaviors.ts +205 -0
- package/packages/visualizer/src/find/corpus.ts +21 -7
- package/packages/visualizer/src/find/doc-search.ts +66 -0
- package/packages/visualizer/src/find/query.ts +247 -0
- package/packages/visualizer/src/find/rank.ts +195 -47
- package/packages/visualizer/src/find/search.test.ts +322 -0
- package/packages/visualizer/src/find/semantic-search.ts +153 -0
- package/packages/visualizer/src/find/semantic.ts +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,18 @@
|
|
|
1
1
|
# Changelog: @crvouga/atlas
|
|
2
2
|
|
|
3
|
-
## 1.
|
|
3
|
+
## 1.7.0 (2026-10-29)
|
|
4
|
+
|
|
5
|
+
### Features
|
|
6
|
+
|
|
7
|
+
- explain selected, aggregate and historical outcomes without contradictory summaries ([018d2fe](https://github.com/crvouga/atlas/commit/018d2fe91a9c17cbefa8f9d24ad6f3d70550c5d1))
|
|
8
|
+
|
|
9
|
+
## 1.6.0 (2026-10-28)
|
|
10
|
+
|
|
11
|
+
### Features
|
|
12
|
+
|
|
13
|
+
- unify fuzzy semantic search with complete filterable sortable results ([3b0b90e](https://github.com/crvouga/atlas/commit/3b0b90e4960dcbcbcfee924f33aca4ce02beaaec))
|
|
14
|
+
|
|
15
|
+
## 1.5.0 (2026-10-28)
|
|
4
16
|
|
|
5
17
|
### Features
|
|
6
18
|
|
package/docs/extending.md
CHANGED
|
@@ -104,6 +104,23 @@ every kind of evidence and values from the future) and `layer-extensibility.test
|
|
|
104
104
|
fields survive at every level, malformed evidence is dropped without losing the result, and a reader of the
|
|
105
105
|
first version of the format still reads it all.
|
|
106
106
|
|
|
107
|
+
### Saying what failed
|
|
108
|
+
|
|
109
|
+
A failed check or step may declare the expectation it broke under the `atlas.failure` extension, so a reviewer sees
|
|
110
|
+
it before the raw log. Every field is optional, and the declaration never changes the result.
|
|
111
|
+
|
|
112
|
+
```json
|
|
113
|
+
{
|
|
114
|
+
"name": "The list includes tea", "kind": "then", "status": "failed",
|
|
115
|
+
"extensions": { "atlas.failure": { "expected": "tea in the list", "actual": "the list is empty", "attribution": "product" } }
|
|
116
|
+
}
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
`expected` and `actual` are shown as written. `attribution` is `product`, `test-harness` or `environment`; leave it out
|
|
120
|
+
when you do not know, and the viewer says the cause is unknown. It is never inferred from the message. Whether setup
|
|
121
|
+
failed before the behavior started comes from the failed step's `kind`: `hook`, `setup`, `fixture`, `before`,
|
|
122
|
+
`beforeAll`, `beforeEach` or `given` mean preparation; `when`, `then`, `assertion` or `expect` mean the behavior.
|
|
123
|
+
|
|
107
124
|
## Attach evidence
|
|
108
125
|
|
|
109
126
|
### From the command line
|
package/docs/ui.md
CHANGED
|
@@ -87,6 +87,20 @@ when Watch is on screen.
|
|
|
87
87
|
- The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
|
|
88
88
|
none are listed under "No domain declared"; the Tests view reports domain problems).
|
|
89
89
|
|
|
90
|
+
## Outcome summary
|
|
91
|
+
|
|
92
|
+
A test or behavior detail states what the **selected** result says apart from what the other current sources say
|
|
93
|
+
(`evidence/outcome.ts`, `outcome-summary.tsx`). The heading is the selected source's outcome first ("Selected result
|
|
94
|
+
passed; another scoped source failed"), never the combined status; the combined status is labelled as the worst current
|
|
95
|
+
result. Each other current source is one click away, and sources that share an environment are told apart by status,
|
|
96
|
+
time, revision and their place among them. A failure leads with whether the behavior was reached (decided only by the
|
|
97
|
+
recorded step `kind`: a `hook`, `setup`, `fixture`, `before*` or `given` step failing is "Behavior not verified: setup
|
|
98
|
+
failed", `when`, `then`, `assertion` or `expect` is "Expectation not met", anything else is unknown), then the expected
|
|
99
|
+
and observed values and the closest step and video, then the original output. The cause (product, test harness,
|
|
100
|
+
environment) is unknown unless the producer declared it under the `atlas.failure` extension; it is never read from error
|
|
101
|
+
wording. Records of one source whose outcome changed are listed oldest first, and a later pass is never called a fix.
|
|
102
|
+
In the explorer the chosen source is the `source` search parameter (a layer id; unlike `layer` it does not pin a run).
|
|
103
|
+
|
|
90
104
|
## Icons and notation marks
|
|
91
105
|
|
|
92
106
|
Use named imports from `@tabler/icons-react` for all UI glyphs. The icon package supplies SVG paths;
|
|
@@ -101,6 +115,16 @@ visible notation label; marks are decorative. Sources, licenses and the DMN nota
|
|
|
101
115
|
and update this mapping when adding a format. Brand artwork preserves upstream colors independently of
|
|
102
116
|
Atlas's theme. SVG data plots and statechart arrow markers are geometry, not handcrafted UI icons.
|
|
103
117
|
|
|
118
|
+
## Search
|
|
119
|
+
|
|
120
|
+
All search boxes (finder, test list, Runs tab, an opened run) use `src/find/query.ts` for the query language and
|
|
121
|
+
`src/find/rank.ts` for matching; do not add a second matcher or filter on `includes`. Build documents from
|
|
122
|
+
`src/find/behaviors.ts` so a behavior has the same fields and facets everywhere. Match first, then count, filter and sort
|
|
123
|
+
the whole set, then page. Show filters typed in the box as removable chips, show what was not understood
|
|
124
|
+
(`SearchNotes`), show why a result matched and mark exact ones, and say when matching is words only (no embedder, or
|
|
125
|
+
a failed one). A bounded list says it is bounded and offers all results. Selection follows result identity, never a
|
|
126
|
+
position, when results change. The query, filters and sort stay in the address.
|
|
127
|
+
|
|
104
128
|
## Accessibility and responsive behavior
|
|
105
129
|
|
|
106
130
|
The shell collapses its index below Mantine's `md` breakpoint; its named Burger toggles mobile navigation.
|
|
@@ -151,7 +175,7 @@ The narrowly scoped exceptions are:
|
|
|
151
175
|
| Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
|
|
152
176
|
| Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |
|
|
153
177
|
| Recordings | Pictures/video/timeline, actor selection, moments, failure jumps, timing, model-state differences | Tabs, Slider, Button, ScrollArea, Table, Progress |
|
|
154
|
-
| Search/help | Ranked and semantic search, shortcut help, recording shortcuts | Spotlight, Modal, Kbd |
|
|
178
|
+
| Search/help | Ranked, filterable and semantic search across behaviors, runs and results; complete results; shortcut help, recording shortcuts | Spotlight, TextInput, NativeSelect, Button chips, Alert, Modal, Kbd |
|
|
155
179
|
| Administration | Token access, S3 storage notice, garbage collection policy (saved in S3), live timeouts, report rules, limits, raw JSON, conflicts | Same provider/theme, Fieldset, Mantine inputs, Accordion, Alert |
|
|
156
180
|
|
|
157
181
|
The hash routes (`/`, `/chart/$chartId`, `/specs/$kind`), query contracts, project scoping, streaming,
|
package/llms-full.txt
CHANGED
|
@@ -2900,6 +2900,23 @@ every kind of evidence and values from the future) and `layer-extensibility.test
|
|
|
2900
2900
|
fields survive at every level, malformed evidence is dropped without losing the result, and a reader of the
|
|
2901
2901
|
first version of the format still reads it all.
|
|
2902
2902
|
|
|
2903
|
+
### Saying what failed
|
|
2904
|
+
|
|
2905
|
+
A failed check or step may declare the expectation it broke under the `atlas.failure` extension, so a reviewer sees
|
|
2906
|
+
it before the raw log. Every field is optional, and the declaration never changes the result.
|
|
2907
|
+
|
|
2908
|
+
```json
|
|
2909
|
+
{
|
|
2910
|
+
"name": "The list includes tea", "kind": "then", "status": "failed",
|
|
2911
|
+
"extensions": { "atlas.failure": { "expected": "tea in the list", "actual": "the list is empty", "attribution": "product" } }
|
|
2912
|
+
}
|
|
2913
|
+
```
|
|
2914
|
+
|
|
2915
|
+
`expected` and `actual` are shown as written. `attribution` is `product`, `test-harness` or `environment`; leave it out
|
|
2916
|
+
when you do not know, and the viewer says the cause is unknown. It is never inferred from the message. Whether setup
|
|
2917
|
+
failed before the behavior started comes from the failed step's `kind`: `hook`, `setup`, `fixture`, `before`,
|
|
2918
|
+
`beforeAll`, `beforeEach` or `given` mean preparation; `when`, `then`, `assertion` or `expect` mean the behavior.
|
|
2919
|
+
|
|
2903
2920
|
## Attach evidence
|
|
2904
2921
|
|
|
2905
2922
|
### From the command line
|
|
@@ -3919,6 +3936,20 @@ when Watch is on screen.
|
|
|
3919
3936
|
- The spec index searches titles, paths and declared domains, and can group by the domain from spec metadata (specs with
|
|
3920
3937
|
none are listed under "No domain declared"; the Tests view reports domain problems).
|
|
3921
3938
|
|
|
3939
|
+
## Outcome summary
|
|
3940
|
+
|
|
3941
|
+
A test or behavior detail states what the **selected** result says apart from what the other current sources say
|
|
3942
|
+
(`evidence/outcome.ts`, `outcome-summary.tsx`). The heading is the selected source's outcome first ("Selected result
|
|
3943
|
+
passed; another scoped source failed"), never the combined status; the combined status is labelled as the worst current
|
|
3944
|
+
result. Each other current source is one click away, and sources that share an environment are told apart by status,
|
|
3945
|
+
time, revision and their place among them. A failure leads with whether the behavior was reached (decided only by the
|
|
3946
|
+
recorded step `kind`: a `hook`, `setup`, `fixture`, `before*` or `given` step failing is "Behavior not verified: setup
|
|
3947
|
+
failed", `when`, `then`, `assertion` or `expect` is "Expectation not met", anything else is unknown), then the expected
|
|
3948
|
+
and observed values and the closest step and video, then the original output. The cause (product, test harness,
|
|
3949
|
+
environment) is unknown unless the producer declared it under the `atlas.failure` extension; it is never read from error
|
|
3950
|
+
wording. Records of one source whose outcome changed are listed oldest first, and a later pass is never called a fix.
|
|
3951
|
+
In the explorer the chosen source is the `source` search parameter (a layer id; unlike `layer` it does not pin a run).
|
|
3952
|
+
|
|
3922
3953
|
## Icons and notation marks
|
|
3923
3954
|
|
|
3924
3955
|
Use named imports from `@tabler/icons-react` for all UI glyphs. The icon package supplies SVG paths;
|
|
@@ -3933,6 +3964,16 @@ visible notation label; marks are decorative. Sources, licenses and the DMN nota
|
|
|
3933
3964
|
and update this mapping when adding a format. Brand artwork preserves upstream colors independently of
|
|
3934
3965
|
Atlas's theme. SVG data plots and statechart arrow markers are geometry, not handcrafted UI icons.
|
|
3935
3966
|
|
|
3967
|
+
## Search
|
|
3968
|
+
|
|
3969
|
+
All search boxes (finder, test list, Runs tab, an opened run) use `src/find/query.ts` for the query language and
|
|
3970
|
+
`src/find/rank.ts` for matching; do not add a second matcher or filter on `includes`. Build documents from
|
|
3971
|
+
`src/find/behaviors.ts` so a behavior has the same fields and facets everywhere. Match first, then count, filter and sort
|
|
3972
|
+
the whole set, then page. Show filters typed in the box as removable chips, show what was not understood
|
|
3973
|
+
(`SearchNotes`), show why a result matched and mark exact ones, and say when matching is words only (no embedder, or
|
|
3974
|
+
a failed one). A bounded list says it is bounded and offers all results. Selection follows result identity, never a
|
|
3975
|
+
position, when results change. The query, filters and sort stay in the address.
|
|
3976
|
+
|
|
3936
3977
|
## Accessibility and responsive behavior
|
|
3937
3978
|
|
|
3938
3979
|
The shell collapses its index below Mantine's `md` breakpoint; its named Burger toggles mobile navigation.
|
|
@@ -3983,7 +4024,7 @@ The narrowly scoped exceptions are:
|
|
|
3983
4024
|
| Statecharts | Graph, screen/event/journey details, developer metadata, chart test results | Mantine cards/navigation and product compositions; graph exception above |
|
|
3984
4025
|
| Evidence | Steps, retries, annotations, facts, unknown extensions, safe downloads and previews | List, Badge, Accordion, Code, Image, native media |
|
|
3985
4026
|
| Recordings | Pictures/video/timeline, actor selection, moments, failure jumps, timing, model-state differences | Tabs, Slider, Button, ScrollArea, Table, Progress |
|
|
3986
|
-
| Search/help | Ranked and semantic search, shortcut help, recording shortcuts | Spotlight, Modal, Kbd |
|
|
4027
|
+
| Search/help | Ranked, filterable and semantic search across behaviors, runs and results; complete results; shortcut help, recording shortcuts | Spotlight, TextInput, NativeSelect, Button chips, Alert, Modal, Kbd |
|
|
3987
4028
|
| Administration | Token access, S3 storage notice, garbage collection policy (saved in S3), live timeouts, report rules, limits, raw JSON, conflicts | Same provider/theme, Fieldset, Mantine inputs, Accordion, Alert |
|
|
3988
4029
|
|
|
3989
4030
|
The hash routes (`/`, `/chart/$chartId`, `/specs/$kind`), query contracts, project scoping, streaming,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@crvouga/atlas",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.7.0",
|
|
4
4
|
"description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -154,6 +154,34 @@ Components read only the view model (`AtlasView`), never raw files.
|
|
|
154
154
|
its own worker. Positions are cached under a hash of the chart's structure only, so new results
|
|
155
155
|
never move anything.
|
|
156
156
|
|
|
157
|
+
## Searching behaviors, runs and recordings
|
|
158
|
+
|
|
159
|
+
Every search box (the finder opened with `Ctrl+K` or `/`, the test list, the Runs tab and the results of an opened
|
|
160
|
+
run) reads one query language and matches with one engine (`src/find/query.ts`, `src/find/rank.ts`), so the same words
|
|
161
|
+
make the same behaviors eligible everywhere.
|
|
162
|
+
|
|
163
|
+
- **Words** match titles, steps, descriptions, specs, tags, environments and failure messages by stem, everyday synonym
|
|
164
|
+
and close spelling (`savde answer` finds "Keep a saved answer"), and the start of a longer word while it is typed.
|
|
165
|
+
Every distinctive word must be found; a word present in most behaviors is optional. When an embedder is available the
|
|
166
|
+
results are also re-ranked by meaning and a paraphrase with no word in common is added (`Similar meaning`); when none
|
|
167
|
+
is, or it fails, the box says it is matching words and spellings only.
|
|
168
|
+
- **Exact matches.** `"a quoted phrase"` and identifiers with `/`, `_` or `.` (`memory/keep`) must appear as written, are
|
|
169
|
+
marked `Exact` and are listed first.
|
|
170
|
+
- **Filters** narrow by structure instead of matching words: `status:failed` (`passed`, `failed`, `flaky`, `not-run`,
|
|
171
|
+
`running`), `env:ios` (also `platform:`, `environment:`, `device:`), `producer:`, `domain:`, `spec:`, `tag:`, `branch:`
|
|
172
|
+
and `has:video` (a playable recording, not the word). Values of one field are alternatives (`status:failed,flaky`);
|
|
173
|
+
different fields all hold. Each shows as a removable chip, and the status, environment and Has video buttons are the
|
|
174
|
+
same filters without the syntax.
|
|
175
|
+
- **Nothing is dropped silently.** An unknown operator, an unknown status, an unclosed quote, or a filter a list does not
|
|
176
|
+
have (`domain:` on runs) is listed under the box with what to type instead.
|
|
177
|
+
- **Complete results.** Counts, filters and sorting (including Best match, offered with words to match) apply to the whole
|
|
178
|
+
dataset before anything is shown in pages. The finder shows its first 40 results, says so, and opens all of them
|
|
179
|
+
(`Shift+Enter`, or the explorer's list for the same query). The query, filters and sort are in the address, so a link
|
|
180
|
+
and the back button restore them. Results that change under the cursor keep the highlighted result highlighted by
|
|
181
|
+
identity, and say so when it is no longer listed.
|
|
182
|
+
|
|
183
|
+
The standing rules are in [the UI conventions](../../docs/ui.md#search).
|
|
184
|
+
|
|
157
185
|
## Fixtures
|
|
158
186
|
|
|
159
187
|
`fixtures/` holds seven scenarios generated by `scripts/make-fixtures.ts` from `example/specs`, a
|
|
@@ -6,12 +6,9 @@
|
|
|
6
6
|
* record: run, environment, source, attempts). Also the coverage summary of the whole feature. A recording is always
|
|
7
7
|
* labelled as illustrative evidence with its own status, provenance and revision; it never stands for approval.
|
|
8
8
|
*/
|
|
9
|
-
import type { EvidenceHolder } from '@crvouga/atlas/reports/evidence-detail';
|
|
10
|
-
import { isVideoAttachment } from '@crvouga/atlas/reports/evidence-detail';
|
|
11
9
|
import { effectiveScope } from '@crvouga/atlas/reports/evidence-scope';
|
|
12
10
|
import type { EvidenceResult } from '@crvouga/atlas/reports/layer-merge';
|
|
13
11
|
import { environmentLabel } from '@crvouga/atlas/reports/layer-axes';
|
|
14
|
-
import type { Attachment } from '@crvouga/atlas/schema';
|
|
15
12
|
import { Alert, Badge, Box, Group, List, Paper, Stack, Tabs, Text, Title } from '@mantine/core';
|
|
16
13
|
import { useMemo, type ReactNode } from 'react';
|
|
17
14
|
|
|
@@ -23,6 +20,7 @@ import { ScopeNotice } from './evidence-scope';
|
|
|
23
20
|
import { plural, statusWord, useNow } from './format';
|
|
24
21
|
import { exampleIndexFor, exampleRows, fillExample, type FeatureStep } from './gherkin';
|
|
25
22
|
import { Origin, StatusMark } from './marks';
|
|
23
|
+
import { outcomeHeadline, videosIn } from './outcome';
|
|
26
24
|
|
|
27
25
|
/** The short, readable form of a spec fingerprint (`<length>:<hash>:<hash>`): its length and the start of its hash. */
|
|
28
26
|
export function shortRevision(fingerprint: string): string {
|
|
@@ -80,19 +78,6 @@ function expectedSteps(steps: readonly FeatureStep[]): FeatureStep[] {
|
|
|
80
78
|
return at < 0 ? [] : steps.slice(at);
|
|
81
79
|
}
|
|
82
80
|
|
|
83
|
-
function videosIn(holder: EvidenceHolder | undefined): Attachment[] {
|
|
84
|
-
if (!holder) return [];
|
|
85
|
-
const own = (holder.attachments ?? []).filter(isVideoAttachment);
|
|
86
|
-
return [
|
|
87
|
-
...own,
|
|
88
|
-
...(holder.steps ?? []).flatMap((step) => (step.attachments ?? []).filter(isVideoAttachment)),
|
|
89
|
-
...(holder.attempts ?? []).flatMap((attempt) => [
|
|
90
|
-
...(attempt.attachments ?? []).filter(isVideoAttachment),
|
|
91
|
-
...(attempt.steps ?? []).flatMap((step) => (step.attachments ?? []).filter(isVideoAttachment))
|
|
92
|
-
])
|
|
93
|
-
];
|
|
94
|
-
}
|
|
95
|
-
|
|
96
81
|
/** The recording of one result, read only when this is on screen. */
|
|
97
82
|
function Player({ result }: { result: EvidenceResult }) {
|
|
98
83
|
const detail = useCheckDetail(result.layer, result.row);
|
|
@@ -171,6 +156,13 @@ export function BehaviorPanel({
|
|
|
171
156
|
const steps = useMemo(() => (entry ? fillExample(scenario.steps, entry) : scenario.steps), [scenario, entry]);
|
|
172
157
|
const expected = expectedSteps(steps);
|
|
173
158
|
const named = example !== null && entry === null;
|
|
159
|
+
// The coverage state is the check's combined status; this says what the recording or result shown here recorded, apart from it.
|
|
160
|
+
const outcome =
|
|
161
|
+
subject && behavior.slot ? (
|
|
162
|
+
<Text component="div" fw={600} data-outcome-headline>
|
|
163
|
+
{outcomeHeadline({ layer: subject.layer, row: subject.row }, behavior.slot).text}
|
|
164
|
+
</Text>
|
|
165
|
+
) : null;
|
|
174
166
|
return (
|
|
175
167
|
<Tabs
|
|
176
168
|
defaultValue={start ?? (demo ? 'watch' : state === 'running' || behavior.slot?.live ? 'inspect' : 'read')}
|
|
@@ -190,6 +182,7 @@ export function BehaviorPanel({
|
|
|
190
182
|
{COVERAGE[state].meaning}
|
|
191
183
|
</Text>
|
|
192
184
|
</Group>
|
|
185
|
+
{outcome}
|
|
193
186
|
{example !== null && (
|
|
194
187
|
<Text component="div" c="dimmed">
|
|
195
188
|
Example: {example}
|
|
@@ -209,6 +202,7 @@ export function BehaviorPanel({
|
|
|
209
202
|
<CoverageBadge state={state} />
|
|
210
203
|
{behavior.representative && <Badge variant="outline">Representative demo</Badge>}
|
|
211
204
|
</Group>
|
|
205
|
+
{outcome}
|
|
212
206
|
{expected.length > 0 && (
|
|
213
207
|
<Box component="section" aria-label="Expected outcome">
|
|
214
208
|
<Title order={4} mb="xs">
|
|
@@ -9,12 +9,14 @@ import { useQuery } from '@tanstack/react-query';
|
|
|
9
9
|
import { useMemo, useState, type ReactNode } from 'react';
|
|
10
10
|
import { TestProgress, useTestProgress } from '../live/test-progress';
|
|
11
11
|
import { reportJson } from './api';
|
|
12
|
-
import { ResultEvidence } from './artifacts';
|
|
12
|
+
import { ResultEvidence, useCheckDetail } from './artifacts';
|
|
13
13
|
import { useEvidence } from './evidence-context';
|
|
14
14
|
import { ago, duration, plural, statusWord, useNow } from './format';
|
|
15
15
|
import type { FeatureExamples, FeatureStep } from './gherkin';
|
|
16
16
|
import { Bounded, Origin, StatusMark } from './marks';
|
|
17
17
|
import { ModelTrace } from './model-trace';
|
|
18
|
+
import { OutcomeSummary } from './outcome-summary';
|
|
19
|
+
import { failedReportStep, failedStep, recordOf, sourceLabels, type ResultRecord } from './outcome';
|
|
18
20
|
import { recordedModelStates } from './quint';
|
|
19
21
|
/** What the address says is open: a check, and optionally one particular layer's record of it. */
|
|
20
22
|
export type CheckSelection = {
|
|
@@ -49,13 +51,6 @@ function environmentMarks(slot: CheckSlot) {
|
|
|
49
51
|
if (seen.size < 2) return [];
|
|
50
52
|
return [...seen].map(([id, status]) => ({ id, status }));
|
|
51
53
|
}
|
|
52
|
-
function outcomeName(outcome: EvidenceResult) {
|
|
53
|
-
const environment = outcome.layer.environment ? environmentLabel(outcome.layer.environment) : '';
|
|
54
|
-
const parameters = Object.entries(outcome.parameters)
|
|
55
|
-
.map(([name, value]) => `${name}: ${value}`)
|
|
56
|
-
.join(', ');
|
|
57
|
-
return [environment, parameters].filter(Boolean).join(' · ') || outcome.layer.producer;
|
|
58
|
-
}
|
|
59
54
|
export function ResultRow({
|
|
60
55
|
label,
|
|
61
56
|
slot,
|
|
@@ -93,7 +88,7 @@ export function ResultRow({
|
|
|
93
88
|
<Flex component="span" display="inline-flex" gap="xs" align="center" wrap="wrap">
|
|
94
89
|
{marks.map((mark) => (
|
|
95
90
|
<Text key={mark.id || 'unspecified'} data-status={mark.status} span inherit>
|
|
96
|
-
{environmentLabel(mark.id || null)}
|
|
91
|
+
{environmentLabel(mark.id || null)} {statusWord(mark.status).toLowerCase()}
|
|
97
92
|
</Text>
|
|
98
93
|
))}
|
|
99
94
|
</Flex>
|
|
@@ -276,6 +271,14 @@ type Props = {
|
|
|
276
271
|
pinned?: HistoryEntry;
|
|
277
272
|
/** Choosing a run in the history always names its layer, even when that result is a current one, so the choice stays pinned. */
|
|
278
273
|
pin?: boolean;
|
|
274
|
+
/** The layer id of the current source the address selected (without pinning a run). Used with `onSource`; otherwise the choice is local. */
|
|
275
|
+
source?: string;
|
|
276
|
+
/** Called with the layer id when a source is chosen (null for the newest), so the page can keep the choice in its address. */
|
|
277
|
+
onSource?: (layerId: string | null) => void;
|
|
278
|
+
/** Whether the outcome summary states its own heading. False when the page already shows it. */
|
|
279
|
+
headline?: boolean;
|
|
280
|
+
/** Every record of this check the page knows (a node's history of it), when the layers held here are only the winners. */
|
|
281
|
+
records?: readonly ResultRecord[];
|
|
279
282
|
};
|
|
280
283
|
/** Everything recorded for one check: the chosen result, how it went, and every other run of it. */
|
|
281
284
|
export function CheckDetail(props: Props) {
|
|
@@ -306,16 +309,28 @@ export function CheckDetail(props: Props) {
|
|
|
306
309
|
</>
|
|
307
310
|
);
|
|
308
311
|
}
|
|
309
|
-
function RecordedCheckDetail({ slot, layerId, select, reads, written, pinned, pin }: Props) {
|
|
312
|
+
function RecordedCheckDetail({ slot, layerId, select, reads, written, pinned, pin, source, onSource, headline = true, records }: Props) {
|
|
310
313
|
const { evidence, backends, ready, pending } = useEvidence();
|
|
311
314
|
const now = useNow();
|
|
312
315
|
const [variant, setVariant] = useState(0);
|
|
313
316
|
const history = useMemo(() => checkHistory(evidence, slot.specPath, slot.id), [evidence, slot.specPath, slot.id]);
|
|
314
|
-
const
|
|
317
|
+
const sourceAt = onSource && source ? slot.outcomes.findIndex((outcome) => outcome.layer.id === source) : -1;
|
|
318
|
+
const index = Math.min(onSource ? Math.max(sourceAt, 0) : variant, slot.outcomes.length - 1);
|
|
319
|
+
const winner = slot.outcomes[index]!;
|
|
315
320
|
const earlier = pinned ?? (layerId ? history.find((entry) => entry.layer.id === layerId) : undefined);
|
|
316
321
|
const chosen: HistoryEntry = earlier ?? { layer: winner.layer, row: winner.row };
|
|
317
322
|
const superseded = !slot.outcomes.some((outcome) => outcome.layer.id === chosen.layer.id);
|
|
318
323
|
const { paths, report } = useRecordedReport(chosen);
|
|
324
|
+
const recorded = useCheckDetail(chosen.layer, chosen.row);
|
|
325
|
+
const point = useMemo(
|
|
326
|
+
() => (chosen.row.status === 'failed' ? (failedStep(recorded.data?.check.steps) ?? failedReportStep(report.data)) : undefined),
|
|
327
|
+
[chosen.row.status, recorded.data, report.data]
|
|
328
|
+
);
|
|
329
|
+
const labels = sourceLabels(slot.outcomes, now);
|
|
330
|
+
const choose = (at: number) => {
|
|
331
|
+
setVariant(at);
|
|
332
|
+
onSource?.(at === 0 ? null : slot.outcomes[at]!.layer.id);
|
|
333
|
+
};
|
|
319
334
|
const moments = report.data ? flattenFrames(report.data.steps).length : 0;
|
|
320
335
|
const timings = superseded ? [] : slot.diagnostics.filter((entry) => entry.layer.id === chosen.layer.id || slot.diagnostics.length === 1);
|
|
321
336
|
const note = superseded ? 'An earlier result. A newer run replaced it in the combined report.' : qualification(winner.warning);
|
|
@@ -328,12 +343,32 @@ function RecordedCheckDetail({ slot, layerId, select, reads, written, pinned, pi
|
|
|
328
343
|
: '';
|
|
329
344
|
return (
|
|
330
345
|
<Stack gap="sm" miw={0}>
|
|
346
|
+
<OutcomeSummary
|
|
347
|
+
slot={slot}
|
|
348
|
+
selected={chosen}
|
|
349
|
+
history={records && records.length > history.length ? records : history.map(recordOf)}
|
|
350
|
+
point={point}
|
|
351
|
+
detail={recorded.data}
|
|
352
|
+
now={now}
|
|
353
|
+
showHeadline={headline}
|
|
354
|
+
choose={(layer) => {
|
|
355
|
+
const at = slot.outcomes.findIndex((outcome) => outcome.layer.id === layer);
|
|
356
|
+
if (at >= 0) choose(at);
|
|
357
|
+
else select({ layer });
|
|
358
|
+
}}
|
|
359
|
+
/>
|
|
331
360
|
{slot.outcomes.length > 1 && !earlier && (
|
|
332
361
|
<Flex role="group" aria-label="Environments" gap="xs" align="center" wrap="wrap">
|
|
333
|
-
{slot.outcomes.map((outcome,
|
|
334
|
-
<Button
|
|
362
|
+
{slot.outcomes.map((outcome, at) => (
|
|
363
|
+
<Button
|
|
364
|
+
key={outcome.key}
|
|
365
|
+
type="button"
|
|
366
|
+
aria-pressed={at === index}
|
|
367
|
+
onClick={() => choose(at)}
|
|
368
|
+
variant="light"
|
|
369
|
+
>
|
|
335
370
|
<StatusMark status={outcome.row.status} size={12} />
|
|
336
|
-
{
|
|
371
|
+
{labels[at]!.text}
|
|
337
372
|
</Button>
|
|
338
373
|
))}
|
|
339
374
|
</Flex>
|
|
@@ -410,7 +445,7 @@ function RecordedCheckDetail({ slot, layerId, select, reads, written, pinned, pi
|
|
|
410
445
|
{moments > 0 && (
|
|
411
446
|
<Button
|
|
412
447
|
type="button"
|
|
413
|
-
onClick={() => select({ view: 'recording', check: slot.id, layer: earlier ||
|
|
448
|
+
onClick={() => select({ view: 'recording', check: slot.id, layer: earlier || index > 0 ? chosen.layer.id : null })}
|
|
414
449
|
variant="filled"
|
|
415
450
|
>
|
|
416
451
|
Open recording{' '}
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @module Outcome summary
|
|
3
|
+
*
|
|
4
|
+
* What the selected result says, stated apart from the other current sources, and the failure explained before its
|
|
5
|
+
* raw output: the heading names the selected source's outcome and what the other scoped sources recorded (each one
|
|
6
|
+
* click away), whether the behavior was reached, the recorded expected and observed values, the closest video and
|
|
7
|
+
* step, the chronology of the source, and who is to blame, which stays unknown unless the producer declared it. The
|
|
8
|
+
* original diagnostics remain below, untouched.
|
|
9
|
+
*/
|
|
10
|
+
import type { CheckSlot, HistoryEntry } from '@crvouga/atlas/reports/evidence-view';
|
|
11
|
+
import type { CheckDetail } from '@crvouga/atlas/reports/evidence-detail';
|
|
12
|
+
import { effectiveScope, scopeBadges } from '@crvouga/atlas/reports/evidence-scope';
|
|
13
|
+
import { Badge, Box, Button, Flex, List, Paper, Stack, Text, Title } from '@mantine/core';
|
|
14
|
+
|
|
15
|
+
import { ago, duration, shortSha, statusWord } from './format';
|
|
16
|
+
import { StatusMark } from './marks';
|
|
17
|
+
import {
|
|
18
|
+
chronology,
|
|
19
|
+
failureNote,
|
|
20
|
+
outcomeHeadline,
|
|
21
|
+
passedAfterFailure,
|
|
22
|
+
recordOf,
|
|
23
|
+
type ResultRecord,
|
|
24
|
+
reachOf,
|
|
25
|
+
sourceLabels,
|
|
26
|
+
videosIn,
|
|
27
|
+
type FailurePoint
|
|
28
|
+
} from './outcome';
|
|
29
|
+
|
|
30
|
+
const BLAME: Record<string, string> = {
|
|
31
|
+
product: 'the product',
|
|
32
|
+
'test-harness': 'the test harness',
|
|
33
|
+
environment: 'the environment'
|
|
34
|
+
};
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* The outcome of the selected result in words, the other current sources beside it, and, for a failure, where and
|
|
38
|
+
* how it failed. `point` is the failed step found in the recording (undefined when none was recorded); `detail` is the
|
|
39
|
+
* check's full record, for its videos and scope. `choose` selects one of the other current sources; `showHeadline`
|
|
40
|
+
* is false when the page already shows the heading.
|
|
41
|
+
*/
|
|
42
|
+
export function OutcomeSummary({
|
|
43
|
+
slot,
|
|
44
|
+
selected,
|
|
45
|
+
history,
|
|
46
|
+
point,
|
|
47
|
+
detail,
|
|
48
|
+
now,
|
|
49
|
+
choose,
|
|
50
|
+
showHeadline = true
|
|
51
|
+
}: {
|
|
52
|
+
slot: CheckSlot;
|
|
53
|
+
selected: HistoryEntry;
|
|
54
|
+
history: readonly ResultRecord[];
|
|
55
|
+
point: FailurePoint | undefined;
|
|
56
|
+
detail: CheckDetail | undefined;
|
|
57
|
+
now: number;
|
|
58
|
+
choose: (layerId: string) => void;
|
|
59
|
+
showHeadline?: boolean;
|
|
60
|
+
}) {
|
|
61
|
+
const headline = outcomeHeadline(selected, slot);
|
|
62
|
+
const failed = selected.row.status === 'failed';
|
|
63
|
+
const reach = failed ? reachOf(point) : undefined;
|
|
64
|
+
const note = point?.note ?? failureNote(detail?.check.extensions);
|
|
65
|
+
const every = sourceLabels(slot.outcomes, now);
|
|
66
|
+
const labels = headline.others.map((outcome) => every[slot.outcomes.indexOf(outcome)]!);
|
|
67
|
+
const video = point?.videos[0] ?? videosIn(detail?.check)[0];
|
|
68
|
+
const scope = video ? effectiveScope(video.scope, detail?.check.scope, detail?.inheritedScope) : undefined;
|
|
69
|
+
const timeline = chronology(recordOf(selected), history);
|
|
70
|
+
return (
|
|
71
|
+
<Paper component="section" aria-label="Outcome summary" data-outcome={selected.row.status} withBorder p="md">
|
|
72
|
+
<Stack gap="sm" miw={0}>
|
|
73
|
+
{showHeadline && (
|
|
74
|
+
<Title order={3} data-outcome-headline>
|
|
75
|
+
{headline.text}
|
|
76
|
+
</Title>
|
|
77
|
+
)}
|
|
78
|
+
{headline.others.length > 0 && (
|
|
79
|
+
<Stack gap="xs" component="section" aria-label="Other current sources">
|
|
80
|
+
<Text component="div" c="dimmed">
|
|
81
|
+
Other current sources of this check
|
|
82
|
+
</Text>
|
|
83
|
+
<List listStyleType="none" spacing="xs">
|
|
84
|
+
{headline.others.map((outcome, index) => (
|
|
85
|
+
<List.Item key={outcome.key}>
|
|
86
|
+
<Button type="button" variant="light" onClick={() => choose(outcome.layer.id)} aria-label={`Open ${labels[index]!.text}`}>
|
|
87
|
+
<StatusMark status={outcome.row.status} size={12} />
|
|
88
|
+
<Text span inherit>
|
|
89
|
+
{labels[index]!.text}
|
|
90
|
+
</Text>
|
|
91
|
+
</Button>
|
|
92
|
+
</List.Item>
|
|
93
|
+
))}
|
|
94
|
+
</List>
|
|
95
|
+
</Stack>
|
|
96
|
+
)}
|
|
97
|
+
{reach && (
|
|
98
|
+
<Flex gap="xs" align="center" wrap="wrap" data-reach={reach.state}>
|
|
99
|
+
<Badge variant={reach.state === 'setup-failed' ? 'filled' : 'light'} color={reach.state === 'unknown' ? 'gray' : 'red'}>
|
|
100
|
+
{reach.state === 'setup-failed'
|
|
101
|
+
? 'Behavior not verified'
|
|
102
|
+
: reach.state === 'expectation-failed'
|
|
103
|
+
? 'Expectation not met'
|
|
104
|
+
: 'Failure point unknown'}
|
|
105
|
+
</Badge>
|
|
106
|
+
<Text span inherit>
|
|
107
|
+
{reach.text}
|
|
108
|
+
</Text>
|
|
109
|
+
</Flex>
|
|
110
|
+
)}
|
|
111
|
+
{failed && (note?.expected !== undefined || note?.actual !== undefined) && (
|
|
112
|
+
<Box component="dl" aria-label="Expected and observed" m={0}>
|
|
113
|
+
{note?.expected !== undefined && (
|
|
114
|
+
<>
|
|
115
|
+
<Text component="dt" fw={600}>
|
|
116
|
+
Expected
|
|
117
|
+
</Text>
|
|
118
|
+
<Text component="dd" m={0}>
|
|
119
|
+
{note.expected}
|
|
120
|
+
</Text>
|
|
121
|
+
</>
|
|
122
|
+
)}
|
|
123
|
+
{note?.actual !== undefined && (
|
|
124
|
+
<>
|
|
125
|
+
<Text component="dt" fw={600}>
|
|
126
|
+
Observed
|
|
127
|
+
</Text>
|
|
128
|
+
<Text component="dd" m={0}>
|
|
129
|
+
{note.actual}
|
|
130
|
+
</Text>
|
|
131
|
+
</>
|
|
132
|
+
)}
|
|
133
|
+
</Box>
|
|
134
|
+
)}
|
|
135
|
+
{failed && (point || video) && (
|
|
136
|
+
<Stack gap="xs" component="section" aria-label="Closest evidence">
|
|
137
|
+
{point && (
|
|
138
|
+
<Text component="div">
|
|
139
|
+
Closest step:{' '}
|
|
140
|
+
<Text span inherit fw={600}>
|
|
141
|
+
{point.name}
|
|
142
|
+
</Text>
|
|
143
|
+
{point.offsetMs !== undefined && (
|
|
144
|
+
<Text span inherit c="dimmed">
|
|
145
|
+
{' '}
|
|
146
|
+
at {duration(point.offsetMs) || '0 ms'} into the check
|
|
147
|
+
</Text>
|
|
148
|
+
)}
|
|
149
|
+
</Text>
|
|
150
|
+
)}
|
|
151
|
+
{video && (
|
|
152
|
+
<Text component="div">
|
|
153
|
+
Closest video:{' '}
|
|
154
|
+
<Text span inherit fw={600}>
|
|
155
|
+
{video.description ?? video.name}
|
|
156
|
+
</Text>
|
|
157
|
+
<Text span inherit c="dimmed">
|
|
158
|
+
{' '}
|
|
159
|
+
{point?.videos.length ? 'attached to that step' : 'recorded for the whole check'}
|
|
160
|
+
{' · '}
|
|
161
|
+
{scopeBadges(scope)
|
|
162
|
+
.map((badge) => badge.text)
|
|
163
|
+
.join(', ')}
|
|
164
|
+
</Text>
|
|
165
|
+
</Text>
|
|
166
|
+
)}
|
|
167
|
+
</Stack>
|
|
168
|
+
)}
|
|
169
|
+
{failed && (
|
|
170
|
+
<Text component="div" c="dimmed" data-attribution={note?.attribution ?? 'unknown'}>
|
|
171
|
+
Cause:{' '}
|
|
172
|
+
{note?.attribution
|
|
173
|
+
? `${BLAME[note.attribution]}, as declared by the producer`
|
|
174
|
+
: 'unknown. Nothing recorded says whether the product, the test harness or the environment is at fault.'}
|
|
175
|
+
</Text>
|
|
176
|
+
)}
|
|
177
|
+
{timeline.length > 0 && (
|
|
178
|
+
<Stack gap="xs" component="section" aria-label="Chronology of this source">
|
|
179
|
+
<Text component="div" c="dimmed">
|
|
180
|
+
Records of this source, oldest first
|
|
181
|
+
</Text>
|
|
182
|
+
<List listStyleType="none" spacing="xs" type="ordered">
|
|
183
|
+
{timeline.map(({ record, selected: current }) => (
|
|
184
|
+
<List.Item key={`${record.layer}|${record.kind}`} aria-current={current || undefined} data-status={record.status}>
|
|
185
|
+
<Flex gap="xs" align="center" wrap="wrap">
|
|
186
|
+
<StatusMark status={record.status} size={12} />
|
|
187
|
+
<Text span inherit fw={current ? 600 : undefined}>
|
|
188
|
+
{statusWord(record.status)}
|
|
189
|
+
</Text>
|
|
190
|
+
<Text span inherit c="dimmed">
|
|
191
|
+
{ago(record.at, now)} · {shortSha(record.sha) ?? 'revision not recorded'}
|
|
192
|
+
{current ? ' · selected' : ''}
|
|
193
|
+
</Text>
|
|
194
|
+
</Flex>
|
|
195
|
+
</List.Item>
|
|
196
|
+
))}
|
|
197
|
+
</List>
|
|
198
|
+
{passedAfterFailure(timeline) && (
|
|
199
|
+
<Text component="div" role="status">
|
|
200
|
+
The latest run passed after an earlier failure. Nothing recorded shows that the failure was fixed or why it passed.
|
|
201
|
+
</Text>
|
|
202
|
+
)}
|
|
203
|
+
</Stack>
|
|
204
|
+
)}
|
|
205
|
+
</Stack>
|
|
206
|
+
</Paper>
|
|
207
|
+
);
|
|
208
|
+
}
|