@crvouga/atlas 0.9.0 → 0.10.0

package/docs/reference/schema.md

@@ -209,6 +209,63 @@ export const PingEventSchema = …
 
 The data of the `ping` event: sent every `heartbeatSeconds`, so a quiet server is told apart from a dead connection.
 
+#### `TestResultEntrySchema`
+
+```ts
+export const TestResultEntrySchema = …
+```
+
+One recorded result of a test in the test catalog: when, where, how it went and how long it took, and where it came from.
+
+#### `TestSummarySchema`
+
+```ts
+export const TestSummarySchema = …
+```
+
+One test of the catalog: what it is, its current result per environment, and how it has gone over its recorded history.
+
+#### `TestCatalogSchema`
+
+```ts
+export const TestCatalogSchema = …
+```
+
+`GET /api/reports/tests`: every test of the report with a summary, insights and facets. The summary and facets cover
+every test; `tests` is the page of those the query matched.
+
+#### `TestHistorySchema`
+
+```ts
+export const TestHistorySchema = …
+```
+
+`GET /api/reports/tests/history`: one test and every result recorded for it (at most 100), newest first.
+
+#### `ReportRunSchema`
+
+```ts
+export const ReportRunSchema = …
+```
+
+A run: the layers that share a run URL (else a branch and commit), with what they recorded.
+
+#### `RunListSchema`
+
+```ts
+export const RunListSchema = …
+```
+
+`GET /api/reports/runs`: runs, newest first.
+
+#### `RunDetailSchema`
+
+```ts
+export const RunDetailSchema = …
+```
+
+`GET /api/reports/runs/detail`: one run and its last result for every test, environment and producer it recorded.
+
 ### Command results (`--format json`)
 
 Source: `packages/schema/src/cli-results.ts`

package/docs/rich-evidence.md

@@ -0,0 +1,198 @@
+# Send rich evidence
+
+A check that says only its title and `passed` answers one question: did it pass? A check that also has its duration,
+the device it ran on, its steps, its retries and its video answers the next ones too: how slow is it, where does it
+fail, what happened, is it flaky. This guide covers which fields a producer should send, what the viewer does with
+each, and two ways to send them: the Playwright Test reporter, which sends all of them for you, and `createReporter` /
+`publishChecks` for any other producer.
+
+Every field here is optional and none of them changes a result: a check's status, its slot and which result wins are
+decided exactly as before ([the report contract](report-contract.md)). Evidence only adds information.
+[Evidence](extending.md) is the full model, with the rules that keep it compatible.
+
+- [What to send, and what it is for](#what-to-send-and-what-it-is-for)
+- [Labels describe, parameters split](#labels-describe-parameters-split)
+- [Playwright Test](#playwright-test)
+- [Any other producer](#any-other-producer)
+
+## What to send, and what it is for
+
+| Field | Where | What the viewer does with it |
+| --- | --- | --- |
+| `durationMs` | check, attempt, step | Averages, p95 and the slowest tests, per environment and over time. |
+| `environment` | layer | Which client was checked (`desktop-web`, `mobile-web`, `ios`, `android`…). A check on two environments is two results, side by side. |
+| `labels` (`browser`, `device`, `os`, `viewport`…) | check, layer | Shows each result's browser, device and system, and lets you filter by them. |
+| `attachments` with a `role` | check, attempt, step, layer | Plays `video`, shows a `screenshot`, opens a `trace`, shows logs and offers anything else for download. |
+| `steps` (`parentId`, `offsetMs`, `durationMs`, `status`, `kind`) | check, attempt | Draws the check's timeline: nested steps, when each started, how long it took, which one failed. |
+| `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
+| `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
+| `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
+| `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
+| `specs[]` (`path`, `domain`, `kind`, `title`) | layer | Names each spec and its group (`domain`), so a server can group checks by spec with a title even when no [spec set](integrating.md#tell-atlas-which-spec-a-check-belongs-to) has been published. |
+| `detail` | check, attempt, step | Why it failed: the error message and stack, as plain text. |
+
+Send `sha`, `branch` and `runUrl` on every layer too. They never change a result, but they tie results to a commit
+and to the CI run that produced them.
+
+## Labels describe, parameters split
+
+A result's **slot** is its producer, check id, part kind, spec, environment and **parameters**. The newest completed
+result of a slot wins. So:
+
+- Put what describes a run in `labels`: browser, device name, viewport, operating system, Playwright version, shard.
+  Two runs that differ only in labels fill the same slot, and the newer one wins.
+- Put what makes a run a different result in `parameters` (a locale, a tenant, a feature flag): each value is a slot
+  of its own, kept side by side.
+- Put the client in `environment`. It is a parameter that every view understands.
+
+If you label a check `device: Pixel 7` on one run and `device: iPhone 15` on the next, both in `mobile-web`, the report
+shows the newer one. To see both, give them different environments or a parameter.
+
+## Playwright Test
+
+`@crvouga/atlas/playwright-reporter` is a Playwright Test reporter. Add it beside your console reporter:
+
+```ts
+// playwright.config.ts
+import { defineConfig, devices } from '@playwright/test';
+
+export default defineConfig({
+  retries: process.env.CI ? 2 : 0,
+  reporter: [['list'], ['@crvouga/atlas/playwright-reporter', { project: 'checkout', producer: 'e2e-web' }]],
+  use: { video: 'retain-on-failure', trace: 'retain-on-failure', screenshot: 'only-on-failure' },
+  projects: [
+    { name: 'Desktop Chrome', use: { ...devices['Desktop Chrome'] } },
+    { name: 'Pixel 7', use: { ...devices['Pixel 7'] } }
+  ]
+});
+```
+
+Then run your tests as usual, with the destination in the environment:
+
+```sh
+ATLAS_SERVER_URL=https://atlas.example.com ATLAS_TOKEN=$ATLAS_PUBLISH_TOKEN ATLAS_PROJECT=checkout npx playwright test
+# … Playwright's own output …
+# Atlas: 3512 checks in 140 layers to https://atlas.example.com
+```
+
+What it sends:
+
+- Each test as `running` when it starts, then with its result when its last try ends, every few seconds, so the
+  report follows the run as it happens.
+- One stream of layers per Playwright project. The environment is `mobile-web` for a project that emulates a mobile
+  device (`use.isMobile`), else `desktop-web`. Every check is labelled `browser`, `device` (the project name),
+  `viewport`, `os` and `playwright` (its version), and `shard` when the run is sharded.
+- A check id of `<specPath>#<describe titles> > <test title>`, the same in every run.
+- `passed` for a test that got the result it expected, `failed` for a failure, a timeout or an interruption,
+  `not-run` for a skipped test or one the run ended before it finished. A test that passed on a retry is `passed`,
+  tagged and labelled `flaky`, with one attempt per try.
+- The failure message and stack without terminal colours, the duration, `test.tags` (without `@`) and annotation
+  types as tags, annotations, links (annotations whose description is a URL), and Playwright's steps as the timeline
+  (`hook`, `fixture`, `action` for `pw:api`, `assertion` for `expect`, `step` for `test.step`).
+- Every attachment, bytes included: `video`, `screenshot` and `trace` take that role, other attachments their own
+  name, and standard output and error are `log`s. Attachments of earlier tries go with their attempt.
+- The project's spec files in `specs[]`, so the server can group checks by spec.
+
+It never fails the run. A destination that is not set, a server that is down or a file that cannot be read is said
+once, and the summary line says how many checks could not be sent.
+
+### Options
+
+| Option | Default | |
+| --- | --- | --- |
+| `url` | `ATLAS_SERVER_URL`, else `ATLAS_STORE` | The Atlas server, or any store location (`./atlas-reports`, `s3://bucket/prefix`). |
+| `token` | `ATLAS_PUBLISH_TOKEN`, else `ATLAS_TOKEN` (or `ATLAS_TOKEN_FILE`) | A publish token. A token is never sent over plain `http` beyond this machine. |
+| `project` | `ATLAS_PROJECT` | The project on a shared Atlas server. |
+| `store` | | Publish to this `ObjectStore` instead of a URL (a directory store, an in-memory store in tests). |
+| `producer` | `playwright` | A stable name for this suite. |
+| `kind` | `checks` | The part kind of every layer. |
+| `environment` | `mobile-web` or `desktop-web` | A string, or `(project) => string`. |
+| `parameters` | `{}` | Choices that make a result its own slot. An object, or `(project) => object`. |
+| `labels` | | More labels for every check. An object, or `(project) => object`. |
+| `sha`, `branch`, `runUrl` | `ATLAS_SHA`, `ATLAS_BRANCH`, then the CI's variables (`GITHUB_SHA`, `GITHUB_HEAD_REF` / `GITHUB_REF_NAME`, the GitHub Actions run URL…), then git | Where the run came from. |
+| `specPath` | the `.feature` of a playwright-bdd test, else the test file | `(test) => string`, relative to the repository. |
+| `rootDir` | the git root above Playwright's `rootDir` | Paths are relative to it. |
+| `attachments` | `all` | `all`, `failures` (failed checks and failed tries only) or `none`. |
+| `maxAttachmentBytes` | 50 MiB | A larger file is not uploaded; the check gets an `attachment-skipped` annotation. |
+| `maxSteps` | 500 | Timeline steps kept per try; the rest are counted in a `steps-truncated` annotation. |
+| `specs` | `true` | Record the project's spec files in each layer. |
+| `flushMs`, `batchSize`, `attempts` | 5000, 50, 5 | How often to send, how many checks to send at once, how many tries per layer at the end. |
+| `quiet` | `false` | Print nothing, not even the summary. |
+
+Playwright decides what there is to upload: set `video`, `trace` and `screenshot` in `use` (`retain-on-failure` and
+`only-on-failure` keep uploads small), and anything you `testInfo.attach()` is uploaded too. Each file is stored once
+by its content, in chunks, so a server's object size limit does not cap a video.
+
+### playwright-bdd
+
+Tests generated by [playwright-bdd](https://github.com/vitalets/playwright-bdd) belong to their `.feature` file: a
+test in `.features-gen/<path>.feature.spec.js` gets the `specPath` of `<path>.feature`, found beside `.features-gen`
+(a per-project output folder in between is skipped). The scenario's title is the check's title (an outline's example
+is `<outline>: Example #1`), feature tags are tags, `Given` / `When` / `Then` steps are timeline steps of that kind,
+and the feature's name is the spec's title in `specs[]`. If your features are elsewhere, give `specPath`:
+
+```ts
+['@crvouga/atlas/playwright-reporter', { specPath: (test) => test.location.file.replace(/^.*\/generated\//, 'features/').replace(/\.spec\.js$/, '') }]
+```
+
+## Any other producer
+
+`createReporter` streams checks as they finish, one layer per flush, to a server or any store
+([streaming a run](server.md#streaming-a-run-as-it-happens)); `publishChecks` publishes one finished layer. Both take
+the same check. Attachment bytes go in as `data` (a `Uint8Array`); read files yourself (only the command line reads a
+`path`). Here is a check that carries everything:
+
+```ts
+import { readFile } from 'node:fs/promises';
+import { createReporter } from '@crvouga/atlas/reports/reporter';
+import { httpStore } from '@crvouga/atlas/reports/http-store';
+
+const reporter = createReporter(
+  {
+    producer: 'e2e-web',
+    environment: 'mobile-web',
+    sha: process.env.GITHUB_SHA ?? null,
+    branch: process.env.GITHUB_HEAD_REF || process.env.GITHUB_REF_NAME || null,
+    runUrl: `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`,
+    // Names and groups each spec, so checks are grouped by spec without a published spec set.
+    specs: [{ path: 'features/checkout/pay.feature', domain: 'checkout', kind: 'feature', title: 'Paying for an order' }]
+  },
+  { store: httpStore({ url: 'https://atlas.example.com', project: 'shop', token: process.env.ATLAS_TOKEN }) }
+);
+
+reporter.check({ id: 'features/checkout/pay.feature#Pays by card', title: 'Pays by card', status: 'running', specPath: 'features/checkout/pay.feature' });
+
+reporter.check({
+  id: 'features/checkout/pay.feature#Pays by card',
+  title: 'Pays by card',
+  status: 'passed',
+  detail: 'Passed on retry 1 (flaky)',
+  specPath: 'features/checkout/pay.feature',
+  durationMs: 4180,
+  tags: ['smoke', 'flaky'],
+  labels: { browser: 'chromium', device: 'Pixel 7', os: 'linux', viewport: '412x839' },
+  steps: [
+    { id: 's1', name: 'Given a cart with one item', kind: 'given', status: 'passed', offsetMs: 0, durationMs: 900 },
+    { id: 's2', name: 'When I pay by card', kind: 'when', status: 'passed', offsetMs: 900, durationMs: 2600 },
+    { id: 's3', parentId: 's2', name: 'click #pay', kind: 'action', status: 'passed', offsetMs: 1200, durationMs: 80 },
+    { id: 's4', name: 'Then I see the receipt', kind: 'then', status: 'passed', offsetMs: 3500, durationMs: 680 }
+  ],
+  attempts: [
+    { index: 0, status: 'failed', durationMs: 30000, startedAtMs: Date.now() - 40000, detail: 'Timeout 30000ms exceeded',
+      attachments: [{ name: 'retry-0.png', role: 'screenshot', contentType: 'image/png', data: await readFile('test-results/retry-0.png') }] },
+    { index: 1, status: 'passed', durationMs: 4180, startedAtMs: Date.now() - 5000 }
+  ],
+  attachments: [
+    { name: 'video.webm', role: 'video', contentType: 'video/webm', data: await readFile('test-results/video.webm') },
+    { name: 'trace.zip', role: 'trace', contentType: 'application/zip', data: await readFile('test-results/trace.zip') }
+  ]
+});
+
+const { checks, layers, lost } = await reporter.close(); // never throws; `lost` counts what never arrived
+```
+
+With `publishChecks`, pass the same identity and checks with `recordedAtMs`, once the run is over. From a checks file
+or the command line, an attachment may name a `path` instead; `atlas report publish` reads it
+([attach evidence](extending.md#attach-evidence)). A CTRF report already carries most of this, and
+`atlas report publish --ctrf` keeps it ([native runner reporters](runners.md) cover CodeceptJS, Mocha, Jest and Node's
+test runner).

package/docs/server.md

@@ -243,6 +243,39 @@ report the server would have sent whole (`applyEffectiveDelta`, tested against e
 runs); a version the server does not remember, or an older server, answers with the whole report, which a client
 takes as it always did. A result streamed into a 100 000-check report costs a client a few kilobytes.
 
+## Reading the report as an agent
+
+The effective report says what each check's result is now. To explore a project, or to answer "what is tested,
+where, how long does it take, what is flaky", read the tests instead. Every endpoint here answers JSON, or Markdown
+with `format=markdown` (or `Accept: text/markdown`), and takes the project prefix like every other
+(`/api/projects/<slug>/reports/tests`).
+
+| Ask | Endpoint |
+| --- | --- |
+| Every test, a summary (by status, environment, producer, domain), insights (failing, flaky, slowest) and facets | `GET /api/reports/tests` |
+| Only some: `q` (all words), `status` (or `flaky`), `environment`, `producer`, `domain`, `spec`, `tag`, `run`; `sort`, `offset`, `limit` | `GET /api/reports/tests?status=failed&environment=ios` |
+| One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
+| The runs that recorded the report (by CI run URL, else branch and commit) | `GET /api/reports/runs` |
+| Every test one run recorded | `GET /api/reports/runs/detail?key=<run key>` |
+
+```sh
+curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdown&limit=50"
+```
+
+- A **test** is a check identity, its spec path and id, across producers and environments. `outcomes` holds its
+  current result on each environment (the report's winners, so a test's status here is always the gate's);
+  `stats` describes its recorded history: pass rate, flips, whether it is flaky and why (it passed and failed on the
+  same commit, passed only after a retry, or its producer said so), durations (avg, p50, p95, min, max, last), runs
+  and branches.
+- The server keeps the newest 100 results of every test (progress reported as `running` is not history) and every run's last result per test, in memory, as layers
+  arrive; it is rebuilt from the layers on restart. History never changes a result.
+- Without a published spec set, tests are grouped under the specs their layers recorded (`specs[]`: path, domain,
+  title), and the gate and the outline do the same, so a project needs no spec set to be browsable. A path no layer
+  described is titled from its file name and grouped by its folder under `specs/`.
+- Statistics are only as rich as what producers send: durations need `durationMs`, devices need an environment and
+  labels (`browser`, `device`, `os`), videos and screenshots need attachments. See
+  [Send rich evidence](rich-evidence.md).
+
 ## API reference
 
 <!-- generated:server-operations -->
@@ -283,6 +316,14 @@ takes as it always did. A result streamed into a 100 000-check report costs a cl
 | `PUT /api/projects/{project}/store/{key}` | publish | Upload an object. |
 | `DELETE /api/store/{key}` | publish | Remove a scratch object. |
 | `DELETE /api/projects/{project}/store/{key}` | publish | Remove a scratch object. |
+| `GET /api/reports/tests` | read | Every test, with insights, facets and history statistics. |
+| `GET /api/projects/{project}/reports/tests` | read | Every test, with insights, facets and history statistics. |
+| `GET /api/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/projects/{project}/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/reports/runs` | read | The runs that recorded the report. |
+| `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
+| `GET /api/reports/runs/detail` | read | Every test one run recorded. |
+| `GET /api/projects/{project}/reports/runs/detail` | read | Every test one run recorded. |
 | `GET /api/reports/outline` | read | The specs a report is laid over. |
 | `GET /api/projects/{project}/reports/outline` | read | The specs a report is laid over. |
 | `GET /api/reports/outline/spec` | read | The text of one spec. |