@crvouga/atlas 0.9.0 → 0.10.1

package/llms-full.txt

@@ -120,6 +120,7 @@ The complete public export map is generated from the package manifest (wildcards
 @crvouga/atlas/codecept
 @crvouga/atlas/jest
 @crvouga/atlas/node-test
+@crvouga/atlas/playwright-reporter
 ```
 <!-- /generated:public-exports -->
 
@@ -605,6 +606,7 @@ bunx atlas report publish --producer e2e --junit out.xml --attach ci.log --attac
 Files are stored in the layer by content, shown by what they are (pictures, videos, page snapshots, text) and
 offered for download otherwise. [Evidence](extending.md) is the full model, and why a new kind of evidence never
 breaks anything.
+[Send rich evidence](rich-evidence.md) lists what to send and what the viewer does with it, and covers the Playwright Test reporter.
 
 ### Statecharts
 
@@ -949,6 +951,207 @@ Run `atlas server` once for the projects the node hosts; point local and CI publ
 Native reporters send a heartbeat every 5 seconds. The server interrupts invocations after 30 seconds of silence, or after the configurable progress-stall timeout. It preserves finished results and never marks missing tests as passed. See [live test runs](server.md#live-test-runs-and-heartbeats).
 </doc>
 
+<doc path="docs/rich-evidence.md">
+# Send rich evidence
+
+A check that says only its title and `passed` answers one question: did it pass? A check that also has its duration,
+the device it ran on, its steps, its retries and its video answers the next ones too: how slow is it, where does it
+fail, what happened, is it flaky. This guide covers which fields a producer should send, what the viewer does with
+each, and two ways to send them: the Playwright Test reporter, which sends all of them for you, and `createReporter` /
+`publishChecks` for any other producer.
+
+Every field here is optional and none of them changes a result: a check's status, its slot and which result wins are
+decided exactly as before ([the report contract](report-contract.md)). Evidence only adds information.
+[Evidence](extending.md) is the full model, with the rules that keep it compatible.
+
+- [What to send, and what it is for](#what-to-send-and-what-it-is-for)
+- [Labels describe, parameters split](#labels-describe-parameters-split)
+- [Playwright Test](#playwright-test)
+- [Any other producer](#any-other-producer)
+
+## What to send, and what it is for
+
+| Field | Where | What the viewer does with it |
+| --- | --- | --- |
+| `durationMs` | check, attempt, step | Averages, p95 and the slowest tests, per environment and over time. |
+| `environment` | layer | Which client was checked (`desktop-web`, `mobile-web`, `ios`, `android`…). A check on two environments is two results, side by side. |
+| `labels` (`browser`, `device`, `os`, `viewport`…) | check, layer | Shows each result's browser, device and system, and lets you filter by them. |
+| `attachments` with a `role` | check, attempt, step, layer | Plays `video`, shows a `screenshot`, opens a `trace`, shows logs and offers anything else for download. |
+| `steps` (`parentId`, `offsetMs`, `durationMs`, `status`, `kind`) | check, attempt | Draws the check's timeline: nested steps, when each started, how long it took, which one failed. |
+| `attempts` | check | Every try of a retried check, with its own status, detail, steps and attachments. A check that passed after failing is flaky. |
+| `tags`, `labels` | check, layer | Filters and groups (`smoke`, `flaky`, an owner, a browser). |
+| `annotations`, `links` | check, step, attempt, layer | Notes, warnings and issue links next to the result. |
+| `specPath` | check | The spec a check belongs to (a `.feature`, a test file), relative to the repository. Checks are grouped by it. |
+| `specs[]` (`path`, `domain`, `kind`, `title`) | layer | Names each spec and its group (`domain`), so a server can group checks by spec with a title even when no [spec set](integrating.md#tell-atlas-which-spec-a-check-belongs-to) has been published. |
+| `detail` | check, attempt, step | Why it failed: the error message and stack, as plain text. |
+
+Send `sha`, `branch` and `runUrl` on every layer too. They never change a result, but they tie results to a commit
+and to the CI run that produced them.
+
+## Labels describe, parameters split
+
+A result's **slot** is its producer, check id, part kind, spec, environment and **parameters**. The newest completed
+result of a slot wins. So:
+
+- Put what describes a run in `labels`: browser, device name, viewport, operating system, Playwright version, shard.
+  Two runs that differ only in labels fill the same slot, and the newer one wins.
+- Put what makes a run a different result in `parameters` (a locale, a tenant, a feature flag): each value is a slot
+  of its own, kept side by side.
+- Put the client in `environment`. It is a parameter that every view understands.
+
+If you label a check `device: Pixel 7` on one run and `device: iPhone 15` on the next, both in `mobile-web`, the report
+shows the newer one. To see both, give them different environments or a parameter.
+
+## Playwright Test
+
+`@crvouga/atlas/playwright-reporter` is a Playwright Test reporter. Add it beside your console reporter:
+
+```ts
+// playwright.config.ts
+import { defineConfig, devices } from '@playwright/test';
+
+export default defineConfig({
+  retries: process.env.CI ? 2 : 0,
+  reporter: [['list'], ['@crvouga/atlas/playwright-reporter', { project: 'checkout', producer: 'e2e-web' }]],
+  use: { video: 'retain-on-failure', trace: 'retain-on-failure', screenshot: 'only-on-failure' },
+  projects: [
+    { name: 'Desktop Chrome', use: { ...devices['Desktop Chrome'] } },
+    { name: 'Pixel 7', use: { ...devices['Pixel 7'] } }
+  ]
+});
+```
+
+Then run your tests as usual, with the destination in the environment:
+
+```sh
+ATLAS_SERVER_URL=https://atlas.example.com ATLAS_TOKEN=$ATLAS_PUBLISH_TOKEN ATLAS_PROJECT=checkout npx playwright test
+# … Playwright's own output …
+# Atlas: 3512 checks in 140 layers to https://atlas.example.com
+```
+
+What it sends:
+
+- Each test as `running` when it starts, then with its result when its last try ends, every few seconds, so the
+  report follows the run as it happens.
+- One stream of layers per Playwright project. The environment is `mobile-web` for a project that emulates a mobile
+  device (`use.isMobile`), else `desktop-web`. Every check is labelled `browser`, `device` (the project name),
+  `viewport`, `os` and `playwright` (its version), and `shard` when the run is sharded.
+- A check id of `<specPath>#<describe titles> > <test title>`, the same in every run.
+- `passed` for a test that got the result it expected, `failed` for a failure, a timeout or an interruption,
+  `not-run` for a skipped test or one the run ended before it finished. A test that passed on a retry is `passed`,
+  tagged and labelled `flaky`, with one attempt per try.
+- The failure message and stack without terminal colours, the duration, `test.tags` (without `@`) and annotation
+  types as tags, annotations, links (annotations whose description is a URL), and Playwright's steps as the timeline
+  (`hook`, `fixture`, `action` for `pw:api`, `assertion` for `expect`, `step` for `test.step`).
+- Every attachment, bytes included: `video`, `screenshot` and `trace` take that role, other attachments their own
+  name, and standard output and error are `log`s. Attachments of earlier tries go with their attempt.
+- The project's spec files in `specs[]`, so the server can group checks by spec.
+
+It never fails the run. A destination that is not set, a server that is down or a file that cannot be read is said
+once, and the summary line says how many checks could not be sent.
+
+### Options
+
+| Option | Default | |
+| --- | --- | --- |
+| `url` | `ATLAS_SERVER_URL`, else `ATLAS_STORE` | The Atlas server, or any store location (`./atlas-reports`, `s3://bucket/prefix`). |
+| `token` | `ATLAS_PUBLISH_TOKEN`, else `ATLAS_TOKEN` (or `ATLAS_TOKEN_FILE`) | A publish token. A token is never sent over plain `http` beyond this machine. |
+| `project` | `ATLAS_PROJECT` | The project on a shared Atlas server. |
+| `store` | | Publish to this `ObjectStore` instead of a URL (a directory store, an in-memory store in tests). |
+| `producer` | `playwright` | A stable name for this suite. |
+| `kind` | `checks` | The part kind of every layer. |
+| `environment` | `mobile-web` or `desktop-web` | A string, or `(project) => string`. |
+| `parameters` | `{}` | Choices that make a result its own slot. An object, or `(project) => object`. |
+| `labels` | | More labels for every check. An object, or `(project) => object`. |
+| `sha`, `branch`, `runUrl` | `ATLAS_SHA`, `ATLAS_BRANCH`, then the CI's variables (`GITHUB_SHA`, `GITHUB_HEAD_REF` / `GITHUB_REF_NAME`, the GitHub Actions run URL…), then git | Where the run came from. |
+| `specPath` | the `.feature` of a playwright-bdd test, else the test file | `(test) => string`, relative to the repository. |
+| `rootDir` | the git root above Playwright's `rootDir` | Paths are relative to it. |
+| `attachments` | `all` | `all`, `failures` (failed checks and failed tries only) or `none`. |
+| `maxAttachmentBytes` | 50 MiB | A larger file is not uploaded; the check gets an `attachment-skipped` annotation. |
+| `maxSteps` | 500 | Timeline steps kept per try; the rest are counted in a `steps-truncated` annotation. |
+| `specs` | `true` | Record the project's spec files in each layer. |
+| `flushMs`, `batchSize`, `attempts` | 5000, 50, 5 | How often to send, how many checks to send at once, how many tries per layer at the end. |
+| `quiet` | `false` | Print nothing, not even the summary. |
+
+Playwright decides what there is to upload: set `video`, `trace` and `screenshot` in `use` (`retain-on-failure` and
+`only-on-failure` keep uploads small), and anything you `testInfo.attach()` is uploaded too. Each file is stored once
+by its content, in chunks, so a server's object size limit does not cap a video.
+
+### playwright-bdd
+
+Tests generated by [playwright-bdd](https://github.com/vitalets/playwright-bdd) belong to their `.feature` file: a
+test in `.features-gen/<path>.feature.spec.js` gets the `specPath` of `<path>.feature`, found beside `.features-gen`
+(a per-project output folder in between is skipped). The scenario's title is the check's title (an outline's example
+is `<outline>: Example #1`), feature tags are tags, `Given` / `When` / `Then` steps are timeline steps of that kind,
+and the feature's name is the spec's title in `specs[]`. If your features are elsewhere, give `specPath`:
+
+```ts
+['@crvouga/atlas/playwright-reporter', { specPath: (test) => test.location.file.replace(/^.*\/generated\//, 'features/').replace(/\.spec\.js$/, '') }]
+```
+
+## Any other producer
+
+`createReporter` streams checks as they finish, one layer per flush, to a server or any store
+([streaming a run](server.md#streaming-a-run-as-it-happens)); `publishChecks` publishes one finished layer. Both take
+the same check. Attachment bytes go in as `data` (a `Uint8Array`); read files yourself (only the command line reads a
+`path`). Here is a check that carries everything:
+
+```ts
+import { readFile } from 'node:fs/promises';
+import { createReporter } from '@crvouga/atlas/reports/reporter';
+import { httpStore } from '@crvouga/atlas/reports/http-store';
+
+const reporter = createReporter(
+  {
+    producer: 'e2e-web',
+    environment: 'mobile-web',
+    sha: process.env.GITHUB_SHA ?? null,
+    branch: process.env.GITHUB_HEAD_REF || process.env.GITHUB_REF_NAME || null,
+    runUrl: `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`,
+    // Names and groups each spec, so checks are grouped by spec without a published spec set.
+    specs: [{ path: 'features/checkout/pay.feature', domain: 'checkout', kind: 'feature', title: 'Paying for an order' }]
+  },
+  { store: httpStore({ url: 'https://atlas.example.com', project: 'shop', token: process.env.ATLAS_TOKEN }) }
+);
+
+reporter.check({ id: 'features/checkout/pay.feature#Pays by card', title: 'Pays by card', status: 'running', specPath: 'features/checkout/pay.feature' });
+
+reporter.check({
+  id: 'features/checkout/pay.feature#Pays by card',
+  title: 'Pays by card',
+  status: 'passed',
+  detail: 'Passed on retry 1 (flaky)',
+  specPath: 'features/checkout/pay.feature',
+  durationMs: 4180,
+  tags: ['smoke', 'flaky'],
+  labels: { browser: 'chromium', device: 'Pixel 7', os: 'linux', viewport: '412x839' },
+  steps: [
+    { id: 's1', name: 'Given a cart with one item', kind: 'given', status: 'passed', offsetMs: 0, durationMs: 900 },
+    { id: 's2', name: 'When I pay by card', kind: 'when', status: 'passed', offsetMs: 900, durationMs: 2600 },
+    { id: 's3', parentId: 's2', name: 'click #pay', kind: 'action', status: 'passed', offsetMs: 1200, durationMs: 80 },
+    { id: 's4', name: 'Then I see the receipt', kind: 'then', status: 'passed', offsetMs: 3500, durationMs: 680 }
+  ],
+  attempts: [
+    { index: 0, status: 'failed', durationMs: 30000, startedAtMs: Date.now() - 40000, detail: 'Timeout 30000ms exceeded',
+      attachments: [{ name: 'retry-0.png', role: 'screenshot', contentType: 'image/png', data: await readFile('test-results/retry-0.png') }] },
+    { index: 1, status: 'passed', durationMs: 4180, startedAtMs: Date.now() - 5000 }
+  ],
+  attachments: [
+    { name: 'video.webm', role: 'video', contentType: 'video/webm', data: await readFile('test-results/video.webm') },
+    { name: 'trace.zip', role: 'trace', contentType: 'application/zip', data: await readFile('test-results/trace.zip') }
+  ]
+});
+
+const { checks, layers, lost } = await reporter.close(); // never throws; `lost` counts what never arrived
+```
+
+With `publishChecks`, pass the same identity and checks with `recordedAtMs`, once the run is over. From a checks file
+or the command line, an attachment may name a `path` instead; `atlas report publish` reads it
+([attach evidence](extending.md#attach-evidence)). A CTRF report already carries most of this, and
+`atlas report publish --ctrf` keeps it ([native runner reporters](runners.md) cover CodeceptJS, Mocha, Jest and Node's
+test runner).
+</doc>
+
 <doc path="docs/server.md">
 # The Atlas server
 
@@ -1195,6 +1398,39 @@ report the server would have sent whole (`applyEffectiveDelta`, tested against e
 runs); a version the server does not remember, or an older server, answers with the whole report, which a client
 takes as it always did. A result streamed into a 100 000-check report costs a client a few kilobytes.
 
+## Reading the report as an agent
+
+The effective report says what each check's result is now. To explore a project, or to answer "what is tested,
+where, how long does it take, what is flaky", read the tests instead. Every endpoint here answers JSON, or Markdown
+with `format=markdown` (or `Accept: text/markdown`), and takes the project prefix like every other
+(`/api/projects/<slug>/reports/tests`).
+
+| Ask | Endpoint |
+| --- | --- |
+| Every test, a summary (by status, environment, producer, domain), insights (failing, flaky, slowest) and facets | `GET /api/reports/tests` |
+| Only some: `q` (all words), `status` (or `flaky`), `environment`, `producer`, `domain`, `spec`, `tag`, `run`; `sort`, `offset`, `limit` | `GET /api/reports/tests?status=failed&environment=ios` |
+| One test: every recorded result (status, duration, environment, branch, commit, CI run), newest first | `GET /api/reports/tests/history?key=<key>` |
+| The runs that recorded the report (by CI run URL, else branch and commit) | `GET /api/reports/runs` |
+| Every test one run recorded | `GET /api/reports/runs/detail?key=<run key>` |
+
+```sh
+curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdown&limit=50"
+```
+
+- A **test** is a check identity, its spec path and id, across producers and environments. `outcomes` holds its
+  current result on each environment (the report's winners, so a test's status here is always the gate's);
+  `stats` describes its recorded history: pass rate, flips, whether it is flaky and why (it passed and failed on the
+  same commit, passed only after a retry, or its producer said so), durations (avg, p50, p95, min, max, last), runs
+  and branches.
+- The server keeps the newest 100 results of every test (progress reported as `running` is not history) and every run's last result per test, in memory, as layers
+  arrive; it is rebuilt from the layers on restart. History never changes a result.
+- Without a published spec set, tests are grouped under the specs their layers recorded (`specs[]`: path, domain,
+  title), and the gate and the outline do the same, so a project needs no spec set to be browsable. A path no layer
+  described is titled from its file name and grouped by its folder under `specs/`.
+- Statistics are only as rich as what producers send: durations need `durationMs`, devices need an environment and
+  labels (`browser`, `device`, `os`), videos and screenshots need attachments. See
+  [Send rich evidence](rich-evidence.md).
+
 ## API reference
 
 <!-- generated:server-operations -->
@@ -1235,6 +1471,14 @@ takes as it always did. A result streamed into a 100 000-check report costs a cl
 | `PUT /api/projects/{project}/store/{key}` | publish | Upload an object. |
 | `DELETE /api/store/{key}` | publish | Remove a scratch object. |
 | `DELETE /api/projects/{project}/store/{key}` | publish | Remove a scratch object. |
+| `GET /api/reports/tests` | read | Every test, with insights, facets and history statistics. |
+| `GET /api/projects/{project}/reports/tests` | read | Every test, with insights, facets and history statistics. |
+| `GET /api/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/projects/{project}/reports/tests/history` | read | One test and every result recorded for it. |
+| `GET /api/reports/runs` | read | The runs that recorded the report. |
+| `GET /api/projects/{project}/reports/runs` | read | The runs that recorded the report. |
+| `GET /api/reports/runs/detail` | read | Every test one run recorded. |
+| `GET /api/projects/{project}/reports/runs/detail` | read | Every test one run recorded. |
 | `GET /api/reports/outline` | read | The specs a report is laid over. |
 | `GET /api/projects/{project}/reports/outline` | read | The specs a report is laid over. |
 | `GET /api/reports/outline/spec` | read | The text of one spec. |
@@ -4201,11 +4445,143 @@ Scoped to one project. The same project may be selected on the unprefixed API us
 
 Responses: `204` Removed (or it was not there).; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `403` Not a scratch object, or the token may not.
 
+### Every test, with insights, facets and history statistics
+
+`GET /api/reports/tests` · role: read
+
+The report as a list of tests, for exploring it or for an agent to read in one request. A test is a check identity (its spec path and id) across producers and environments; `outcomes` holds its current result on each environment, and `stats` how it went over its recorded history (pass rate, flips, flaky, durations avg/p50/p95, runs, branches).
+
+`summary` and `facets` cover every test (counts by status, by environment, by producer and by domain, the slowest, flakiest and failing tests); `tests` is the page the query matched. Without a published spec set, specs come from the `specs[]` the layers recorded.
+
+Send `format=markdown` (or `Accept: text/markdown`) for the same as a Markdown report. The current status of a test is always the report's winner; history never changes it.
+
+- `q` (query): Words that must all appear in the title, id, spec, domain, tags, labels, environments or producers. Case does not matter.
+- `status` (query): Current status, or `flaky`. Repeatable or comma-separated.
+- `environment` (query): Only tests with a result on this environment. Repeatable or comma-separated.
+- `producer` (query): Only tests from this producer. Repeatable or comma-separated.
+- `domain` (query): Only tests whose spec is in this domain (`(no spec)` for none). Repeatable or comma-separated.
+- `spec` (query): Only tests of this spec path. Repeatable or comma-separated.
+- `tag` (query): Only tests with this tag. Repeatable or comma-separated.
+- `run` (query): Only tests recorded in this run (a `key` from `/api/reports/runs`).
+- `sort` (query): The order: `spec` (default: domain, spec, title), `title`, `recent`, `slowest`, `failures`, `flaky` or `runs`.
+- `offset` (query): Skip this many matching tests.
+- `limit` (query): At most this many tests (default 500, at most 20000).
+- `format` (query): `markdown` for a Markdown report instead of JSON.
+- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+
+Responses: `200` The catalog.; `304` Unchanged since the `If-None-Match` ETag.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### Every test, with insights, facets and history statistics
+
+`GET /api/projects/{project}/reports/tests` · role: read
+
+The report as a list of tests, for exploring it or for an agent to read in one request. A test is a check identity (its spec path and id) across producers and environments; `outcomes` holds its current result on each environment, and `stats` how it went over its recorded history (pass rate, flips, flaky, durations avg/p50/p95, runs, branches).
+
+`summary` and `facets` cover every test (counts by status, by environment, by producer and by domain, the slowest, flakiest and failing tests); `tests` is the page the query matched. Without a published spec set, specs come from the `specs[]` the layers recorded.
+
+Send `format=markdown` (or `Accept: text/markdown`) for the same as a Markdown report. The current status of a test is always the report's winner; history never changes it.
+
+Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
+
+- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+- `q` (query): Words that must all appear in the title, id, spec, domain, tags, labels, environments or producers. Case does not matter.
+- `status` (query): Current status, or `flaky`. Repeatable or comma-separated.
+- `environment` (query): Only tests with a result on this environment. Repeatable or comma-separated.
+- `producer` (query): Only tests from this producer. Repeatable or comma-separated.
+- `domain` (query): Only tests whose spec is in this domain (`(no spec)` for none). Repeatable or comma-separated.
+- `spec` (query): Only tests of this spec path. Repeatable or comma-separated.
+- `tag` (query): Only tests with this tag. Repeatable or comma-separated.
+- `run` (query): Only tests recorded in this run (a `key` from `/api/reports/runs`).
+- `sort` (query): The order: `spec` (default: domain, spec, title), `title`, `recent`, `slowest`, `failures`, `flaky` or `runs`.
+- `offset` (query): Skip this many matching tests.
+- `limit` (query): At most this many tests (default 500, at most 20000).
+- `format` (query): `markdown` for a Markdown report instead of JSON.
+
+Responses: `200` The catalog.; `304` Unchanged since the `If-None-Match` ETag.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### One test and every result recorded for it
+
+`GET /api/reports/tests/history` · role: read
+
+The test (as `/api/reports/tests` lists it) and up to 100 recorded results, newest first: status, duration, environment, producer, branch, commit, CI run, layer and the first 500 characters of the detail. Open a result's evidence (attachments, steps, retries) with `/api/reports/evidence/detail/{backend}/{id}?check=<id>`. Send `format=markdown` for Markdown.
+
+- `key` (query, required): The test's `key` from `/api/reports/tests`: `JSON.stringify([specPath, id])`, URL-encoded.
+- `format` (query): `markdown` for Markdown instead of JSON.
+- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+
+Responses: `200` The test and its history.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No test in the report has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### One test and every result recorded for it
+
+`GET /api/projects/{project}/reports/tests/history` · role: read
+
+The test (as `/api/reports/tests` lists it) and up to 100 recorded results, newest first: status, duration, environment, producer, branch, commit, CI run, layer and the first 500 characters of the detail. Open a result's evidence (attachments, steps, retries) with `/api/reports/evidence/detail/{backend}/{id}?check=<id>`. Send `format=markdown` for Markdown.
+
+Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
+
+- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+- `key` (query, required): The test's `key` from `/api/reports/tests`: `JSON.stringify([specPath, id])`, URL-encoded.
+- `format` (query): `markdown` for Markdown instead of JSON.
+
+Responses: `200` The test and its history.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No test in the report has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### The runs that recorded the report
+
+`GET /api/reports/runs` · role: read
+
+Every run, newest first: layers grouped by their CI run URL (else branch and commit), with their environments, producers, how many tests they recorded, their results by status, the time they spanned and up to twenty failures. Send `format=markdown` for Markdown.
+
+- `limit` (query): At most this many runs (default 100).
+- `format` (query): `markdown` for Markdown instead of JSON.
+- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+
+Responses: `200` The runs.; `304` Unchanged since the `If-None-Match` ETag.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### The runs that recorded the report
+
+`GET /api/projects/{project}/reports/runs` · role: read
+
+Every run, newest first: layers grouped by their CI run URL (else branch and commit), with their environments, producers, how many tests they recorded, their results by status, the time they spanned and up to twenty failures. Send `format=markdown` for Markdown.
+
+Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
+
+- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+- `limit` (query): At most this many runs (default 100).
+- `format` (query): `markdown` for Markdown instead of JSON.
+
+Responses: `200` The runs.; `304` Unchanged since the `If-None-Match` ETag.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### Every test one run recorded
+
+`GET /api/reports/runs/detail` · role: read
+
+One run and its last result for every test, environment and producer it recorded, failures first. Send `format=markdown` for Markdown.
+
+- `key` (query, required): The run's `key` from `/api/reports/runs`, URL-encoded.
+- `format` (query): `markdown` for Markdown instead of JSON.
+- `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+
+Responses: `200` The run and its results.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No run has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
+### Every test one run recorded
+
+`GET /api/projects/{project}/reports/runs/detail` · role: read
+
+One run and its last result for every test, environment and producer it recorded, failures first. Send `format=markdown` for Markdown.
+
+Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
+
+- `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
+- `key` (query, required): The run's `key` from `/api/reports/runs`, URL-encoded.
+- `format` (query): `markdown` for Markdown instead of JSON.
+
+Responses: `200` The run and its results.; `304` Unchanged since the `If-None-Match` ETag.; `400` No `key`.; `401` No token, or an unknown one. Send `Authorization: Bearer <token>`.; `404` No run has that key.; `503` The server is still reading its backends. Retry after the number of seconds in `Retry-After`.
+
 ### The specs a report is laid over
 
 `GET /api/reports/outline` · role: read
 
-The specs of the newest committed spec set (of `main`, else the newest), so the report needs no checkout. A spec here with no result shows as having none.
+The specs of the newest committed spec set (of `main`, else the newest), so the report needs no checkout. Without a spec set, the specs the layers recorded (`specs[]`), newest per path. A spec here with no result shows as having none.
 
 - `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
 
@@ -4215,7 +4591,7 @@ Responses: `200` The outline.; `401` No token, or an unknown one. Send `Authoriz
 
 `GET /api/projects/{project}/reports/outline` · role: read
 
-The specs of the newest committed spec set (of `main`, else the newest), so the report needs no checkout. A spec here with no result shows as having none.
+The specs of the newest committed spec set (of `main`, else the newest), so the report needs no checkout. Without a spec set, the specs the layers recorded (`specs[]`), newest per path. A spec here with no result shows as having none.
 
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 

package/llms.txt

@@ -10,6 +10,7 @@ Given only the repository link, start at docs/agents.md: install `@crvouga/atlas
 - [Verify Atlas locally](https://raw.githubusercontent.com/crvouga/atlas/main/docs/quickstart.md): Prove that the installed Atlas can publish, combine and gate evidence before connecting your project's real suites.
 - [Integrating Atlas into a project](https://raw.githubusercontent.com/crvouga/atlas/main/docs/integrating.md): This guide takes a project from nothing to a layered, gated report of its specs.
 - [Native test runner reporters](https://raw.githubusercontent.com/crvouga/atlas/main/docs/runners.md): Projects upload results to a shared Atlas node under their own project slug.
+- [Send rich evidence](https://raw.githubusercontent.com/crvouga/atlas/main/docs/rich-evidence.md): A check that says only its title and passed answers one question:
 
 ## Run and use the server
 
@@ -43,6 +44,7 @@ Given only the repository link, start at docs/agents.md: install `@crvouga/atlas
 - [TypeScript API: detox](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/detox.md): The Detox driver for statechart runs of React Native apps.
 - [TypeScript API: runners](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/runners.md): Native reporters that upload to the project server.
 - [TypeScript API: vitest](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/vitest.md): Run a statechart plan as Vitest tests.
+- [TypeScript API: playwright-reporter](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/playwright-reporter.md): A Playwright Test reporter that streams every test to Atlas with its duration, device labels, steps, retries, videos, screenshots and traces.
 - [JSON Schemas](https://unpkg.com/@crvouga/atlas/packages/schema/schemas/): the layer format, checks file, report status, command results, server responses, server configuration.
 
 ## Optional

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/atlas",
-  "version": "0.9.0",
+  "version": "0.10.1",
   "description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
   "license": "MIT",
   "type": "module",
@@ -111,6 +111,10 @@
     "./node-test": {
       "types": "./dist/runners/src/node-test.d.ts",
       "default": "./dist/runners/src/node-test.js"
+    },
+    "./playwright-reporter": {
+      "types": "./dist/playwright-reporter/src/index.d.ts",
+      "default": "./dist/playwright-reporter/src/index.js"
     }
   },
   "publishConfig": {
@@ -141,6 +145,7 @@
     "docs/deploying.md",
     "docs/server.md",
     "docs/extending.md",
+    "docs/rich-evidence.md",
     "docs/statecharts.md",
     "docs/report-contract.md",
     "docs/report-backends.md",
@@ -202,10 +207,10 @@
     "react-dom": "^19.3.0",
     "solid-js": "^1.9.15",
     "tsx": "4.23.13",
-    "vite": "7.3.6",
+    "vite": "7.3.7",
     "xstate": "5.33.2",
     "yaml": "^2.9.0",
-    "zod": "^3.25.76",
+    "zod": "^4.6.7",
     "zustand": "^5.0.15"
   },
   "peerDependencies": {

package/packages/schema/schemas/check-detail.schema.json

@@ -13,9 +13,9 @@
       "type": "object",
       "properties": {
         "id": {
-          "description": "The check's stable identity within its producer. The same check must keep the same id in every layer, or layers cannot replace one another",
           "type": "string",
-          "minLength": 1
+          "minLength": 1,
+          "description": "The check's stable identity within its producer. The same check must keep the same id in every layer, or layers cannot replace one another"
         },
         "title": {
           "type": "string"
@@ -30,8 +30,8 @@
           ]
         },
         "detail": {
-          "description": "Why it failed, or what was measured",
           "default": "",
+          "description": "Why it failed, or what was measured",
           "type": "string"
         },
         "specPath": {
@@ -49,8 +49,8 @@
                 "minLength": 1
               },
               "value": {
-                "description": "A finite number",
-                "type": "number"
+                "type": "number",
+                "description": "A finite number"
               },
               "unit": {
                 "type": "string"
@@ -215,9 +215,9 @@
             "type": "object",
             "properties": {
               "type": {
-                "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…",
                 "type": "string",
-                "minLength": 1
+                "minLength": 1,
+                "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…"
               },
               "message": {
                 "type": "string"
@@ -443,9 +443,9 @@
                   "type": "object",
                   "properties": {
                     "type": {
-                      "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…",
                       "type": "string",
-                      "minLength": 1
+                      "minLength": 1,
+                      "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…"
                     },
                     "message": {
                       "type": "string"
@@ -708,9 +708,9 @@
                         "type": "object",
                         "properties": {
                           "type": {
-                            "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…",
                             "type": "string",
-                            "minLength": 1
+                            "minLength": 1,
+                            "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…"
                           },
                           "message": {
                             "type": "string"
@@ -908,9 +908,9 @@
                   "type": "object",
                   "properties": {
                     "type": {
-                      "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…",
                       "type": "string",
-                      "minLength": 1
+                      "minLength": 1,
+                      "description": "An open vocabulary: note, warning, deprecation, issue, accessibility, coverage…"
                     },
                     "message": {
                       "type": "string"
@@ -1016,10 +1016,10 @@
       "additionalProperties": {}
     },
     "run": {
-      "description": "What the run as a whole carries: its logs, links, labels, annotations and extensions",
       "type": "object",
       "properties": {},
-      "additionalProperties": {}
+      "additionalProperties": {},
+      "description": "What the run as a whole carries: its logs, links, labels, annotations and extensions"
     }
   },
   "required": [

package/packages/schema/schemas/checks-result.schema.json

@@ -8,8 +8,8 @@
       "const": true
     },
     "id": {
-      "description": "The layer id",
-      "type": "string"
+      "type": "string",
+      "description": "The layer id"
     },
     "checks": {
       "type": "integer",

package/packages/schema/schemas/command-error.schema.json

@@ -11,13 +11,13 @@
       "type": "object",
       "properties": {
         "message": {
-          "description": "What is wrong, and usually what to do about it",
-          "type": "string"
+          "type": "string",
+          "description": "What is wrong, and usually what to do about it"
         },
         "code": {
-          "description": "`usage`: the command or its inputs were wrong. Nothing was written",
           "type": "string",
-          "const": "usage"
+          "const": "usage",
+          "description": "`usage`: the command or its inputs were wrong. Nothing was written"
         }
       },
       "required": [

package/packages/schema/schemas/doctor-result.schema.json

@@ -4,8 +4,8 @@
   "type": "object",
   "properties": {
     "ok": {
-      "description": "True when every check of every store passed (otherwise the exit code is 1)",
-      "type": "boolean"
+      "type": "boolean",
+      "description": "True when every check of every store passed (otherwise the exit code is 1)"
     },
     "stores": {
       "type": "array",
@@ -21,8 +21,8 @@
               "type": "object",
               "properties": {
                 "name": {
-                  "description": "`store`, `list`, `contents`, `write`, …",
-                  "type": "string"
+                  "type": "string",
+                  "description": "`store`, `list`, `contents`, `write`, …"
                 },
                 "ok": {
                   "type": "boolean"