@crvouga/atlas 1.12.0 → 1.13.0

package/docs/reference/http-api.md

@@ -414,7 +414,7 @@ Responses: `200` It exists.; `401` No token, or an unknown one. Send `Authorizat
 
 ### Upload an object
 
-`PUT /api/store/{key}` · role: publish
+`PUT /api/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -426,6 +426,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 - `key` (path, required): An object key under `layers/`, such as `layers/manifests/<id>.json`. It contains slashes, which are not percent-encoded; every other character is.
 - `content-type` (header): Stored with the object and returned when it is read.
 - `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
@@ -434,7 +436,7 @@ Responses: `200` Already there with the same bytes.; `201` Stored.; `401` No tok
 
 ### Upload an object
 
-`PUT /api/projects/{project}/store/{key}` · role: publish
+`PUT /api/projects/{project}/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -446,6 +448,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 
 - `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.

package/docs/reference/reports.md

@@ -1366,10 +1366,10 @@ A report: the results laid over the specs, with what was read and what could not
 #### `countResults`
 
 ```ts
-export function countResults(results: readonly EvidenceResult[]): StatusCounts & { total: number }
+export function countResults(all: readonly EvidenceResult[]): StatusCounts & { total: number }
 ```
 
-Counts results by status. Diagnostic results (parts of kind `performance`, which only qualify a check) are left out
+Counts results by status. Opinion results (parts of kind `critique`) are never counted. Diagnostic results (parts of kind `performance`, which only qualify a check) are left out
 whenever any other result is present; if only diagnostics exist, they are counted so the report is not empty.
 
 #### `summarize`
@@ -2371,7 +2371,8 @@ export function checkSlots(results: readonly EvidenceResult[]): CheckSlot[]
 
 Groups the combined results (the winners of the slots) by check, in the order the checks were first seen.
 A check's status is the worst of its non-diagnostic results; if it has only diagnostics (timings), those are its
-outcomes. The title and id shown are the newest result's.
+outcomes. The title and id shown are the newest result's. Opinions (parts of kind `critique`, see `isOpinionKind`) are
+not checks and are left out.
 
 #### `SpecStatus`
 
@@ -4582,6 +4583,17 @@ export function isDiagnosticKind(kind: string): boolean
 
 Whether a part `kind` only qualifies an outcome (today only `performance`) and so is not counted as a check when anything else is recorded.
 
+#### `isOpinionKind`
+
+```ts
+export function isOpinionKind(kind: string): boolean
+```
+
+Whether a part `kind` records an opinion rather than an outcome (today only `critique`, an automated reviewer's
+judgement of a captured UI). An opinion is never counted as a check, never gates, and is not folded into a spec's
+rollup, even when it is the only thing recorded: a report of opinions alone has no test results. The check's `status`
+of such a part says whether the reviewer ran, not whether anything passed.
+
 #### `emptyCounts`
 
 ```ts
@@ -4593,11 +4605,12 @@ A fresh status tally with every status at zero.
 #### `outcomeCounts`
 
 ```ts
-export function outcomeCounts(rows: readonly LayerRow[]): StatusCounts
+export function outcomeCounts(allRows: readonly LayerRow[]): StatusCounts
 ```
 
-Counts rows by status, leaving out the diagnostic rows (see `isDiagnosticKind`) that only qualify an outcome.
-When every row is diagnostic they are counted, so a layer of timings alone is not empty.
+Counts rows by status, leaving out the opinion rows (see `isOpinionKind`) always, and the diagnostic rows (see
+`isDiagnosticKind`) that only qualify an outcome. When every remaining row is diagnostic they are counted, so a
+layer of timings alone is not empty.
 
 #### `layerEvidence`
 
@@ -6981,7 +6994,7 @@ Returns a `ReportStatus` (schemaVersion 1): `ok` is true exactly when `reasons`
 checks (or none completed), failed checks, running checks, specs without results (only with `requireAllSpecs`),
 each required environment with no checks (the name is normalised, so `iOS` means `ios`), then every notice of the
 report (an unreadable layer or backend). `failing` lists each failed or running winner, in a fixed order that does
-not depend on read order; diagnostic `performance` checks are left out unless nothing else was recorded.
+not depend on read order; diagnostic `performance` checks are left out unless nothing else was recorded, and `critique` opinions are always left out.
 `recordings` reports recording completeness of the same results apart from `ok`: an incomplete or unknown
 completeness never changes the gate (use `atlas report publish --require-recordings` to fail on a missing video).
 

package/docs/reference/schema.md

@@ -775,12 +775,128 @@ export function transitionId(source: string, event: string)
 Builds the id of a transition, the key of `transitions` in a manifest: the source state's name,
 `TRANSITION_ID_SEPARATOR`, then the event name.
 
+### The `atlas.critique` evidence schema
+
+Source: `packages/schema/src/critique.ts`
+
+What an automated reviewer (a critic) says about one captured UI. It is not a new layer format: a critique is a
+check inside a part of kind `critique`, and the opinion rides in the check's `extensions["atlas.critique"]`, so by the
+rules of docs/extending.md it can never decide a result. The check's `status` says only whether the critic ran
+(`passed` a verdict was produced, `failed` the critic could not produce one, `running`, `not-run`).
+
+Everything except `schema`, `subject` and `critic` is optional, so a single score (a model-based critic) and rich
+findings (an LLM critic) are one shape. Every object is open, and `kind`, `category`, `severity`, `label` and `reason`
+are open strings: a value nobody has heard of is kept and shown generically, never rejected. A reader that cannot read
+the whole extension drops it and keeps the check and its status (`readCritique` in `@crvouga/atlas/critics`).
+
+See docs/critics.md
+See docs/extending.md
+
+#### `CRITIQUE_EXTENSION`
+
+```ts
+export const CRITIQUE_EXTENSION = …
+```
+
+The key under which a critique is stored in a check's `extensions`.
+
+#### `CRITIQUE_SCHEMA_VERSION`
+
+```ts
+export const CRITIQUE_SCHEMA_VERSION = …
+```
+
+The version of the `atlas.critique` extension a reader of this release understands. A critique of another version is unknown, not wrong.
+
+#### `CRITIQUE_PART_KIND`
+
+```ts
+export const CRITIQUE_PART_KIND = …
+```
+
+The `kind` of the report part that holds critiques. Parts of this kind are opinions: never counted, never part of the gate.
+
+#### `CRITIC_PRODUCER_PREFIX`
+
+```ts
+export const CRITIC_PRODUCER_PREFIX = …
+```
+
+The prefix of the `producer` of a layer of critiques: `critic.<critic name>`.
+
+#### `CritiqueRegionSchema`
+
+```ts
+export const CritiqueRegionSchema = …
+```
+
+The region of the image a finding is about, in the stated unit (`px` of the stored image, or `normalized` 0..1).
+
+#### `CritiqueFindingSchema`
+
+```ts
+export const CritiqueFindingSchema = …
+```
+
+One thing a critic found: where, what kind, how bad, why, and what to do. Nothing is required.
+
+#### `CritiqueRelativeSchema`
+
+```ts
+export const CritiqueRelativeSchema = …
+```
+
+How a score reads in context. A raw score is not absolute: a loading overlay is compared with other loading overlays,
+not with full screens, and a change against the same entry's previous image is more telling than the level.
+
+#### `CritiqueVerdictSchema`
+
+```ts
+export const CritiqueVerdictSchema = …
+```
+
+The judgement: a score and how to read it, a label, a confidence. A critic that cannot score may give only a label.
+
+#### `CritiqueFailureSchema`
+
+```ts
+export const CritiqueFailureSchema = …
+```
+
+Why a critic produced no verdict: a reason code in an open vocabulary (`error`, `timeout`, `rate-limited`, `invalid-output`) and a message that holds no credential.
+
+#### `CritiqueSchema`
+
+```ts
+export const CritiqueSchema = …
+```
+
+The `atlas.critique` extension, version 1: the image judged (`subject`), who judged (`critic`, `model`,
+`parameters`), the opinion (`verdict`, `findings`) or why there is none (`failure`), and what it cost (`usage`).
+The prompt is recorded only as a hash (`parameters.rubric`). Emitted as `critique.schema.json`.
+
+#### `Critique`
+
+```ts
+export type Critique = z.output<typeof CritiqueSchema>;
+```
+
+A critique as stored in a check's `extensions["atlas.critique"]`.
+
+#### `CritiqueFinding`
+
+```ts
+export type CritiqueFinding = z.output<typeof CritiqueFindingSchema>;
+```
+
+One finding of a critique.
+
 ### Schema package entry point (`@crvouga/atlas/schema`)
 
 Source: `packages/schema/src/index.ts`. Import: `@crvouga/atlas/schema`
 
 Re-exports every Zod schema, constant and inferred type of the Atlas contract: the layer format and report status
-(`layer`), the statechart run manifest (`manifest`, `common`), the spec file formats (`spec`), the run index and
+(`layer`), the `atlas.critique` evidence of automated UI reviewers (`critique`), the statechart run manifest (`manifest`, `common`), the spec file formats (`spec`), the run index and
 report sources the viewer reads (`runs-index`, `report-sources`) and the server configuration (`server-config`).
 The JSON Schemas shipped in `schemas/` are generated from these by `bun run schemas`.
 
@@ -1869,7 +1985,8 @@ See docs/server.md
 export const ServerRoleSchema = …
 ```
 
-What a bearer token may do: `read` (look), `publish` (also upload) or `admin` (also remove scratch objects).
+What a bearer token may do: `read` (look), `critic` (also publish critiques, and nothing else), `publish` (also upload
+anything) or `admin` (also remove scratch objects).
 
 #### `ServerRole`
 

package/docs/reference/server.md

@@ -93,7 +93,7 @@ What it knows lives in its backends (by default a folder on its own disk), so re
 
 Source: `packages/server/src/auth.ts`
 
-Who is asking, and may they. Tokens have a role (`read` < `publish` < `admin`) and are compared in constant time. Reading
+Who is asking, and may they. Tokens have a role (`read` < `critic` < `publish` < `admin`) and are compared in constant time. Reading
 accepts a Bearer token or the browser's sign-in box (any user name, the token as the password); changing anything needs
 the Bearer header, because a browser sends the sign-in box's credentials to any page that asks, which would let another
 site upload with them. A server with no tokens is open, and refuses to listen beyond this machine unless told it may.
@@ -530,10 +530,10 @@ The version of the API described, `major.minor.patch`: the minor grows when an o
 #### `Role`
 
 ```ts
-export type Role = 'none' | 'read' | 'publish' | 'admin';
+export type Role = 'none' | 'read' | 'critic' | 'publish' | 'admin';
 ```
 
-Who may call an operation: `none` (anyone), or the least role that may (`read` < `publish` < `admin`).
+Who may call an operation: `none` (anyone), or the least role that may (`read` < `critic` < `publish` < `admin`).
 
 #### `Operation`
 

package/docs/report-contract.md

@@ -230,6 +230,10 @@ An Atlas path that passed on retry (`flaky`) is `passed`, with the detail "Passe
 Timings that only qualify a check (parts of kind `performance`) are not counted as checks when
 anything else is recorded. A measurement over its allowance is reported and never fails a check.
 
+Opinions (parts of kind `critique`, the verdicts of [automated UI reviewers](critics.md)) are never counted as checks,
+even when they are all that was recorded, and are never part of the gate: a low score, or a critic that crashed, leaves
+`atlas report status` as it was. The `status` of a critique check says only whether the critic ran.
+
 **Freshness never changes a status.** A layer whose spec fingerprint no longer matches, whose
 checkout was modified, or whose related code changed still shows its recorded outcome, with a
 warning ("Potentially outdated: spec changed"). Atlas favours quick, possibly stale evidence over

package/docs/server.md

@@ -205,7 +205,7 @@ is said in words and the server does not start. The file has a published
 | How often (seconds) the backends are looked at for layers written by something else | `refreshSeconds` | `--refresh <seconds>` | `ATLAS_REFRESH_SECONDS` | `10` |
 | Let anyone look, not upload | `auth.anonymousRead` | `--anonymous-read` | `ATLAS_ANONYMOUS_READ` | no |
 | Listen publicly with no tokens (something in front signs people in) | `auth.insecure` | `--insecure` | `ATLAS_INSECURE` | no |
-| The admin token (also `ATLAS_PUBLISH_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
+| The admin token (also `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
 | The configuration file | `(the file itself)` | `--config <file>` | `ATLAS_CONFIG` | `./atlas.server.json` when it exists |
 | Largest uploaded object in MiB | `limits.maxObjectMiB` |  | `ATLAS_MAX_OBJECT_MIB` | `256` |
 | Largest JSON upload in MiB | `limits.maxJsonMiB` |  | `ATLAS_MAX_JSON_MIB` | `32` |
@@ -238,16 +238,18 @@ backend in the client and `/readyz`, and recovers by itself.
 | Role | May |
 | --- | --- |
 | `read` | look: the client, the report, `/api/status` |
+| `critic` | also publish [critiques](critics.md#authentication-and-authorization) and nothing else: blobs and layers whose producer is `critic.<name>` and whose parts are all of kind `critique`. It cannot post test results, start live runs or change settings |
 | `publish` | also upload layers, specs and runs, and remove the scratch objects `atlas report doctor` writes |
 | `admin` | all server operations |
 
 Tokens are at least 12 characters (`openssl rand -hex 24`) and different from one another. Give them as
-`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
+`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
 with `tokenEnv` so the file holds no secret.
 
 - **Sending a token:** `Authorization: Bearer <token>`. A browser's sign-in box also works for *looking* (any
   user name, the token as the password). **Uploading needs the Bearer header, never the sign-in box**: a browser
   sends those credentials by itself to any page that asks, which would let another site upload with them.
+- **Run critics against a server with tokens.** With no tokens every caller is an administrator, a critic included.
 - **No tokens** means open, and then the server refuses to listen beyond this machine unless told
   `--insecure` (for when a proxy in front does the signing in).
 - `/healthz` and `/readyz` never need a token, so a load balancer can use them.
@@ -490,8 +492,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/store/{key}` | read | Read an object. |
 | `HEAD /api/store/{key}` | read | Does an object exist. |
 | `HEAD /api/projects/{project}/store/{key}` | read | Does an object exist. |
-| `PUT /api/store/{key}` | publish | Upload an object. |
-| `PUT /api/projects/{project}/store/{key}` | publish | Upload an object. |
+| `PUT /api/store/{key}` | critic | Upload an object. |
+| `PUT /api/projects/{project}/store/{key}` | critic | Upload an object. |
 | `DELETE /api/store/{key}` | publish | Remove a scratch object. |
 | `DELETE /api/projects/{project}/store/{key}` | publish | Remove a scratch object. |
 | `GET /api/reports/tests` | read | Every test, with insights, facets and history statistics. |

package/llms-full.txt

@@ -112,6 +112,7 @@ The complete public export map is generated from the package manifest (wildcards
 @crvouga/atlas/vitest
 @crvouga/atlas/schema
 @crvouga/atlas/server
+@crvouga/atlas/critics
 @crvouga/atlas/reports/*
 @crvouga/atlas/schemas/*
 @crvouga/atlas/openapi.json
@@ -1671,7 +1672,7 @@ is said in words and the server does not start. The file has a published
 | How often (seconds) the backends are looked at for layers written by something else | `refreshSeconds` | `--refresh <seconds>` | `ATLAS_REFRESH_SECONDS` | `10` |
 | Let anyone look, not upload | `auth.anonymousRead` | `--anonymous-read` | `ATLAS_ANONYMOUS_READ` | no |
 | Listen publicly with no tokens (something in front signs people in) | `auth.insecure` | `--insecure` | `ATLAS_INSECURE` | no |
-| The admin token (also `ATLAS_PUBLISH_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
+| The admin token (also `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN` and `ATLAS_READ_TOKEN` for the other roles, or `auth.tokens` in the file) | `auth.tokens` |  | `ATLAS_TOKEN` | none: the server is open and listens on this machine only |
 | The configuration file | `(the file itself)` | `--config <file>` | `ATLAS_CONFIG` | `./atlas.server.json` when it exists |
 | Largest uploaded object in MiB | `limits.maxObjectMiB` |  | `ATLAS_MAX_OBJECT_MIB` | `256` |
 | Largest JSON upload in MiB | `limits.maxJsonMiB` |  | `ATLAS_MAX_JSON_MIB` | `32` |
@@ -1704,16 +1705,18 @@ backend in the client and `/readyz`, and recovers by itself.
 | Role | May |
 | --- | --- |
 | `read` | look: the client, the report, `/api/status` |
+| `critic` | also publish [critiques](critics.md#authentication-and-authorization) and nothing else: blobs and layers whose producer is `critic.<name>` and whose parts are all of kind `critique`. It cannot post test results, start live runs or change settings |
 | `publish` | also upload layers, specs and runs, and remove the scratch objects `atlas report doctor` writes |
 | `admin` | all server operations |
 
 Tokens are at least 12 characters (`openssl rand -hex 24`) and different from one another. Give them as
-`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
+`ATLAS_TOKEN` (admin), `ATLAS_PUBLISH_TOKEN`, `ATLAS_CRITIC_TOKEN`, `ATLAS_READ_TOKEN`, or under `auth.tokens` in the file, preferably
 with `tokenEnv` so the file holds no secret.
 
 - **Sending a token:** `Authorization: Bearer <token>`. A browser's sign-in box also works for *looking* (any
   user name, the token as the password). **Uploading needs the Bearer header, never the sign-in box**: a browser
   sends those credentials by itself to any page that asks, which would let another site upload with them.
+- **Run critics against a server with tokens.** With no tokens every caller is an administrator, a critic included.
 - **No tokens** means open, and then the server refuses to listen beyond this machine unless told
   `--insecure` (for when a proxy in front does the signing in).
 - `/healthz` and `/readyz` never need a token, so a load balancer can use them.
@@ -1956,8 +1959,8 @@ curl "$ATLAS_SERVER_URL/api/projects/$ATLAS_PROJECT/reports/tests?format=markdow
 | `GET /api/projects/{project}/store/{key}` | read | Read an object. |
 | `HEAD /api/store/{key}` | read | Does an object exist. |
 | `HEAD /api/projects/{project}/store/{key}` | read | Does an object exist. |
-| `PUT /api/store/{key}` | publish | Upload an object. |
-| `PUT /api/projects/{project}/store/{key}` | publish | Upload an object. |
+| `PUT /api/store/{key}` | critic | Upload an object. |
+| `PUT /api/projects/{project}/store/{key}` | critic | Upload an object. |
 | `DELETE /api/store/{key}` | publish | Remove a scratch object. |
 | `DELETE /api/projects/{project}/store/{key}` | publish | Remove a scratch object. |
 | `GET /api/reports/tests` | read | Every test, with insights, facets and history statistics. |
@@ -2798,6 +2801,10 @@ An Atlas path that passed on retry (`flaky`) is `passed`, with the detail "Passe
 Timings that only qualify a check (parts of kind `performance`) are not counted as checks when
 anything else is recorded. A measurement over its allowance is reported and never fails a check.
 
+Opinions (parts of kind `critique`, the verdicts of [automated UI reviewers](critics.md)) are never counted as checks,
+even when they are all that was recorded, and are never part of the gate: a low score, or a critic that crashed, leaves
+`atlas report status` as it was. The `status` of a critique check says only whether the critic ran.
+
 **Freshness never changes a status.** A layer whose spec fingerprint no longer matches, whose
 checkout was modified, or whose related code changed still shows its recorded outcome, with a
 warning ("Potentially outdated: spec changed"). Atlas favours quick, possibly stale evidence over
@@ -3224,6 +3231,7 @@ Annotation `type`: `note`, `warning`, `deprecation`, `issue`, `accessibility`, `
 | A kind of file Atlas has never seen | Attach it with its media type and a `role`. It is stored and offered for download today. |
 | Structured data of your own | Put it in `extensions` under a name you own. It is kept, and shown as raw data. |
 | A new annotation type, role or step kind | Use it. Open vocabularies need no registration. |
+| An automated reviewer's opinion of a captured UI | A [critique](critics.md): a part of kind `critique` with the `atlas.critique` extension. It is never counted and never gates. |
 | Something that should be shown well | Ask for a renderer (below). |
 | A field every tool should have | Open an issue. It is added as an optional field, and the rules above guarantee nothing breaks. |
 
@@ -3245,6 +3253,199 @@ listed with its name, role, type and size, and offered for download, and everyth
 unaffected.
 </doc>
 
+<doc path="docs/critics.md">
+# Critics: automated reviewers that mark which UIs need polish
+
+A **critic** looks at one captured UI (an image and what is known about it) and says how polished it is, and ideally
+where and why it is not. A critic may be a pretrained model that gives a score, a vision-capable language model that
+follows a rubric, or a classic measure such as a contrast ratio. All three are the same thing to Atlas, through one small
+contract, so people review the likeliest problems first.
+
+**Critics are producers that publish layers.** Atlas supplies the contract, the runner, the evidence format and the
+credential. It never executes a model on the server and never holds a provider key. A critique is not a test result and
+is not human review: it never changes whether a report passes.
+
+- [What is available](#what-is-available)
+- [The critic contract](#the-critic-contract)
+- [A critique is a layer](#a-critique-is-a-layer)
+- [Running critics](#running-critics)
+- [State-aware judging, calibration and modes](#state-aware-judging-calibration-and-modes)
+- [Authentication and authorization](#authentication-and-authorization)
+- [Credentials](#credentials)
+- [What is not here yet](#what-is-not-here-yet)
+
+## What is available
+
+Import `@crvouga/atlas/critics`. It needs no model, no provider SDK and no network; a critic brings its own.
+
+| You want | Use |
+| --- | --- |
+| Define a critic | `defineCritic` (the contract below) |
+| Judge many images with caching, retries, rate limits, timeouts and budgets | `runCritic` |
+| Store the opinions | `critiqueLayer`, `publishCritiques` (a layer of parts of kind `critique`) |
+| Read them back | `readCritique`, `critiquesOfLayer`, `critiqueCurrency` |
+| Read a score in context, measure a threshold, decide an exit code | `relativeVerdict`, `calibrate`, `gateDecision` |
+| Keep keys out of configuration and logs | `parseCredentialRef`, `resolveCredential`, `assertNoLiteralCredentials`, `createRedactor` |
+| Let a token publish critiques and nothing else | the `critic` role (below) |
+
+The JSON Schema of the evidence is [`critique.schema.json`](../packages/schema/schemas/critique.schema.json). The
+function-level reference is generated from the code: [critics](reference/critics.md).
+
+## The critic contract
+
+```ts
+import { defineCritic } from '@crvouga/atlas/critics';
+
+export default defineCritic({
+  name: 'contrast', version: '1.0.0', kind: 'heuristic',   // kind is an open string: model, llm, ml, heuristic…
+  describe: () => ({ sendsImagesTo: null, needsSetup: false }),
+  evaluate: async ({ image, imageSha256, entry, signal }) => ({
+    verdict: { score: measureContrast(image), scale: '0..1, higher is better' }
+  })
+});
+```
+
+A critic only judges one image. `describe()` says where images are sent (`sendsImagesTo`, `null` when they never leave
+the machine), the model, and the fixed parameters (temperature, the SHA-256 of the rubric): together with the critic's
+`name` and `version` and the image hash they are the identity of a judgement. `evaluate` returns a `verdict` (a score
+and how to read it), `findings`, or both, and may throw: throw a `RateLimitedError` (with the provider's `Retry-After` in
+milliseconds), an `InvalidOutputError`, or any `CriticError` with your own reason code. The runner records the failure
+and the next run retries it. A failure never becomes a score.
+
+## A critique is a layer
+
+A critique is a check inside a part of kind `critique`, in a layer whose `producer` is `critic.<name>`. The check's
+`status` says only whether the critic ran: `passed` a verdict was produced, `failed` the critic could not produce one.
+The opinion is evidence, by the rules of [extending.md](extending.md#the-rules-that-keep-it-compatible-forever), so it
+can never decide a result. It lives in `extensions["atlas.critique"]` and is repeated as standard evidence, so a viewer
+that has never heard of critics still shows it: the score is a measurement named `polish`, and each finding is an
+annotation of type `critique`.
+
+```json
+{
+  "producer": "critic.vision-llm",
+  "kind": "critique",
+  "checks": [{
+    "id": "critique/checkout/payment-sheet",
+    "status": "passed",
+    "measurements": [{ "name": "polish", "value": 0.62, "unit": "score" }],
+    "annotations": [{ "type": "critique", "level": "warning", "message": "spacing: Label touches both edges of the button. (suggestion: Increase horizontal padding.)" }],
+    "extensions": { "atlas.critique": {
+      "schema": 1,
+      "subject": { "entryKey": "checkout/payment-sheet", "imageSha256": "…", "state": "overlay" },
+      "critic": { "name": "vision-llm", "kind": "llm", "version": "1.0.0" },
+      "model": { "id": "…", "provider": "…" },
+      "parameters": { "temperature": 0, "rubric": "<sha256 of the rubric>" },
+      "verdict": { "score": 0.62, "scale": "0..1, higher is better", "label": "needs-polish", "confidence": 0.7 },
+      "findings": [{ "id": "f1", "category": "spacing", "severity": "minor", "region": { "unit": "px", "x": 24, "y": 612, "w": 342, "h": 48 }, "rationale": "Label touches both edges of the button.", "suggestion": "Increase horizontal padding." }]
+    } }
+  }]
+}
+```
+
+- A single score (a model-based critic) is a valid critique: everything except `schema`, `subject` and `critic` is optional.
+- `kind`, `category`, `severity`, `label` and failure `reason` are open strings, and every object is open. A value or field
+  nobody has heard of is kept and shown generically, never rejected.
+- A reader that cannot read the whole extension (a missing `subject`, a score that is not a number, a `schema` version it
+  does not know) drops the extension and keeps the check and its status. An unreadable opinion is unknown.
+- **Anchored to pixels.** `subject.imageSha256` is the hash of the image that was judged. When an entry's image changes,
+  `critiqueCurrency` says the old critique judged an earlier image: it is shown, never hidden, and the entry counts as not
+  yet judged.
+- **Slots.** The producer is the critic's name; the critic and model versions are labels. A newer version of a critic
+  replaces an older one's critique of the same entry. Put the version in `parameters` of `critiqueLayer` to compare
+  versions side by side instead.
+- **The prompt is recorded as a hash** (`parameters.rubric`), not as text.
+- **Failures are data.** A crash, a timeout, a rate limit that never cleared and an unparseable answer are `failed`
+  critiques with `failure.reason` (`error`, `timeout`, `rate-limited`, `invalid-output`, `image-mismatch`…) and a message.
+
+**Critiques never change the gate.** Parts of kind `critique` are left out of every count and of `atlas report status`,
+even when they are the only thing recorded (a report of opinions alone has no test results, so the gate says so). A low
+score does not fail it, and neither does a critic that crashed. A project that wants a score to fail something opts in
+with `gateDecision` in `gate` mode, which is computed by the reader from critique data and never changes a check's
+status, a slot or a winner.
+
+## Running critics
+
+`runCritic` owns everything around the judgement; a critic only judges one image.
+
+- **Caching.** An entry is skipped, without fetching its image, when a critique with a verdict already exists for the same
+  critic, version, model, parameters and image hash (`existing`, read with `critiquesOfLayer`). Identical images across
+  runs are judged once; only a changed image is judged again. A failed critique is not a cache hit. `force` judges again.
+  Layers are content-addressed, so publishing the same outcomes twice is harmless.
+- **Concurrency, timeout, rate limits.** A bounded number of calls in flight. A call over `timeoutMs` is aborted and
+  recorded as `timeout`; the run goes on. A rate-limit answer waits at least `Retry-After`, with exponential backoff and
+  jitter, for a bounded number of attempts, and then is recorded as `rate-limited`.
+- **Budgets.** `budget: { maxImages, maxTokens, maxCostUsd }`. When one is reached the run stops cleanly and returns what
+  it finished with `stoppedBy: 'budget'`. Cost and tokens come from the `usage` a critic reports.
+- **Dry run.** `dryRun` loads and sends nothing and returns what would be judged and skipped, and where images would go
+  (`sendsImagesTo`).
+- **Reproducibility.** Results come back in the order of the entries, so the same inputs publish the same layer; a
+  deterministic critic gives identical scores on every run. Atlas records what explains a result (critic, version,
+  model, rubric hash, parameters) and does not promise that a language model answers the same twice. A re-run adds a
+  newer layer instead of changing an old one.
+- **Images may leave the machine.** `describe().sendsImagesTo` is the thing to show a person before a run that sends
+  images to a third party, and to require explicit confirmation for.
+
+```ts
+import { critiquesOfLayer, publishCritiques, runCritic } from '@crvouga/atlas/critics';
+
+const result = await runCritic({
+  critic, entries, loadImage: (entry) => fetchImage(entry),
+  existing: layers.flatMap((layer) => critiquesOfLayer(layer).map((found) => found.critique)),
+  budget: { maxImages: 200 }, secrets: [providerKey]
+});
+await publishCritiques(store, { critic, outcomes: result.outcomes, recordedAtMs: Date.now() });
+```
+
+## State-aware judging, calibration and modes
+
+A raw score is not absolute: a loading overlay is not a full screen.
+
+- `relativeVerdict` reads a score against the project's scores of the **same state kind** (a percentile within the kind)
+  and against the same entry's previous image (`delta`), and goes into `verdict.relative`.
+- `calibrate` measures a critic against labelled examples **per state kind**: sample sizes, precision, recall, the
+  threshold chosen, and `inSample`, which is true unless held-out examples were given. In-sample figures flatter a critic
+  and are labelled as such.
+- `gateDecision` turns scores into an exit code. Modes: `report` (default: shown, always exit 0), `warn` (a warning for each
+  score below its threshold, exit 0), `gate` (exit 1 when a threshold is crossed). `gate` is **refused** (exit 2, with the
+  reasons) unless every state kind present has a calibration of this critic and version with a threshold, at least 10 bad
+  and 10 good examples, and held-out figures. An explicit threshold never waives that evidence.
+
+## Authentication and authorization
+
+Run critics against a server that has tokens: a server with no tokens is open and every caller is an administrator, a
+critic included.
+
+A token of role `critic` (`ATLAS_CRITIC_TOKEN`, or `role: "critic"` under `auth.tokens`) may do what `read` may, and
+publish critiques, and nothing else. Through the store protocol (`PUT /api/store/…`) it may upload blobs and layers, and
+the catalog entries that make them visible, whose `producer` is `critic.<name>` and whose parts are all of kind
+`critique`. Anything else is `403` naming the rule it broke, and the layer is not stored. It cannot post test results,
+start live runs, change settings or remove anything. `publish` and `admin` tokens may publish critiques too. As for every
+token: send it as `Authorization: Bearer`, never in a URL, never over plain `http` beyond this machine.
+
+## Credentials
+
+Developers bring their own models and AI accounts. Configuration names a credential by **reference**
+(`env:SOME_API_KEY`, `file:/run/secrets/key`) and never contains one, so it can be committed;
+`assertNoLiteralCredentials` refuses a configuration with a literal key and says where, never what. Credentials are read at
+run time (`resolveCredential`) and every message the runner records goes through a redactor (`secrets` in `runCritic`,
+`createRedactor`) that removes them, in their URL-encoded and base64 forms too, and anything shaped like a provider key,
+token or JWT. A credential never reaches a layer, a progress event or an error message.
+
+## What is not here yet
+
+This release delivers the contract, the evidence format, the runner, state-aware judging and calibration, credential
+handling and the scoped `critic` role. Still to come, tracked in [issue 50](https://github.com/crvouga/atlas/issues/50):
+
+- the **catalog read surface** for critics (cursor pagination, `since`, realtime events, images by hash), which builds on
+  the UI catalog of [issue 43](https://github.com/crvouga/atlas/issues/43);
+- the **`atlas critic` command** (`list`, `setup`, `doctor`, `run`, `watch`, `review`, `calibrate`), its interactive mode
+  and its `--format json` documents;
+- the **built-in UIClip critic** and the **LLM critic** with provider adapters;
+- the **viewer**: a ranked "needs polish" list, finding overlays and agreement across critics. Until then critiques are
+  stored, composed and kept out of test counts, and the viewer lists only test results.
+</doc>
+
 <doc path="docs/report-backends.md">
 
 </doc>
@@ -5265,7 +5466,7 @@ Responses: `200` It exists.; `401` No token, or an unknown one. Send `Authorizat
 
 ### Upload an object
 
-`PUT /api/store/{key}` · role: publish
+`PUT /api/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -5277,6 +5478,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 - `key` (path, required): An object key under `layers/`, such as `layers/manifests/<id>.json`. It contains slashes, which are not percent-encoded; every other character is.
 - `content-type` (header): Stored with the object and returned when it is read.
 - `project` (query): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.
@@ -5285,7 +5488,7 @@ Responses: `200` Already there with the same bytes.; `201` Stored.; `401` No tok
 
 ### Upload an object
 
-`PUT /api/projects/{project}/store/{key}` · role: publish
+`PUT /api/projects/{project}/store/{key}` · role: critic
 
 The write half of the `ObjectStore` interface. The server checks what it can from the bytes in hand and says no to what would corrupt the report for everyone:
 
@@ -5297,6 +5500,8 @@ The write half of the `ObjectStore` interface. The server checks what it can fro
 
 Upload in dependency order: blobs, then the manifest, then the catalog entry. `atlas report publish`, `atlas report stream` and `httpStore` do. Needs a `Bearer` token of role `publish` or `admin`; a browser's sign-in box is refused.
 
+A token of role `critic` may upload only what a critic publishes: blobs, and layers (and their catalog entries) whose `producer` is `critic.<name>` and whose parts are all of kind `critique`. Anything else is `403` with the rule it broke, and nothing is stored.
+
 Scoped to one project. The same project may be selected on the unprefixed API using ?project= or X-Atlas-Project. Unknown projects return 404 on reads and are created by an authenticated upload.
 
 - `project` (path, required): Project slug. Alternatively send X-Atlas-Project or use the project-prefixed route. Omit for the default project. Conflicting selectors are refused.

package/llms.txt

@@ -21,6 +21,7 @@ Given only the repository link, start at docs/agents.md: install `@crvouga/atlas
 
 - [The report contract](https://raw.githubusercontent.com/crvouga/atlas/main/docs/report-contract.md): This is the normative description of what Atlas reads and writes when it reports on specs of any kind:
 - [Evidence: attachments, timelines, snapshots and everything else](https://raw.githubusercontent.com/crvouga/atlas/main/docs/extending.md): Atlas is opinionated about specs and open about evidence.
+- [Critics: automated reviewers that mark which UIs need polish](https://raw.githubusercontent.com/crvouga/atlas/main/docs/critics.md): A critic looks at one captured UI (an image and what is known about it) and says how polished it is, and ideally where and why it is not.
 - [docs/report-backends.md](https://raw.githubusercontent.com/crvouga/atlas/main/docs/report-backends.md): 
 - [Executable statecharts](https://raw.githubusercontent.com/crvouga/atlas/main/docs/statecharts.md): Write down how a product behaves as statecharts in open formats, then let Atlas lint them, plan paths that cover every transition, run those paths against the real product through any driver (a browser, a phone, a pla…
 
@@ -46,6 +47,7 @@ Given only the repository link, start at docs/agents.md: install `@crvouga/atlas
 - [TypeScript API: detox](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/detox.md): The Detox driver for statechart runs of React Native apps.
 - [TypeScript API: runners](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/runners.md): Native reporters that upload to the project server.
 - [TypeScript API: vitest](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/vitest.md): Run a statechart plan as Vitest tests.
+- [TypeScript API: critics](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/critics.md): Automated reviewers of the UI catalog: the critic contract, the runner (caching, retries, rate limits, budgets), credential references and redaction, the `atlas.critique` evidence as layers, calibration and report, warn and gate modes.
 - [TypeScript API: playwright-reporter](https://raw.githubusercontent.com/crvouga/atlas/main/docs/reference/playwright-reporter.md): A Playwright Test reporter that streams every test to Atlas with its duration, device labels, steps, retries, videos, screenshots and traces.
 - [JSON Schemas](https://unpkg.com/@crvouga/atlas/packages/schema/schemas/): the layer format, checks file, report status, command results, server responses, server configuration.