@crvouga/atlas 0.17.0 → 1.1.0

package/llms.txt

@@ -21,7 +21,7 @@ Given only the repository link, start at docs/agents.md: install `@crvouga/atlas
 
 - [The report contract](https://raw.githubusercontent.com/crvouga/atlas/main/docs/report-contract.md): This is the normative description of what Atlas reads and writes when it reports on specs of any kind:
 - [Evidence: attachments, timelines, snapshots and everything else](https://raw.githubusercontent.com/crvouga/atlas/main/docs/extending.md): Atlas is opinionated about specs and open about evidence.
-- [Report backends and live snapshots](https://raw.githubusercontent.com/crvouga/atlas/main/docs/report-backends.md): Atlas maps the product's areas, screens, events and journeys from a spec, then overlays evidence of what the product did.
+- [docs/report-backends.md](https://raw.githubusercontent.com/crvouga/atlas/main/docs/report-backends.md): 
 - [Executable statecharts](https://raw.githubusercontent.com/crvouga/atlas/main/docs/statecharts.md): Write down how a product behaves as statecharts in open formats, then let Atlas lint them, plan paths that cover every transition, run those paths against the real product through any driver (a browser, a phone, a pla…
 
 ## Working with this project

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/atlas",
-  "version": "0.17.0",
+  "version": "1.1.0",
   "description": "Atlas: specs of any kind, reported as layered evidence. Publish checks from any runner (JUnit, CTRF, a script), combine them across nodes, branches, environments and backends, gate CI on the result, and run statecharts against the real app.",
   "license": "MIT",
   "type": "module",
@@ -164,7 +164,7 @@
     "schemas": "bun packages/schema/scripts/emit-json-schemas.ts",
     "docs": "bun scripts/docs/generate.ts",
     "docs:check": "bun test scripts/docs scripts/release/docs-links.test.ts packages/schema/tests/schemas.test.ts packages/server/tests/openapi.test.ts packages/server/tests/settings.test.ts",
-    "dev": "bun packages/core/bin/atlas.mjs dev --data .atlas-data",
+    "dev": "bun packages/core/bin/atlas.mjs dev",
     "client:dev": "bun packages/core/bin/atlas.mjs dev",
     "client:build": "bun packages/core/bin/atlas.mjs client build --no-data",
     "client:publish-data": "bun packages/visualizer/scripts/publish-data.ts",

package/packages/reports/STORAGE.md

@@ -69,7 +69,7 @@ adapter records each part independently.
 
 The existing selected-attempt pointer, reusable content hashes and frame bundles
 continue to own test-run publication. An empty closed run is `skipped`, never a
-successful result inferred from absent evidence. Retention keeps historical evidence and its referenced content, including previous attempts and deleted branches. `pruneReports` reports retained commits and deletes nothing; storage usage grows with unique evidence.
+successful result inferred from absent evidence. Retention keeps historical evidence and its referenced content, including previous attempts and deleted branches. `pruneReports` reports retained commits and deletes nothing; storage usage grows with unique evidence unless an operator saves a garbage collection policy for the server, which deletes only superseded layers (`garbage.ts`, [the server](../../docs/server.md#user-state-and-garbage-collection)).
 
 Large bundles overlap at most four 4 MiB chunk uploads per source. Entry parts
 retain input order even when uploads complete out of order. Backpressure bounds

package/packages/schema/schemas/gc-policy.schema.json

@@ -0,0 +1,36 @@
+{
+  "$id": "https://github.com/crvouga/atlas/schemas/gc-policy.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "enabled": {
+      "type": "boolean",
+      "description": "Delete eligible layers on a schedule. `false` keeps the policy but deletes nothing"
+    },
+    "maxAgeDays": {
+      "type": "integer",
+      "minimum": 1,
+      "maximum": 3650,
+      "description": "A superseded layer is eligible once it was recorded more than this many days ago"
+    },
+    "keepLatest": {
+      "default": 20,
+      "description": "Always keep this many of the newest layers, however old, so recent history stays",
+      "type": "integer",
+      "minimum": 0,
+      "maximum": 1000000
+    },
+    "intervalHours": {
+      "default": 24,
+      "description": "How often the server applies the policy",
+      "type": "number",
+      "minimum": 1,
+      "maximum": 720
+    }
+  },
+  "required": [
+    "enabled",
+    "maxAgeDays"
+  ],
+  "additionalProperties": false
+}

package/packages/schema/schemas/gc-run-result.schema.json

@@ -0,0 +1,106 @@
+{
+  "$id": "https://github.com/crvouga/atlas/schemas/gc-run-result.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "runs": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "project": {
+            "type": "string"
+          },
+          "dryRun": {
+            "type": "boolean"
+          },
+          "at": {
+            "type": "string",
+            "description": "When the pass finished, ISO 8601"
+          },
+          "candidates": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991,
+            "description": "Layers the policy would delete"
+          },
+          "deletedLayers": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "deletedBlobs": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "keptLayers": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "error": {
+            "description": "Why the pass stopped early, when it did",
+            "type": "string"
+          }
+        },
+        "required": [
+          "project",
+          "dryRun",
+          "at",
+          "candidates",
+          "deletedLayers",
+          "deletedBlobs",
+          "keptLayers"
+        ],
+        "additionalProperties": {}
+      }
+    },
+    "policy": {
+      "anyOf": [
+        {
+          "type": "object",
+          "properties": {
+            "enabled": {
+              "type": "boolean",
+              "description": "Delete eligible layers on a schedule. `false` keeps the policy but deletes nothing"
+            },
+            "maxAgeDays": {
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 3650,
+              "description": "A superseded layer is eligible once it was recorded more than this many days ago"
+            },
+            "keepLatest": {
+              "default": 20,
+              "description": "Always keep this many of the newest layers, however old, so recent history stays",
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 1000000
+            },
+            "intervalHours": {
+              "default": 24,
+              "description": "How often the server applies the policy",
+              "type": "number",
+              "minimum": 1,
+              "maximum": 720
+            }
+          },
+          "required": [
+            "enabled",
+            "maxAgeDays"
+          ],
+          "additionalProperties": false
+        },
+        {
+          "type": "null"
+        }
+      ]
+    }
+  },
+  "required": [
+    "runs",
+    "policy"
+  ],
+  "additionalProperties": {}
+}

package/packages/schema/schemas/gc-snapshot.schema.json

@@ -0,0 +1,122 @@
+{
+  "$id": "https://github.com/crvouga/atlas/schemas/gc-snapshot.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "revision": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{64}$"
+    },
+    "policy": {
+      "anyOf": [
+        {
+          "type": "object",
+          "properties": {
+            "enabled": {
+              "type": "boolean",
+              "description": "Delete eligible layers on a schedule. `false` keeps the policy but deletes nothing"
+            },
+            "maxAgeDays": {
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 3650,
+              "description": "A superseded layer is eligible once it was recorded more than this many days ago"
+            },
+            "keepLatest": {
+              "default": 20,
+              "description": "Always keep this many of the newest layers, however old, so recent history stays",
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 1000000
+            },
+            "intervalHours": {
+              "default": 24,
+              "description": "How often the server applies the policy",
+              "type": "number",
+              "minimum": 1,
+              "maximum": 720
+            }
+          },
+          "required": [
+            "enabled",
+            "maxAgeDays"
+          ],
+          "additionalProperties": false
+        },
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "saved": {
+      "type": "boolean",
+      "description": "Whether a policy exists in S3. `false` means garbage collection is off: nothing is deleted"
+    },
+    "lastRuns": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "project": {
+            "type": "string"
+          },
+          "dryRun": {
+            "type": "boolean"
+          },
+          "at": {
+            "type": "string",
+            "description": "When the pass finished, ISO 8601"
+          },
+          "candidates": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991,
+            "description": "Layers the policy would delete"
+          },
+          "deletedLayers": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "deletedBlobs": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "keptLayers": {
+            "type": "integer",
+            "minimum": 0,
+            "maximum": 9007199254740991
+          },
+          "error": {
+            "description": "Why the pass stopped early, when it did",
+            "type": "string"
+          }
+        },
+        "required": [
+          "project",
+          "dryRun",
+          "at",
+          "candidates",
+          "deletedLayers",
+          "deletedBlobs",
+          "keptLayers"
+        ],
+        "additionalProperties": {}
+      },
+      "description": "The most recent pass over each project, newest first; empty before the first pass"
+    },
+    "stateKey": {
+      "type": "string",
+      "description": "Where the user state lives in the bucket, relative to the configured prefix"
+    }
+  },
+  "required": [
+    "revision",
+    "policy",
+    "saved",
+    "lastRuns",
+    "stateKey"
+  ],
+  "additionalProperties": {}
+}

package/packages/schema/schemas/server-config.schema.json

@@ -17,47 +17,6 @@
       "minimum": 0,
       "maximum": 65535
     },
-    "data": {
-      "default": "./atlas-data",
-      "description": "The folder that is the default backend and the default write target: layers, specs and runs live here",
-      "type": "string"
-    },
-    "backends": {
-      "default": [],
-      "description": "Further backends, read together with the data folder: S3, another server, a device lab",
-      "type": "array",
-      "items": {
-        "type": "object",
-        "properties": {
-          "id": {
-            "type": "string",
-            "pattern": "^[a-z0-9-]+$",
-            "description": "Lowercase letters, digits and hyphens. Names where a result came from"
-          },
-          "label": {
-            "type": "string"
-          },
-          "url": {
-            "type": "string",
-            "minLength": 1,
-            "description": "A folder, s3://bucket/prefix, r2://, gs://, or another Atlas server (https://…)"
-          },
-          "write": {
-            "description": "Uploads go here instead of the data folder. At most one backend is the write target",
-            "type": "boolean"
-          }
-        },
-        "required": [
-          "id",
-          "url"
-        ],
-        "additionalProperties": false
-      }
-    },
-    "write": {
-      "description": "The id of the backend uploads go to. Default: the data folder",
-      "type": "string"
-    },
     "auth": {
       "default": {
         "tokens": [],

package/packages/schema/schemas/server-editable-settings.schema.json

@@ -3,47 +3,6 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "type": "object",
   "properties": {
-    "data": {
-      "default": "./atlas-data",
-      "description": "The folder that is the default backend and the default write target: layers, specs and runs live here",
-      "type": "string"
-    },
-    "backends": {
-      "default": [],
-      "description": "Further backends, read together with the data folder: S3, another server, a device lab",
-      "type": "array",
-      "items": {
-        "type": "object",
-        "properties": {
-          "id": {
-            "type": "string",
-            "pattern": "^[a-z0-9-]+$",
-            "description": "Lowercase letters, digits and hyphens. Names where a result came from"
-          },
-          "label": {
-            "type": "string"
-          },
-          "url": {
-            "type": "string",
-            "minLength": 1,
-            "description": "A folder, s3://bucket/prefix, r2://, gs://, or another Atlas server (https://…)"
-          },
-          "write": {
-            "description": "Uploads go here instead of the data folder. At most one backend is the write target",
-            "type": "boolean"
-          }
-        },
-        "required": [
-          "id",
-          "url"
-        ],
-        "additionalProperties": false
-      }
-    },
-    "write": {
-      "description": "The id of the backend uploads go to. Default: the data folder",
-      "type": "string"
-    },
     "outlineBranch": {
       "description": "Which branch's specs the report is laid over. Default main, then master, then the newest",
       "type": "string"

package/packages/schema/schemas/server-settings-snapshot.schema.json

@@ -10,47 +10,6 @@
     "settings": {
       "type": "object",
       "properties": {
-        "data": {
-          "default": "./atlas-data",
-          "description": "The folder that is the default backend and the default write target: layers, specs and runs live here",
-          "type": "string"
-        },
-        "backends": {
-          "default": [],
-          "description": "Further backends, read together with the data folder: S3, another server, a device lab",
-          "type": "array",
-          "items": {
-            "type": "object",
-            "properties": {
-              "id": {
-                "type": "string",
-                "pattern": "^[a-z0-9-]+$",
-                "description": "Lowercase letters, digits and hyphens. Names where a result came from"
-              },
-              "label": {
-                "type": "string"
-              },
-              "url": {
-                "type": "string",
-                "minLength": 1,
-                "description": "A folder, s3://bucket/prefix, r2://, gs://, or another Atlas server (https://…)"
-              },
-              "write": {
-                "description": "Uploads go here instead of the data folder. At most one backend is the write target",
-                "type": "boolean"
-              }
-            },
-            "required": [
-              "id",
-              "url"
-            ],
-            "additionalProperties": false
-          }
-        },
-        "write": {
-          "description": "The id of the backend uploads go to. Default: the data folder",
-          "type": "string"
-        },
         "outlineBranch": {
           "description": "Which branch's specs the report is laid over. Default main, then master, then the newest",
           "type": "string"
@@ -140,47 +99,6 @@
     "active": {
       "type": "object",
       "properties": {
-        "data": {
-          "default": "./atlas-data",
-          "description": "The folder that is the default backend and the default write target: layers, specs and runs live here",
-          "type": "string"
-        },
-        "backends": {
-          "default": [],
-          "description": "Further backends, read together with the data folder: S3, another server, a device lab",
-          "type": "array",
-          "items": {
-            "type": "object",
-            "properties": {
-              "id": {
-                "type": "string",
-                "pattern": "^[a-z0-9-]+$",
-                "description": "Lowercase letters, digits and hyphens. Names where a result came from"
-              },
-              "label": {
-                "type": "string"
-              },
-              "url": {
-                "type": "string",
-                "minLength": 1,
-                "description": "A folder, s3://bucket/prefix, r2://, gs://, or another Atlas server (https://…)"
-              },
-              "write": {
-                "description": "Uploads go here instead of the data folder. At most one backend is the write target",
-                "type": "boolean"
-              }
-            },
-            "required": [
-              "id",
-              "url"
-            ],
-            "additionalProperties": false
-          }
-        },
-        "write": {
-          "description": "The id of the backend uploads go to. Default: the data folder",
-          "type": "string"
-        },
         "outlineBranch": {
           "description": "Which branch's specs the report is laid over. Default main, then master, then the newest",
           "type": "string"

package/packages/schema/schemas/user-state.schema.json

@@ -0,0 +1,140 @@
+{
+  "$id": "https://github.com/crvouga/atlas/schemas/user-state.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "schemaVersion": {
+      "type": "number",
+      "const": 1
+    },
+    "gc": {
+      "description": "The garbage collection policy. Absent until someone saves one: then nothing is deleted",
+      "type": "object",
+      "properties": {
+        "enabled": {
+          "type": "boolean",
+          "description": "Delete eligible layers on a schedule. `false` keeps the policy but deletes nothing"
+        },
+        "maxAgeDays": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 3650,
+          "description": "A superseded layer is eligible once it was recorded more than this many days ago"
+        },
+        "keepLatest": {
+          "default": 20,
+          "description": "Always keep this many of the newest layers, however old, so recent history stays",
+          "type": "integer",
+          "minimum": 0,
+          "maximum": 1000000
+        },
+        "intervalHours": {
+          "default": 24,
+          "description": "How often the server applies the policy",
+          "type": "number",
+          "minimum": 1,
+          "maximum": 720
+        }
+      },
+      "required": [
+        "enabled",
+        "maxAgeDays"
+      ],
+      "additionalProperties": false
+    },
+    "settings": {
+      "description": "Report and live-run settings saved from the settings page, applied at the next start",
+      "type": "object",
+      "properties": {
+        "outlineBranch": {
+          "description": "Which branch's specs the report is laid over. Default main, then master, then the newest",
+          "type": "string"
+        },
+        "include": {
+          "default": {},
+          "type": "object",
+          "properties": {
+            "branches": {
+              "description": "Only layers recorded on these branches take part",
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
+            "ignoreModified": {
+              "description": "Leave out layers from modified checkouts and unknown origins",
+              "type": "boolean"
+            }
+          },
+          "additionalProperties": false
+        },
+        "refreshSeconds": {
+          "default": 10,
+          "description": "How often the backends are looked at for layers written by something other than this server",
+          "type": "number",
+          "minimum": 1
+        },
+        "limits": {
+          "default": {
+            "maxObjectMiB": 256,
+            "maxJsonMiB": 32,
+            "maxStreams": 500
+          },
+          "type": "object",
+          "properties": {
+            "maxObjectMiB": {
+              "default": 256,
+              "description": "The largest object an upload may be",
+              "type": "number",
+              "exclusiveMinimum": 0
+            },
+            "maxJsonMiB": {
+              "default": 32,
+              "description": "The largest JSON body (a manifest, a checks request)",
+              "type": "number",
+              "exclusiveMinimum": 0
+            },
+            "maxStreams": {
+              "default": 500,
+              "description": "Viewers that may be connected for live updates at once",
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 9007199254740991
+            }
+          },
+          "additionalProperties": false
+        },
+        "heartbeatSeconds": {
+          "default": 15,
+          "description": "How often a live connection is touched, so proxies keep it open",
+          "type": "number",
+          "minimum": 1
+        },
+        "runTimeoutSeconds": {
+          "default": 30,
+          "description": "Interrupt a live run after this many seconds without a producer heartbeat",
+          "type": "number",
+          "minimum": 10,
+          "maximum": 3600
+        },
+        "runStallSeconds": {
+          "default": 900,
+          "description": "Interrupt after this many seconds without test progress; 0 disables progress expiry",
+          "type": "number",
+          "minimum": 0,
+          "maximum": 86400
+        },
+        "log": {
+          "default": true,
+          "description": "One JSON line per request",
+          "type": "boolean"
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+  "required": [
+    "schemaVersion"
+  ],
+  "additionalProperties": {}
+}

package/packages/server/README.md

@@ -4,17 +4,21 @@ The Atlas server: one process per project that takes in results from every envir
 everyone the combined report, live. A Bun API that serves the Vite-built viewer from the same host (and, under
 `atlas dev`, the viewer's dev server proxies `/api` to it, so development is one host too).
 
-Use it from the command line: `atlas serve`, `atlas dev` ([the guide](../../docs/server.md)). Use it from code when
-you want the server inside something else:
+Use it from the command line: `atlas serve`, `atlas dev` ([the guide](../../docs/server.md)). The command-line server keeps
+**all reports and all user state (the garbage collection policy, saved settings) in S3-compatible storage and nowhere
+else**, set by the required `ATLAS_S3_ENDPOINT`, `ATLAS_S3_BUCKET`, `ATLAS_S3_REGION`, `ATLAS_S3_ACCESS_KEY_ID` and
+`ATLAS_S3_SECRET_ACCESS_KEY` variables (see [`.env.example`](https://github.com/crvouga/atlas/blob/main/.env.example)); it exits with status 2, listing every
+variable and what is wrong, if they are missing or invalid. Use it from code when you want the server inside something else,
+and bring your own `ObjectStore` (a bucket, or a memory store in tests):
 
 ```ts
 import { createAtlasServer } from '@crvouga/atlas/server';
 import { listen } from '@crvouga/atlas/server';
-import { directoryStore } from '@crvouga/atlas/reports/directory-store';
+import { s3Store } from '@crvouga/atlas/reports/s3-store';
 
 const server = createAtlasServer({
-  backends: [{ id: 'data', store: directoryStore('./atlas-data') }],
-  write: 'data',                                  // where uploads go
+  backends: [{ id: 's3', store: s3Store({ bucket: 'my-atlas-reports', region: 'eu-west-1' }) }],
+  write: 's3',                                    // where uploads go
   tokens: [{ name: 'ci', role: 'publish', token: process.env.ATLAS_PUBLISH_TOKEN! }],
   site: './atlas-site'                            // the built viewer; without it only the API is served
 });
@@ -33,7 +37,8 @@ to viewers, `server.close()` stops both.
 | `store-api`, `validate` | The upload protocol and what it refuses (content-addressed layers, dependency order, immutability). |
 | `events` | `EventHub`: server-sent events to every viewer, with a heartbeat and a backlog limit. |
 | `auth` | Roles (`read`, `publish`, `admin`), Bearer tokens, constant-time comparison. |
-| `config` | `loadServerConfig`: the file, then the environment, then the flags; every mistake said in words. |
+| `config`, `storage` | `loadServerConfig`: the saved settings, the file, then the environment, then the flags; every mistake said in words. `resolveS3Config`: the required `ATLAS_S3_*` variables, all problems reported together. |
+| `user-state`, `settings` | The user state JSON in S3 (`server/state.json`): the garbage collection policy and saved settings. |
 | `main`, `dev` | `atlas serve` and `atlas dev`. |
 
 The server describes itself: `GET /api/openapi.json` is an OpenAPI 3.1 document of every route (also `@crvouga/atlas/openapi.json`),