@lossless.org/objectstorage 11.0.0 → 11.2.0

package/readme.md

@@ -69,7 +69,7 @@ described below.
 ### Clustering Features
 
 - 🔗 **Erasure coding** — Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance
-- 🚄 **QUIC transport** — multiplexed inter-node communication via `quinn` with zero head-of-line blocking. Encrypted but not authenticated: run it only on a trusted network (see Cluster transport trust).
+- 🚄 **QUIC transport** — multiplexed inter-node communication via `quinn` with zero head-of-line blocking. Mutual TLS with a cluster CA when `cluster.tls` is set; without it, encrypted but not authenticated, for trusted networks only (see Cluster transport trust).
 - 💽 **Multi-drive awareness** — each node manages multiple independent storage paths with health monitoring
 - 🩺 **Cluster health introspection** — query native node, drive, quorum, and healing status for product dashboards
 - 🤝 **Cluster membership** — static seed config + runtime join, heartbeat-based failure detection
@@ -184,8 +184,9 @@ Cluster mode requires the control root and every drive root to exist before
 startup. The process user must own them, no path component may be a symlink,
 and group/world write permission must be disabled.
 
-The cluster transport does not authenticate nodes: run cluster mode only on a
-network where every host that can reach `quicPort` is a cluster node (see
+Without `cluster.tls` the cluster transport does not authenticate nodes: run
+cluster mode only on a network where every host that can reach `quicPort` is a
+cluster node, or set `cluster.tls` on every node (see
 [Cluster transport trust](#cluster-transport-trust)).
 
 ```typescript
@@ -286,7 +287,7 @@ const config: IObjectStorageConfig = {
   },
   cluster: {                 // Optional — omit for standalone mode
     enabled: true,
-    nodeId: 'node-1',        // Auto-generated UUID if omitted
+    nodeId: 'node-1',        // Auto-generated UUID if omitted; required with tls
     quicPort: 4000,          // Default: 4000
     seedNodes: [],           // Addresses of existing cluster members
     erasure: {
@@ -301,6 +302,16 @@ const config: IObjectStorageConfig = {
     heartbeatIntervalMs: 5000,  // Default: 5000
     heartbeatTimeoutMs: 30000,  // Default: 30000
     getDeadlineMs: 21600000,    // Default: 6 h; 60000 (1 min) to 604800000 (7 d)
+    tls: {                      // Optional mutual TLS; set on every node or on none
+      caFile: '/etc/objectstorage/tls/ca.pem',
+      certFile: '/etc/objectstorage/tls/node-1.pem', // its DNS SAN is nodeId
+      keyFile: '/etc/objectstorage/tls/node-1.key',  // mode 0600 or 0400
+    },
+  },
+  dashboard: {               // Optional read-only dashboard; off unless enabled
+    enabled: true,
+    address: '127.0.0.1',    // Default: '127.0.0.1'
+    port: 3001,              // Default: 3001; 0 allocates an ephemeral TCP port; never the S3 port
   },
 };
 
@@ -617,7 +628,8 @@ to the prefix (its folder marker) is covered.
     (see [Supported S3 Operations](#supported-s3-operations)), so a parameter
     the operation does not implement gets `NotImplemented` (501).
 - **DeleteObjects.** Every member is checked on its own: covered keys are
-  deleted, and the others get a per-key `AccessDenied`.
+  deleted, and the others get a per-key `AccessDenied`, as does every member
+  that names a `VersionId`.
 - **Listings.** ListObjects V1, ListObjectsV2, ListObjectVersions and
   ListMultipartUploads need a `prefix` parameter that is covered. A missing or
   wider prefix is refused, never clamped.
@@ -625,7 +637,8 @@ to the prefix (its folder marker) is covered.
   - ListBuckets returns only the credential's bucket.
   - HeadBucket and `GET /{bucket}?location` work on that bucket.
   - Any other bucket-level query (`?acl`, `?versioning`, `?lifecycle`, …) is
-    refused with `AccessDenied`, and so is any repeated query parameter.
+    refused with `AccessDenied`, and so is any repeated query parameter, so a
+    prefix credential can neither read nor change a bucket's versioning.
   - Creating or deleting buckets and bucket policy operations are refused.
 - **Policies.** The scope check runs before bucket policies, so a policy Allow
   (including `Principal: "*"`) never widens a prefix credential. A policy Deny
@@ -924,8 +937,10 @@ Absent-bucket fenced ensure removes any residual policy before creating the new
 resource; existing-bucket ensure preserves policy and object data. Once a bucket
 has durable fence history, legacy `createBucket()`, `createBucketTenant()`,
 `rotateBucketTenantCredentials()`, `deleteBucketTenant()`, and unfenced
-`importBucket()` mutations fail with `EFENCE_REQUIRED`. Use fenced ensure,
-exact import, and exact delete for its lifecycle.
+`importBucket()` mutations fail with `EFENCE_REQUIRED`. S3 `CreateBucket` and
+`DeleteBucket` on such a bucket get `InvalidBucketState` (`409`), whose message
+is the `EFENCE_REQUIRED: …` text. Use fenced ensure, exact import, and exact
+delete for its lifecycle.
 
 `replaceCredentials()` is a trusted process-local administrative override, not
 a fenced lifecycle API. If it changes exact-owned credential material, completed
@@ -941,7 +956,8 @@ const metrics = await storage.getMetrics();
 
 - `getHealth()` reports running state, storage directory and pool observation, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
 - `getHealth().auth` also reports `prefixCredentialCount` (live prefix credentials), `prefixCredentialTombstoneCount` (unexpired revocations) and `prefixCredentialStoreAvailable` (`false` while the prefix credential file is unreadable and every prefix credential is refused).
-- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (`objectstorage_prefix_credentials_total`, `objectstorage_prefix_credential_tombstones_total`, `objectstorage_prefix_credential_store_available`), and cluster-enabled metrics.
+- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (`objectstorage_prefix_credentials_total`, `objectstorage_prefix_credential_tombstones_total`, `objectstorage_prefix_credential_store_available`), version (`objectstorage_noncurrent_versions_total`, `objectstorage_noncurrent_bytes_total`, `objectstorage_delete_markers_total`) and cluster-enabled metrics.
+- `getHealth().stampClockAheadOfWallMs` is present while version commit times run ahead of the wall clock (a wall clock that went backwards, see [Bucket versioning](#bucket-versioning)).
 - `getHealth()` reads the prefix credential figures through the `getPrefixCredentialStatus` IPC command (no TypeScript method of its own), and `getMetrics()` derives its prefix metrics from `getHealth()`. The command takes no parameters and returns `{ available, credentialCount, tombstoneCount }`: counts and the store state only, never an access key ID, secret, bucket or prefix.
 
 ## Runtime Stats
@@ -962,6 +978,11 @@ interface IBucketSummary {
   objectCount: number;
   totalSizeBytes: number;
   creationDate?: number;
+  // Versioned buckets only:
+  versioning?: 'Enabled' | 'Suspended';
+  noncurrentVersionCount?: number;
+  noncurrentBytes?: number;
+  deleteMarkerCount?: number;
 }
 
 interface IStorageLocationSummary {
@@ -979,9 +1000,22 @@ interface IStorageStats {
   buckets: IBucketSummary[];
   storageDirectory: string;
   storageLocations?: IStorageLocationSummary[];
+  // With at least one versioned bucket:
+  totalNoncurrentVersionCount?: number;
+  totalNoncurrentBytes?: number;
+  totalDeleteMarkerCount?: number;
+  stampClockAheadOfWallMs?: number;
 }
 ```
 
+- Object counts and bytes count current objects only. A versioned bucket also
+  reports its noncurrent data versions and their payload bytes
+  (`noncurrentVersionCount`, `noncurrentBytes`) and its delete markers, current
+  and noncurrent (`deleteMarkerCount`); the stats totals add them up, and
+  `getMetrics()` exports them as `objectstorage_noncurrent_versions_total`,
+  `objectstorage_noncurrent_bytes_total` and `objectstorage_delete_markers_total`.
+  Stored capacity is current plus noncurrent bytes.
+
 - `bucketCount`, `totalObjectCount`, `totalStorageBytes`, and per-bucket totals are logical object stats maintained by the Rust runtime. They count object payload bytes, not sidecar files or erasure-coded shard overhead.
 - objectstorage initializes these values from native on-disk state at startup, then keeps them in memory and updates them when bucket/object mutations succeed. Stats reads do not issue S3 `ListObjects` or rescan every object.
 - Values are exact for mutations performed through objectstorage after startup. Direct filesystem edits outside objectstorage are not watched; restart the server to resync.
@@ -1011,6 +1045,23 @@ interface IClusterHealth {
   drives?: IClusterDriveHealth[];
   erasure?: IClusterErasureHealth;
   repairs?: IClusterRepairHealth;
+  transport?: IClusterTransportHealth;
+}
+
+interface IClusterTransportHealth {
+  mode: 'mutualTls' | 'unauthenticated';
+  certificate?: { sha256: string; notAfter: number };      // only with mutualTls
+  caCertificates?: { sha256: string; notAfter: number }[]; // file order; only with mutualTls
+}
+
+type TDriveMedia = 'rotational' | 'smr' | 'flash' | 'zoned';
+
+interface IClusterDriveHealth {
+  // index, path, status, capacity, error and erasure set fields, plus:
+  media: {
+    declared?: TDriveMedia;              // not reported yet: drives cannot declare their media
+    detected: TDriveMedia | 'unknown';   // detected once at startup
+  };
 }
 ```
 
@@ -1018,10 +1069,88 @@ interface IClusterHealth {
 - Standalone mode returns `{ enabled: false }`.
 - Peer status is the local node's current view of cluster membership and heartbeats, so it is best-effort and may lag real network state.
 - Drive health is based on live native probe checks on the configured local drive paths. Capacity values are cheap filesystem snapshots.
+- `drives[].media.detected` is the medium the engine detected for each drive once at startup, never per health call. On Linux it follows the drive root's filesystem to the whole disk behind it (`st_dev` through `/sys/dev/block`; a partition counts as its disk) and reads the disk's `queue/zoned` and `queue/rotational`: `zoned` for host-managed or host-aware devices (SMR or ZNS), otherwise `rotational` or `flash`, as the disk's driver sets `queue/rotational`. Virtual and network block devices report the flag their driver or hypervisor sets: QEMU disks show `rotational` unless they are configured with `rotation_rate=1`, and NVMe volumes show `flash`. Declaring a drive's media, which is planned, is what will correct such a reading. It is `unknown`, never a guess, for filesystems without a block device of their own (tmpfs, overlay, network and FUSE filesystems, btrfs subvolumes), for device-mapper, md, loop and other virtual block devices (LVM, dm-crypt and software RAID included), when sysfs gives no usable answer, and on every platform but Linux. Drive-managed SMR reports itself as rotational and cannot be detected (`smr` can only ever be declared). The detected media change no behaviour today.
 - `quorumHealthy` means the local node currently sees majority quorum and enough available placements in every erasure set to satisfy the configured write quorum.
+- `transport.mode` is `mutualTls` when `cluster.tls` is set and `unauthenticated` otherwise. With mutual TLS, `certificate` and `caCertificates` describe the files the engine loaded at startup: the SHA-256 of each DER certificate (lowercase hex) and its `notAfter` in milliseconds since the Unix epoch. The files are read only at startup, so during a certificate or CA rotation these show what each node actually runs.
 - Repair fields expose the background healer's currently available runtime state. They are best-effort and limited to what the engine tracks today, such as whether a scan is active, the last completed run, and the last error.
 - `repairs.shardsChecked`, `shardsHealed` and `failed` count the last completed scan's placement checks, shards written and failures. `repairs.shardsPending` counts the pending shards (see [Self-Healing](#self-healing)) that scan left pending because their drive was out of reach, refused the write, or their object was busy. While a scan runs, `lastRunStartedAt` is the running scan's start and the counts are still the previous scan's; `runHealingScan()` returns one scan's start and counts together.
 
+## Dashboard
+
+Every node can serve a small read-only dashboard for quick visibility: the
+cluster's nodes, drives, erasure layout and quorum, object and byte totals,
+request, error and byte rates, healing state and recent engine events. It is
+off by default:
+
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  // ...
+  dashboard: { enabled: true, address: '10.0.0.5', port: 3001 },
+});
+console.log(storage.dashboardPort); // the bound port; undefined while off
+```
+
+Open `http://<address>:<port>/` in a browser. The page is bundled into the
+package (`ts_web/`, built into `ts/dashboard.bundled.ts`), so nothing is
+fetched from elsewhere, and it updates itself over one TypedSocket connection
+to the same origin.
+
+- **Configuration.** `dashboard.enabled` must be set. `address` defaults to
+  `127.0.0.1` and `port` to `3001`; `port: 0` asks the OS for a port. A port
+  equal to `server.port` is refused at construction. A dashboard that is
+  enabled but cannot bind fails `start()`, and the engine is stopped: it never
+  runs silently without it. The `dashboard` section is not sent to the engine.
+- **What it shows.** Nodes (status and last heartbeat in this node's view,
+  start time); local drives of every node (status, usage, errors,
+  last probe); the erasure layout and quorum; objects and bytes per node and
+  summed; request, error and byte rates per node and summed, with two-minute
+  sparklines kept in the browser; healing per node (last scan, checked,
+  healed, pending shards, failures); and the newest events of each node (peer
+  status changes, drive status changes, healing scan results, engine start).
+  A standalone node shows its totals, rates and events. Rates are derived
+  from counters of the S3 listener; byte counts are taken at the TCP stream
+  and include HTTP headers. Nothing is stored: a reload starts the sparklines
+  again.
+- **Where the data comes from.** One management command, `getClusterSnapshot`
+  (also `ObjectStorage.getClusterSnapshot()`), built by the engine from memory
+  only: the drive states the last heartbeat probe left, the in-memory runtime
+  stats, healing state, S3 listener counters and a ring of the newest 64
+  events. It never probes a drive, syncs, walks a directory or reads the
+  control store. A cluster node asks each online peer for that peer's own
+  snapshot over QUIC (`NodeSnapshot`), in parallel, with one second per peer;
+  a peer that is not online or does not answer in time shows as unavailable in
+  that sample. Any node with the dashboard on therefore shows the whole
+  cluster, and the other nodes need no dashboard of their own.
+- **Cost.** With no dashboard connected there is no timer, no IPC and no QUIC
+  traffic. While at least one is connected, one shared sampler takes a
+  snapshot every two seconds, keeps at most one in flight and skips ticks
+  while one runs, and pushes the same sample (a few KB per node) to every
+  connected dashboard. At most 16 dashboard sockets are admitted at once; the
+  17th is refused with `503`. On the engine side, request and byte counting is
+  a relaxed atomic add per response and per socket read or write. An enabled
+  dashboard with nobody watching runs no timer and sends no traffic; on a
+  three-node cluster against 11.0.0 its CPU cost per request was within 0.4 %,
+  and the engine's memory was unchanged. Its server modules and bundled page
+  add about 50 MiB to the Node.js process of each node once, at start, and
+  that does not grow with time or stored objects. With 16 dashboards open on
+  every node, large uploads ran about 1.5 % slower and the p99 latency was
+  about 5 % higher.
+- **Security.** The dashboard is read-only by construction: its socket
+  registers one read handler and nothing else, so it cannot start a healing
+  scan, change configuration or touch S3 traffic or stored data. It has no
+  authentication: protect it with the bind address and a firewall, for
+  example by binding it to a private interface only. It shows node IDs,
+  addresses, drive paths, counts and event messages, never credentials or
+  object keys.
+- **Versions.** Run every node on the same version, as clusters already
+  require. An engine that predates `NodeSnapshot` cannot decode the request,
+  logs a stream error and closes the stream, and the asking node shows it as
+  unavailable.
+
+`getClusterHealth()` probes every local drive (a write, sync and read of a
+probe file) on each call; poll `getClusterSnapshot()` instead when you need a
+cheap periodic view.
+
 ## Supported S3 Operations
 
 The router fails closed, in standalone and cluster mode alike. A request
@@ -1044,7 +1173,7 @@ current object, and `GET ?versioning` answered with an object listing.
 | ListObjectVersions | `GET /{bucket}?versions` | `prefix`, `delimiter`, `max-keys`, `key-marker`, `version-id-marker`, `encoding-type` |
 | ListMultipartUploads | `GET /{bucket}?uploads` | `prefix`, `encoding-type` |
 | GetBucketLocation | `GET /{bucket}?location` | none |
-| GetBucketVersioning | `GET /{bucket}?versioning` | none |
+| GetBucketVersioning, PutBucketVersioning | `GET`, `PUT /{bucket}?versioning` | none (PUT: standalone only, `NotImplemented` in cluster mode) |
 | GetBucketLifecycleConfiguration | `GET /{bucket}?lifecycle` | none |
 | GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy | `GET`, `PUT`, `DELETE /{bucket}?policy` | none |
 | DeleteObjects | `POST /{bucket}?delete` | none |
@@ -1082,9 +1211,129 @@ override such as `response-content-type=text/html` would let a public object
 serve script from the storage origin; a presigned URL is such a request. With
 auth disabled the overrides apply.
 
-Versioning is never enabled on a bucket:
+### Bucket versioning
+
+Standalone buckets support S3 versioning exactly as S3 defines it: a bucket is
+unversioned until `PutBucketVersioning` sets `Enabled` or `Suspended`, and it
+never goes back to unversioned. New buckets are unversioned. A bucket that was
+never versioned behaves exactly as before versioning existed, on disk and on
+the wire: no version ids, no `x-amz-version-id` header, `null` as its only
+version id. In cluster mode `PUT ?versioning` answers `NotImplemented` (501),
+so every cluster bucket stays unversioned for now.
+
+- **Version ids.** A version written while versioning is `Enabled` gets a
+  lowercase UUID as its version id. Versions written before versioning was
+  enabled, and every version written while it is `Suspended`, are the key's
+  `null` version; a key has at most one.
+- **Enabled.** PUT, CopyObject and CompleteMultipartUpload add a new current
+  version and keep the previous one as noncurrent; the response carries
+  `x-amz-version-id`. A DELETE without a version id adds a delete marker as
+  the current version (also for a key with no object, as S3 does): GET and
+  HEAD then answer `NoSuchKey` (404) with `x-amz-delete-marker: true` and the
+  marker's `x-amz-version-id`, and ListObjects no longer lists the key. The
+  DELETE response carries `x-amz-delete-marker: true` and the marker's id.
+- **Suspended.** Writes create the `null` version and replace any previous
+  `null` version of the key, wherever it is in the history; versions with ids
+  stay. A DELETE replaces the `null` version with a `null` delete marker.
+  Responses carry `x-amz-version-id: null`.
+- **By version id.** GET, HEAD and DELETE take `?versionId=` (an id or
+  `null`). Reading a delete marker by its id answers `MethodNotAllowed` (405)
+  with `x-amz-delete-marker: true`, its id and its `Last-Modified`. DELETE with
+  a version id removes that version for good; when it was the current one, the
+  next newest version becomes current again (a delete marker, or the object it
+  covered). A well-formed id that names no version of the key is
+  `NoSuchVersion` (404), also on DELETE; a malformed one is `InvalidArgument`.
+  DeleteObjects accepts a `VersionId` per object and reports `VersionId`,
+  `DeleteMarker` and `DeleteMarkerVersionId` per deleted entry.
+- **Copies.** A copy source may name a version (`?versionId=` after the
+  source); `x-amz-copy-source-if-*` apply to that version, and the response
+  carries `x-amz-copy-source-version-id` (and `x-amz-version-id` for the new
+  version in a versioned bucket). A delete marker as a copy source is
+  `InvalidRequest` (400); a source whose current version is a delete marker is
+  `NoSuchKey`. UploadPartCopy works the same way.
+- **Conditional requests.** `If-None-Match: *` on PUT and
+  CompleteMultipartUpload succeeds when the key has no current version or its
+  current version is a delete marker. `If-Match` on DELETE is evaluated against
+  the current version, also with a version id; a current delete marker fails
+  it (412).
+- **ListObjectVersions** lists every version and delete marker in key order,
+  each key newest first, with `IsLatest` on the current one; versions and
+  delete markers, and common prefixes, all count against `max-keys`. A
+  truncated page carries `NextKeyMarker` and, after a version,
+  `NextVersionIdMarker`; `key-marker` alone resumes after every version of
+  that key, with `version-id-marker` after that version of it. A
+  `version-id-marker` naming a version that was deleted between the pages
+  resumes at that key's newest remaining version: no version of the key is
+  skipped, but versions the previous page already listed are listed again.
+  `Last-Modified` of a version is its commit time.
+- **Permissions.** `PutBucketVersioning` is authorized as
+  `s3:PutBucketVersioning`, `GetBucketVersioning` as `s3:GetBucketVersioning`,
+  ListObjectVersions as `s3:ListBucketVersions`; a `versionId` (also `null`,
+  also in DeleteObjects) as `s3:GetObjectVersion` or `s3:DeleteObjectVersion`.
+  Only root credentials may change a bucket's versioning: bucket tenant
+  credentials get `AccessDenied` for `PutBucketVersioning` (they may read it
+  with `GetBucketVersioning`, list versions, and read and delete versions in
+  their bucket), because the operator's bucket flows (export, import, fenced
+  replacement and deletion, migrations) do not carry version history and
+  refuse versioned buckets. This lifts once a format that carries history
+  exists. Prefix credentials never name a version and cannot change
+  versioning; they can list versions inside their prefix, and their DELETEs
+  add delete markers.
+- **Not combined with.** A bucket with engine-enforced immutable retention
+  refuses versioning, and a versioned bucket refuses retention
+  (`InvalidBucketState`, 409). Bucket export and import, fenced bucket
+  replacement and deletion, and bucket migrations refuse versioned buckets,
+  because none of them carries version history; a migration whose source
+  became versioned after it was created cannot be sealed
+  (`EMIGRATION_VERSIONED`, the source stays active). `MfaDelete` is not
+  implemented (501). A versioned bucket is empty for DeleteBucket only once
+  every version and delete marker is deleted (`BucketNotEmpty` before);
+  deleting it forgets its versioning, so a new bucket of the same name starts
+  unversioned. `deleteBucketTenant` deletes every version with the bucket.
+- **Requests by version wait for writes.** Every request whose answer depends
+  on a bucket's versioning state reads that state, and the history, under the
+  bucket's gate: DELETE and each DeleteObjects member, GET and HEAD with a
+  version id (also `null` on a bucket that was never versioned), a copy or
+  UploadPartCopy source with a version id, the delete-marker answer of a key
+  without a current object, and ListObjectVersions. A request that arrives
+  while `PutBucketVersioning` or a write to the bucket is in progress is
+  therefore served by the state that change leaves. A write holds the gate
+  while it receives its body, so these requests, like every listing, wait
+  for uploads to the same bucket that are in progress: measured with one
+  1 GiB upload into the bucket, a GET by version id issued during it
+  answered once the upload had finished, after 2.6 s at full loopback speed
+  and after 20.3 s at 50 MB/s, while GETs without a version id stayed at a
+  1.6 ms median (3.5 ms p99). GET and HEAD without a version id never wait.
+- **Startup.** A standalone engine checks every version chain once when it
+  starts, reading every version's record and opening its payload (and
+  repairing what a crash left), and takes the bucket counters from that
+  check. Startup time therefore grows with the number of stored versions.
+  Measured with 1,000,000 versions (100,000 keys × 10) on 8 CPUs with a
+  12 GiB memory limit, on a KVM host whose page cache did not keep the root
+  between starts: the first start after writing them took 346 s; six clean
+  starts took 127 to 264 s (127, 145, 150, 193, 203 and 264 s, median about
+  170 s); three starts after a SIGKILL during writes by 32 concurrent
+  writers took 215 to 256 s. A root on which no bucket was ever versioned
+  pays nothing for this and starts exactly as before. `start()` waits for as
+  long as startup takes; once recovery passes 10 s, the engine logs its
+  progress (chains, versions, elapsed time) to the host's stderr every 10 s.
+- **Commit times.** Version commit times come from one stamp clock per
+  engine: `max(wall clock, last stamp + monotonic time since)`, persisted to
+  `.smartstorage/versioning/stamp-clock.json` by every `PutBucketVersioning`
+  that changes a bucket's state (before the state itself, and before the
+  response), every 60 seconds and on a graceful stop. Within a process, a wall clock
+  that jumps back never produces a commit time earlier than true time. After
+  a restart with the wall clock behind, commit times can lag true time by at
+  most the downtime plus 60 seconds (a suspended host counts as downtime). A
+  forward jump that is corrected later keeps commit times ahead by the jump
+  for the life of the process. While commit times run ahead of the wall clock,
+  `getStorageStats()` and `getHealth()` report `stampClockAheadOfWallMs`.
+  Within a key, commit times never go backwards, and a version becomes
+  noncurrent at its successor's commit time.
+
+The reads of an unversioned bucket keep their 11.0.0 semantics:
 - `GET ?versioning` returns the empty `<VersioningConfiguration/>` of a bucket
-  that was never versioned. `PUT ?versioning` is `NotImplemented`.
+  that was never versioned.
 - `GET ?lifecycle` returns `NoSuchLifecycleConfiguration` (404). `PUT` and
   `DELETE ?lifecycle` are `NotImplemented`.
 - `GET ?versions` lists every current object as its S3 null version
@@ -1092,11 +1341,12 @@ Versioning is never enabled on a bucket:
   `key-marker` and `version-id-marker`. A truncated page returns the last key
   or common prefix as `NextKeyMarker`, plus `NextVersionIdMarker` `null` after a
   key. A `version-id-marker` needs a `key-marker`.
-- `versionId` on GET, HEAD and DELETE, and a `?versionId=` after the copy source
-  of CopyObject and UploadPartCopy: `null` is the current object. A
-  well-formed id (a lowercase UUID) names no version and returns
-  `NoSuchVersion` (404; `NoSuchBucket` when the bucket is missing). Anything
-  else, the empty string included, returns `InvalidArgument` (400). A `?`
+- `versionId` on GET, HEAD and DELETE, a `VersionId` of a DeleteObjects
+  member, and a `?versionId=` after the copy source of CopyObject and
+  UploadPartCopy: `null` is the current object. A well-formed id (a lowercase
+  UUID) names no version and returns `NoSuchVersion` (404; `NoSuchBucket` when
+  the bucket is missing). Anything else, the empty string included, returns
+  `InvalidArgument` (400); in DeleteObjects both fail only their member. A `?`
   inside a copy-source key must be encoded as `%3F`.
 - These reads are authorized as `s3:GetBucketVersioning`,
   `s3:GetLifecycleConfiguration` and `s3:ListBucketVersions`. A `versionId`,
@@ -1205,6 +1455,12 @@ durable cluster metadata. Descriptor-confined standalone startup reclaims
 abandoned private object materializations from interrupted create-only, copy,
 or multipart publication without removing an already published hard link.
 
+`CompleteMultipartUpload` refuses, in standalone and cluster mode, a part list
+that names a part that was never uploaded or an ETag that does not match the
+uploaded part with `InvalidPart` (`400`), and a part listed twice with
+`InvalidPartOrder` (`400`). The upload stays open, so a completion with the
+correct parts still succeeds.
+
 Standalone `DeleteObject` also honors `If-Match: *` and comma-separated strong
 ETags. It compares the current ETag and deletes inside the same bucket mutation
 gate used by PUT, COPY, and multipart completion. A mismatch returns the S3
@@ -1455,16 +1711,79 @@ ahead of the client.
 ### QUIC Transport
 
 Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the `quinn` library:
-- **Encrypted, not authenticated** — each node generates a self-signed certificate at startup and accepts any peer; run it only on a trusted network (see [Cluster transport trust](#cluster-transport-trust))
+- **Mutual TLS or unauthenticated** — with `cluster.tls`, every connection is TLS 1.3 with a CA-signed certificate on both ends; without it, each node generates a self-signed certificate at startup and accepts any peer (see [Cluster transport trust](#cluster-transport-trust))
 - 🔀 **Multiplexed streams** — concurrent shard transfers without head-of-line blocking
 - ⚡ **Connection pooling** — persistent connections to peer nodes
 - 🌊 **Natural backpressure** — QUIC flow control prevents overloading slow peers
 
 ### Cluster transport trust
 
-The cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's `quicPort` can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port.
+Without `cluster.tls` the cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's `quicPort` can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port. Run such a cluster only on a network where every host that can reach `quicPort` (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with `quicPort` firewalled from everything else.
 
-Run cluster mode only on a network where every host that can reach `quicPort` (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with `quicPort` firewalled from everything else. This has been the case since cluster mode was introduced, and 11.0.0 does not change it. Mutual TLS with a cluster CA and certificate-checked joins are planned.
+With `cluster.tls` set, the transport is mutual TLS (TLS 1.3, no session resumption, no early data) with a cluster CA you run:
+
+- **Identity.** A node's identity is the single DNS subject alternative name (SAN) of its certificate, and it equals `cluster.nodeId`. With `cluster.tls`, `nodeId` is required and must be a lowercase DNS name such as `node-1` or `node-1.storage.internal`; it is never generated, and it must still match the node id persisted in the control root. One certificate serves as both the server and the client certificate, so it must be valid for both: an extended key usage extension listing `serverAuth` and `clientAuth`, or no extended key usage extension at all (the recipe below issues both). Use a dedicated CA per cluster: every certificate it signs with one lowercase DNS SAN is a member of the cluster.
+- **Checks.** A node connects to a known peer only if the peer's certificate names exactly that peer's node id (one DNS SAN, no wildcard), and to a seed (`seedNodes`, known only by its address) only if the seed presents a certificate from the cluster CA. A node accepts a connection only with a client certificate from the cluster CA, and a join or heartbeat only for the node id that certificate names. A peer without such a certificate fails the handshake before it can send any request. Every CA-signed node is a full member: it may sync topology and read, write, tombstone and purge any shard, so the way to remove a compromised node is to stop trusting its certificate (a new CA, see the rotation below).
+- **Startup.** A node checks its own certificate the way its peers will (chain, validity, server and client authentication, the SAN, and that `keyFile` matches `certFile`) and refuses to start if that check or any file check below fails. The error names the file and the reason. A node never falls back to the unauthenticated transport. At startup it logs one line with its certificate's `notAfter` and the SHA-256 of each CA certificate, and `getClusterHealth().transport` reports them (see [Cluster Health](#cluster-health)).
+- **Files.** `caFile` holds one CA certificate, or two during a CA rotation; anything else is refused, which keeps a system CA bundle out. `certFile` holds the node certificate, optionally followed by intermediates, and nothing but certificates. `keyFile` holds exactly one unencrypted PKCS#8, SEC1 or PKCS#1 key; ECDSA P-256 is recommended. Each path must be absolute and canonical, with no symbolic link anywhere, and lie outside the control root and every drive root (roots are backed up and replaced, and a key must not travel with them). That check compares paths, so a bind mount that makes a root reachable under another path is not detected. Each file must be a regular file of at most 64 KiB. The files and their directories must be owned by the engine's user or by root and must not be group- or world-writable, and `keyFile` must be owned by the engine's user and grant no group or world permission (mode 0600 or 0400).
+- **Reload.** The files are read once at startup; a restart is the reload. Certificates are checked at every handshake, so a connection established before a certificate's `notAfter` keeps running until it reconnects, and new handshakes after `notAfter` fail. Renew certificates and restart the nodes before they expire.
+
+#### Making the CA and the node certificates
+
+The engine ships no CA tooling. With OpenSSL 3 or LibreSSL 3.3 (the `openssl` of macOS), create the CA once and keep `ca.key` offline. `ec_param_enc:named_curve` is required for LibreSSL, which otherwise writes the curve as explicit parameters that the engine refuses; OpenSSL 3 writes the named curve either way:
+
+```bash
+openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
+  -pkeyopt ec_param_enc:named_curve -out ca.key
+openssl req -x509 -new -key ca.key -sha256 -days 3650 \
+  -subj "/CN=objectstorage cluster CA" \
+  -addext "basicConstraints=critical,CA:TRUE" \
+  -addext "keyUsage=critical,keyCertSign,cRLSign" \
+  -out ca.pem
+```
+
+Then issue one certificate per node, with the node id as its only DNS SAN:
+
+```bash
+NODE_ID=node-1
+openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
+  -pkeyopt ec_param_enc:named_curve -out "$NODE_ID.key"
+openssl req -new -key "$NODE_ID.key" -subj "/CN=$NODE_ID" -out "$NODE_ID.csr"
+cat > "$NODE_ID.ext" <<EXT
+basicConstraints = critical,CA:FALSE
+keyUsage = critical,digitalSignature
+extendedKeyUsage = serverAuth,clientAuth
+subjectAltName = DNS:$NODE_ID
+EXT
+openssl x509 -req -in "$NODE_ID.csr" -CA ca.pem -CAkey ca.key -CAcreateserial \
+  -days 365 -sha256 -extfile "$NODE_ID.ext" -out "$NODE_ID.pem"
+```
+
+Install `ca.pem` and the node's certificate and key on the node, as the user that runs the engine and outside its control and drive roots:
+
+```bash
+install -d -m 0755 /etc/objectstorage/tls
+install -m 0644 ca.pem "$NODE_ID.pem" /etc/objectstorage/tls/
+install -m 0600 "$NODE_ID.key" /etc/objectstorage/tls/
+```
+
+Then set `cluster.tls` to the three paths (see [Configuration](#configuration)). The test suite runs exactly these commands (`test/test.cluster-tls.node.ts`).
+
+**Docker secrets** are mounted owned by root with mode 0444 by default, which the key check refuses. Set the key secret's `uid` to the engine's user and its `mode` to `0400` in the service's long `secrets` syntax. Docker Compose ignores `uid`, `gid` and `mode` for secrets read from a file, because it bind-mounts the file; there the host file itself must be owned by the engine's uid with mode 0600 or 0400. Kubernetes secret volumes present their files through symbolic links, which the canonical-path rule refuses.
+
+#### Enabling mutual TLS
+
+Nodes with and without `cluster.tls` cannot talk to each other: a TLS node refuses peers without a client certificate and refuses self-signed servers. Enabling TLS is therefore not a rolling change: stop every node, configure `cluster.tls` on every node, then start every node. A node whose `nodeId` was generated keeps that id (the control root persists it, and `getClusterHealth().nodeId` reports it): configure it as `nodeId` and issue the node's certificate for it, or the node refuses to start. Disabling TLS takes the same full stop.
+
+#### Rotating certificates and the CA
+
+A node certificate is renewed by issuing a new one from the same CA and restarting the node; its peers need no change. A new CA takes three rolling restarts, with no cluster stop:
+
+1. Put both CAs into every node's `caFile` (two certificates) and restart the nodes one at a time.
+2. Issue every node a certificate from the new CA and restart the nodes one at a time.
+3. Remove the old CA from every `caFile` and restart the nodes one at a time.
+
+Every pair of nodes stays connected throughout, because trust only grows before the certificates change and only shrinks after. The order matters: a node with a certificate from the new CA cannot connect to a node that trusts only the old one.
 
 ### Cluster Membership
 
@@ -1616,6 +1935,10 @@ Spawn the Rust binary and start the HTTP server. Concurrent starts join the same
 attempt; starting an already running instance succeeds without restarting it.
 A start is rejected while shutdown or unconfirmed startup cleanup remains
 pending. After completed shutdown, an explicit start begins a new lifecycle.
+A start has no time limit, because startup grows with the stored data (a
+standalone root with bucket versions checks every version chain); it still
+fails at once when the engine process exits. A long start shows its progress
+on the host's stderr, where every engine log line is forwarded.
 
 #### `stop(): Promise<void>`
 
@@ -1847,7 +2170,7 @@ Atomically replace the active runtime credential set without restarting the serv
 
 #### `getClusterHealth(): Promise<IClusterHealth>`
 
-Read the Rust core's current cluster, drive, quorum, and repair health snapshot. Standalone mode returns `{ enabled: false }`.
+Read the Rust core's current cluster, drive, quorum, repair, and transport health snapshot. Standalone mode returns `{ enabled: false }`.
 
 In cluster mode, `quicAddress` is the local QUIC/UDP socket's actual bound
 address. With `cluster.quicPort: 0`, the operating system assigns its port; use
@@ -1855,6 +2178,20 @@ address. With `cluster.quicPort: 0`, the operating system assigns its port; use
 Membership advertisements and peer `quicAddress` values contain that same bound
 address, including after a restart.
 
+#### `getClusterSnapshot(): Promise<IObjectStorageClusterSnapshot>`
+
+The [dashboard](#dashboard)'s snapshot, built by the engine from memory only:
+`local` is this node's `IObjectStorageNodeSnapshot` (counters, totals, drives
+from the last heartbeat probe, healing state, recent events) and, in cluster
+mode, `cluster` adds quorum, the erasure layout and every peer, with the peer's
+own snapshot when it answered within a second. One snapshot runs at a time; a
+call while another runs rejects. Standalone mode returns `cluster: null`.
+
+#### `dashboardPort: number | undefined`
+
+The dashboard's bound TCP port while it runs; `undefined` while it is off or
+stopped.
+
 #### `runHealingScan(options?: IRunHealingScanOptions): Promise<IClusterHealingScanResult>`
 
 Run a healing scan on this cluster node (see [Self-Healing](#self-healing)) and
@@ -1913,7 +2250,7 @@ objectstorage uses a **hybrid Rust + TypeScript** architecture:
 
 **Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves these with bounded-memory streaming I/O, backpressure, and direct `seek()` for range requests.
 
-**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, `createPrefixCredential`, `revokePrefixCredential`, `listPrefixCredentials`, `getPrefixCredentialStatus`, `getClusterHealth`, and `runHealingScan`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes. `runHealingScan` answers when its scan completes, with the request's `id`; the engine serves other commands meanwhile, and a shutdown answers a pending request with an error before the `stop` response.
+**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, `createPrefixCredential`, `revokePrefixCredential`, `listPrefixCredentials`, `getPrefixCredentialStatus`, `getClusterHealth`, `getClusterSnapshot`, and `runHealingScan`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes. `runHealingScan` answers when its scan completes, with the request's `id`; the engine serves other commands meanwhile, and a shutdown answers a pending request with an error before the `stop` response. `getClusterSnapshot` is answered the same deferred way, because a cluster node waits up to a second for each peer.
 
 ### S3-Compatible Operations
 
@@ -1965,6 +2302,8 @@ those versions cannot read the canonical v2 objects.
 | AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId` | |
 | ListMultipartUploads | `GET /{bucket}?uploads` | Optional `prefix` |
 | GetBucketLocation | `GET /{bucket}?location` | Returns `server.region`; an empty constraint for `us-east-1` |
+| GetBucketVersioning, PutBucketVersioning | `GET`, `PUT /{bucket}?versioning` | Standalone; see [Bucket versioning](#bucket-versioning) |
+| ListObjectVersions | `GET /{bucket}?versions` | Every version and delete marker of a versioned bucket |
 | GetBucketPolicy | `GET /{bucket}?policy` | |
 | PutBucketPolicy | `PUT /{bucket}?policy` | |
 | DeleteBucketPolicy | `DELETE /{bucket}?policy` | |
@@ -2045,8 +2384,47 @@ prefix credentials must supply one inside their prefix.
                                          # key leaf appears only with the complete payload
   .smartstorage/credentials.json         # Runtime and bucket tenant credentials (0600)
   .smartstorage/prefix-credentials.json  # Prefix credentials and revocations (0600)
+  .smartstorage/versioning/v1/{bucket}.json
+                                         # Versioning state of a versioned bucket
+  .smartstorage/versioning/stamp-clock.json
+                                         # Persisted floor of the version stamp clock
+  .smartstorage/versions/{bucket}/{whole-key-hex-frames}/
+    {seq:016x}-{generation}              # Record of a noncurrent version or delete marker
+    {seq:016x}-{generation}.object       # Payload of a noncurrent data version
 ```
 
+Bucket versioning keeps the layout above for current objects and adds a
+format that later standalone engines read unchanged:
+- `.smartstorage/versioning/v1/{bucket}.json` is
+  `{"format":"smartstorage.bucket-versioning.v1","bucket":…,"status":"Enabled"|"Suspended"}`.
+  A bucket without it was never versioned.
+- The current object of a key stays at its key path. A payload written while
+  its bucket is versioned ends in a trailer record of schema 2 whose `version`
+  holds its generation UUID, whether it is the null version, its sequence
+  number and its commit time in epoch milliseconds; payloads written before
+  keep schema 1 and are the key's null version with sequence 0.
+- Every other version lives in the key's chain directory below
+  `.smartstorage/versions/{bucket}/`, whose frame directories are the same
+  hexadecimal frames as below `.smartstorage-objects-v2`. A record
+  `{seq:016x}-{generation}` (JSON, `"format":"smartstorage.object-version.v1"`,
+  with the key, sequence, generation, null flag, delete marker flag, commit
+  time, the time it became noncurrent, payload length, ETag and metadata)
+  exists for every noncurrent version and every delete marker; a data version
+  also has `{seq:016x}-{generation}.object`, the payload inode it had while it
+  was current. Sequence order is chain order; the version id is the generation
+  UUID, or `null`.
+- A payload is linked into the chain before its record exists and unlinked
+  after its record is gone. Startup recovery brings every chain to a
+  consistent state after a crash (it removes payloads without a record,
+  finishes or undoes an interrupted change of the current version, clears the
+  noncurrent time of a delete marker that is current again and keeps one null
+  version per key) and refuses to start on a record that is corrupt or misses
+  its payload, on a chain directory with a foreign entry, and on versions of a
+  bucket without versioning state.
+- Engines before 11.2.0 cannot read a payload with a schema-2 trailer: a
+  standalone root on which any bucket was ever versioned (`PutBucketVersioning`
+  with `Enabled` or `Suspended`) cannot be rolled back.
+
 Rolling a standalone root back from 10.7.0 or later to 8.0.3–10.6.0: a crash
 between staging and publication of a PUT, CopyObject or CompleteMultipartUpload
 can leave a staged leaf directory

package/third-party-notices.md

@@ -55,12 +55,28 @@ Developer ID signed or notarized.
 ## Locked Cargo graph
 
 [Cargo dependency notices](./third_party_licenses/cargo-dependencies.html)
-contain the selected license texts and original attributions for 260 packages.
-The complete locked graph also contains three feature-inactive packages:
+contain the selected license texts and original attributions for 279 packages.
+The complete locked graph also contains 18 feature-inactive packages:
 [aho-corasick 1.1.4](./third_party_licenses/aho-corasick-1.1.4-license.txt),
-[valuable 0.1.1](./third_party_licenses/valuable-license.txt) and
+[borsh 1.8.1](./third_party_licenses/borsh-1.8.1-license.txt),
+[borsh-derive 1.8.1](./third_party_licenses/borsh-derive-1.8.1-apache-2.0.txt),
+[dyn-clone 1.0.20](./third_party_licenses/dyn-clone-1.0.20-license.txt),
+[equivalent 1.0.3](./third_party_licenses/equivalent-1.0.3-license.txt),
+[hashbrown 0.17.1](./third_party_licenses/hashbrown-0.17.1-license.txt),
+[indexmap 2.14.2](./third_party_licenses/indexmap-2.14.2-license.txt),
+[proc-macro-crate 3.5.0](./third_party_licenses/proc-macro-crate-3.5.0-license.txt),
+[ref-cast 1.0.27](./third_party_licenses/ref-cast-1.0.27-license.txt),
+[ref-cast-impl 1.0.27](./third_party_licenses/ref-cast-impl-1.0.27-license.txt),
+[schemars 1.2.3](./third_party_licenses/schemars-1.2.3-license.txt),
+[syn 3.0.9](./third_party_licenses/syn-3.0.9-license.txt),
+[toml_datetime 1.1.2](./third_party_licenses/toml_datetime-1.1.2-license.txt),
+[toml_edit 0.25.17](./third_party_licenses/toml_edit-0.25.17-license.txt),
+[toml_parser 1.1.5](./third_party_licenses/toml_parser-1.1.5-license.txt),
+[valuable 0.1.1](./third_party_licenses/valuable-license.txt),
+[winnow 1.0.4](./third_party_licenses/winnow-1.0.4-license.txt) and
 [zerocopy-derive 0.8.47](./third_party_licenses/zerocopy-derive-0.8.47-license.txt).
-Their exact MIT notices bring the inventory to all 261 locked third-party
+Their exact notices (Apache-2.0 for borsh-derive, which offers no MIT
+alternative, MIT for the others) bring the inventory to all 297 locked third-party
 packages, conservatively including build-only, development and inactive targets.
 Listing a package does not claim that it is linked into every distributed executable.