@lossless.org/objectstorage 10.7.1 → 11.1.0

package/readme.md

@@ -69,13 +69,13 @@ described below.
 ### Clustering Features
 
 - 🔗 **Erasure coding** — Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance
-- 🚄 **QUIC transport** — multiplexed, encrypted inter-node communication via `quinn` with zero head-of-line blocking
+- 🚄 **QUIC transport** — multiplexed inter-node communication via `quinn` with zero head-of-line blocking. Mutual TLS with a cluster CA when `cluster.tls` is set; without it, encrypted but not authenticated, for trusted networks only (see Cluster transport trust).
 - 💽 **Multi-drive awareness** — each node manages multiple independent storage paths with health monitoring
 - 🩺 **Cluster health introspection** — query native node, drive, quorum, and healing status for product dashboards
 - 🤝 **Cluster membership** — static seed config + runtime join, heartbeat-based failure detection
 - ✍️ **Quorum writes** — data is only acknowledged after k+1 shards are persisted
 - 📖 **Quorum reads** — reconstruct from any k available shards, local-first fast path
-- 🩹 **Self-healing** — background scanner detects and reconstructs missing/corrupt shards
+- 🩹 **Self-healing** — background scanner rebuilds missing shards on the drives their manifest names
 
 ## Installation
 
@@ -114,7 +114,7 @@ glibc LGPL terms, Cargo licenses and native distribution inventory.
 ## Native builds and verification
 
 `pnpm build` builds the current host architecture (GNU and musl on Linux), then
-builds the TypeScript facade. Test preparation uses `tsrust --configured-targets`
+builds the TypeScript facade. Test preparation uses `rsrepo --configured-targets`
 to cover all configured host-OS binaries in the packaging checks. Rust 1.95.0 is
 pinned in `rust-toolchain.toml`. Mac builders
 need Xcode command-line tools, both Darwin Rust targets and Rosetta 2 for Intel
@@ -126,7 +126,7 @@ with aliases from their full `*-unknown-linux-musl-gcc` names to the names above
 Rust supplies its pinned, patched musl runtime. Native runtime notices are included
 in the package; retain them when distributing standalone binaries.
 
-Releases use `tsrust matrix build` to assemble all six binaries from one clean
+Releases use `rsrepo matrix build` to assemble all six binaries from one clean
 Git commit. Configure these environment variables on the release coordinator:
 
 ```sh
@@ -184,6 +184,11 @@ Cluster mode requires the control root and every drive root to exist before
 startup. The process user must own them, no path component may be a symlink,
 and group/world write permission must be disabled.
 
+Without `cluster.tls` the cluster transport does not authenticate nodes: run
+cluster mode only on a network where every host that can reach `quicPort` is a
+cluster node, or set `cluster.tls` on every node (see
+[Cluster transport trust](#cluster-transport-trust)).
+
 ```typescript
 import { ObjectStorage } from '@lossless.org/objectstorage';
 
@@ -211,6 +216,21 @@ const storage = await ObjectStorage.createAndStart({
 
 Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data.
 
+Each drive root counts as one independent drive, so startup refuses two drive
+roots on the same device (`st_dev`). Only the device number is checked:
+partitions of one disk, LVM volumes, btrfs subvolumes and datasets of one ZFS
+pool report distinct devices, are not detected, and keeping them apart is the
+operator's responsibility. The control root may share a device with a drive.
+For dev and test clusters whose drive roots are directories on one filesystem,
+opt in explicitly; startup then logs which roots share a device:
+
+```typescript
+drives: {
+  paths: ['/srv/objectstorage-dev/drive-1', '/srv/objectstorage-dev/drive-2', '/srv/objectstorage-dev/drive-3'],
+  allowSharedDevice: true, // dev/test only: these roots are not independent drives
+},
+```
+
 ## Configuration
 
 All config fields are optional — sensible defaults are applied automatically.
@@ -263,22 +283,35 @@ const config: IObjectStorageConfig = {
   multipart: {
     expirationDays: 7,             // Must be greater than zero
     cleanupIntervalMinutes: 60,    // Must be greater than zero
+    maxUploadsPerPrefixScope: 100, // Open uploads per prefix credential scope; must be non-zero
   },
   cluster: {                 // Optional — omit for standalone mode
     enabled: true,
-    nodeId: 'node-1',        // Auto-generated UUID if omitted
+    nodeId: 'node-1',        // Auto-generated UUID if omitted; required with tls
     quicPort: 4000,          // Default: 4000
     seedNodes: [],           // Addresses of existing cluster members
     erasure: {
       dataShards: 4,         // Default: 4
       parityShards: 2,       // Default: 2
-      chunkSizeBytes: 4194304, // Default: 4 MB
+      chunkSizeBytes: 4194304, // Default: 4 MB; between 1 and 64 MiB
     },
     drives: {
       paths: ['/mnt/disk1', '/mnt/disk2'],
+      allowSharedDevice: false, // Default: false; true only for dev/test roots on one device
     },
     heartbeatIntervalMs: 5000,  // Default: 5000
     heartbeatTimeoutMs: 30000,  // Default: 30000
+    getDeadlineMs: 21600000,    // Default: 6 h; 60000 (1 min) to 604800000 (7 d)
+    tls: {                      // Optional mutual TLS; set on every node or on none
+      caFile: '/etc/objectstorage/tls/ca.pem',
+      certFile: '/etc/objectstorage/tls/node-1.pem', // its DNS SAN is nodeId
+      keyFile: '/etc/objectstorage/tls/node-1.key',  // mode 0600 or 0400
+    },
+  },
+  dashboard: {               // Optional read-only dashboard; off unless enabled
+    enabled: true,
+    address: '127.0.0.1',    // Default: '127.0.0.1'
+    port: 3001,              // Default: 3001; 0 allocates an ephemeral TCP port; never the S3 port
   },
 };
 
@@ -460,6 +493,7 @@ interface IStorageCredential {
 - Requests that were already authenticated before the replacement keep running; auth is evaluated when each request starts.
 - No restart is required, and runtime-created credentials survive restart unless `storage.cleanSlate` clears the bounded storage contents; the owned root, maintenance-lock inode, and any durable provider/pool identity are preserved.
 - Replacement input must contain at least one credential, each `accessKeyId` and `secretAccessKey` must be non-empty, and `accessKeyId` values must be unique.
+- A new `accessKeyId` must not be held by a live or revoked prefix credential (`ECREDENTIAL_CONFLICT`); the same check applies to bucket tenant create, ensure and rotate. Prefix credentials are not part of `listCredentials()`; see [Prefix Credentials](#prefix-credentials).
 
 ## Bucket Tenants
 
@@ -499,6 +533,207 @@ const tenants = await storage.listBucketTenants();
 - Tenant credentials can list, read, write, and delete objects in their assigned bucket, but cannot list all buckets, access other buckets, copy from other buckets, delete buckets, or mutate bucket policies.
 - Bucket tenant APIs require `auth.enabled: true`.
 
+## Prefix Credentials
+
+A prefix credential is an ordinary SigV4 key that reaches only the keys below one
+prefix of one bucket, and only until it expires. It is meant for handing a
+short-lived key to a guest (a VM, a job, a mounted drive) that must see one
+folder and nothing else.
+
+```typescript
+const credential = await storage.createPrefixCredential({
+  bucketName: 'drives',
+  prefix: 'drives/user-42/',
+  ttlSeconds: 3600,          // 1 to 43200 (12 h)
+});
+// { accessKeyId, secretAccessKey, bucketName, prefix, region, createdAt, expiresAt }
+
+const client = new S3Client({
+  endpoint: 'http://localhost:3000',
+  region: credential.region,
+  credentials: {
+    accessKeyId: credential.accessKeyId,
+    secretAccessKey: credential.secretAccessKey,
+  },
+  forcePathStyle: true,
+});
+
+await storage.revokePrefixCredential({ accessKeyId: credential.accessKeyId }); // { revoked: true }
+const live = await storage.listPrefixCredentials({ bucketName: 'drives' });   // metadata, no secrets
+```
+
+```typescript
+interface IPrefixCredentialInput {
+  bucketName: string;
+  prefix: string;
+  ttlSeconds?: number;      // first node: lifetime in seconds
+  expiresAt?: number;       // further nodes: the first node's expiresAt (epoch ms)
+  accessKeyId?: string;     // supplied together with secretAccessKey and expiresAt only
+  secretAccessKey?: string;
+  region?: string;          // default: server.region
+}
+```
+
+**Input rules.**
+- Exactly one of `ttlSeconds` (an integer, 1 to 43 200) and `expiresAt` (an
+  integer epoch-ms time after now and at most 43 200 s ahead).
+- Key material is generated when omitted (`SP` plus 20 uppercase hex
+  characters, a 64-hex-character secret). Supplied material needs `expiresAt`,
+  and both `accessKeyId` and `secretAccessKey`: an ID of 16 to 128 characters
+  of `A-Z0-9`, and a secret of 32 to 128 printable ASCII characters without
+  spaces.
+- The prefix is validated, never normalised. It is 1 to 1023 bytes of
+  `A-Z a-z 0-9 - . _ ~ /` (RFC 3986 unreserved characters plus `/`, no
+  percent-encoding). It ends with `/` and does not start with `/`, and has no
+  empty, `.` or `..` segment.
+- The bucket must exist. The access key ID must not belong to a static or bucket
+  tenant credential, and it must not have been revoked.
+- The secret appears only in the create result. It is never listed, logged, or
+  stored in `storage.config`.
+- Errors are `ObjectStorageCredentialError` with a `code`:
+
+| Code | Meaning |
+|---|---|
+| `ECREDENTIAL_AUTH_DISABLED` | `auth.enabled` is false |
+| `ECREDENTIAL_INVALID_PREFIX` | the prefix fails the grammar |
+| `ECREDENTIAL_INVALID_TTL` | not exactly one of `ttlSeconds` and `expiresAt`, or `ttlSeconds` out of range |
+| `ECREDENTIAL_INVALID_EXPIRY` | `expiresAt` is not in the future, or is more than 12 h ahead |
+| `ECREDENTIAL_INVALID_KEY` | the key material or the region is malformed, or only part of it was supplied |
+| `ECREDENTIAL_NO_SUCH_BUCKET` | the bucket does not exist |
+| `ECREDENTIAL_BUCKET_UNAVAILABLE` | the bucket is a sealed or retired migration source |
+| `ECREDENTIAL_CONFLICT` | the ID is held by another credential, or a replay differs from the stored record |
+| `ECREDENTIAL_REVOKED` | the ID was revoked |
+| `ECREDENTIAL_NOT_PREFIX` | a revoke named a static or bucket tenant credential |
+| `ECREDENTIAL_LIMIT` | 10 000 live credentials and revocations are stored |
+| `ECREDENTIAL_PERSISTENCE` | the store could not be written |
+| `ECREDENTIAL_STORE_UNAVAILABLE` | the store file is unreadable (see below) |
+
+**Access.** A key is covered when its bytes start with the prefix; a key equal
+to the prefix (its folder marker) is covered.
+- **Objects.** GetObject, HeadObject, PutObject, DeleteObject and the multipart
+  operations work on covered keys.
+  - Writes also refuse keys with `.`, `..` or empty segments after the prefix,
+    and keys with control characters.
+  - CopyObject and UploadPartCopy need the source to be covered too.
+  - Object requests accept only the query parameters the operation uses:
+    `uploads`, `uploadId`, `partNumber`, `max-parts` and `part-number-marker`,
+    the six `response-*` header overrides on GetObject and HeadObject, plus
+    the `x-id` operation label that AWS SDKs append. Any other object query
+    (`?acl`, `?tagging`, `?versionId`, `?retention`, …) is refused with
+    `AccessDenied` (403), also a name S3 does not define (`?v=1`), and so is
+    any repeated query parameter. A prefix
+    credential never names a version: `?versionId` is refused even as `null`,
+    and so is a copy source with a `?versionId=` suffix.
+  - Requests that pass these checks still go through the router's allowlist
+    (see [Supported S3 Operations](#supported-s3-operations)), so a parameter
+    the operation does not implement gets `NotImplemented` (501).
+- **DeleteObjects.** Every member is checked on its own: covered keys are
+  deleted, and the others get a per-key `AccessDenied`.
+- **Listings.** ListObjects V1, ListObjectsV2, ListObjectVersions and
+  ListMultipartUploads need a `prefix` parameter that is covered. A missing or
+  wider prefix is refused, never clamped.
+- **Bucket level.**
+  - ListBuckets returns only the credential's bucket.
+  - HeadBucket and `GET /{bucket}?location` work on that bucket.
+  - Any other bucket-level query (`?acl`, `?versioning`, `?lifecycle`, …) is
+    refused with `AccessDenied`, and so is any repeated query parameter.
+  - Creating or deleting buckets and bucket policy operations are refused.
+- **Policies.** The scope check runs before bucket policies, so a policy Allow
+  (including `Principal: "*"`) never widens a prefix credential. A policy Deny
+  still applies.
+
+**Expiry and deadlines.**
+- A credential authenticates while the engine clock is before `expiresAt`.
+  After that, or after revocation, requests get `InvalidAccessKeyId` (403).
+- Requests that were admitted end at `expiresAt` as well: a trickled upload
+  fails with nothing stored, and a slowly read download is cut off.
+- Revocation affects only requests admitted after it. Requests already running
+  continue until they finish or until `expiresAt`.
+- There is no extend operation. To refresh, create a new key and revoke the old
+  one.
+- **Multipart cap.** All prefix credentials of one scope (the same bucket and
+  prefix) together hold at most `multipart.maxUploadsPerPrefixScope`
+  (default 100, must be non-zero) open multipart uploads. The next
+  CreateMultipartUpload of that scope gets `AccessDenied` (403) until one of
+  them ends: completed or aborted by any credential, or removed by the
+  multipart expiry cleanup. Revoking, expiring or replacing a credential
+  frees none of its scope's slots, so issuing new keys does not raise the cap.
+  The count is held in memory and resets when the engine restarts. The
+  engine-wide limit of 100 000 uploads still applies.
+
+**Bucket deletion.** Deletion attempts revoke first; callers re-issue.
+- Every path that deletes a bucket revokes the bucket's prefix credentials
+  before it removes anything: S3 DeleteBucket, `deleteBucket()`,
+  `deleteBucketTenant()` without an `accessKeyId`, and migration source
+  cleanup. A recreated bucket of the same name is not reachable with them.
+- S3 DeleteBucket checks that the bucket is empty before it revokes, so a
+  `BucketNotEmpty` (409) keeps the credentials.
+- If the revocation cannot be stored, the deletion is refused.
+- A deletion that fails or is refused after the revocation (for example a
+  stale resource fence, `EFENCE_STALE`) returns that error, never a success,
+  and leaves the credentials revoked. Callers that keep the bucket create new
+  prefix credentials.
+
+**Persistence.**
+- Credentials and revocations live in `.smartstorage/prefix-credentials.json`,
+  under the same root as `credentials.json`. The engine writes the file
+  atomically with mode `0600`, and `storage.cleanSlate` clears it.
+- A sweep every 60 seconds drops expired entries.
+- Engines older than prefix credentials never read the file, so a rollback
+  disables every prefix credential rather than widening it.
+- **Fail closed.** If the file is unreadable, larger than 32 MiB, malformed,
+  or inconsistent (duplicate IDs, an ID held by a static credential, a record
+  that fails validation), the store is unavailable:
+  - every prefix credential is refused;
+  - create, revoke and list fail with `ECREDENTIAL_STORE_UNAVAILABLE`;
+  - no new static or tenant access key ID can be introduced, because the ID
+    collision check cannot run: `replaceCredentials()`,
+    `createBucketTenant()`, `ensureBucketTenant()` and
+    `rotateBucketTenantCredentials()` throw `ObjectStorageCredentialError`
+    with code `ECREDENTIAL_STORE_UNAVAILABLE` when they would add an access
+    key ID that is not in the current set;
+  - bucket deletion is refused, because the bucket's credentials cannot be
+    revoked: `deleteBucket()` and `deleteBucketTenant()` without an
+    `accessKeyId` throw `ObjectStorageCredentialError` with code
+    `ECREDENTIAL_STORE_UNAVAILABLE`, and S3 DeleteBucket returns
+    `InvalidRequest` (400) with a message starting
+    `ECREDENTIAL_STORE_UNAVAILABLE:`;
+  - static and tenant credentials keep working;
+  - `getHealth().auth.prefixCredentialStoreAvailable` is `false`;
+  - the engine logs the condition without secrets and leaves the file as found.
+
+  To recover, repair or delete the file and restart. Deleting it revokes every
+  prefix credential and forgets every revocation, so never reuse a revoked ID
+  afterwards. Generated IDs are random.
+
+**Cluster mode.** Prefix credentials are node-local, like runtime credentials:
+nothing replicates them, and the caller fans out.
+1. Create on one node with `ttlSeconds`.
+2. Create on every other node with the first node's `accessKeyId`,
+   `secretAccessKey`, `expiresAt` and `region`.
+   - Each node records its own `createdAt`; every other field matches the first
+     node's record.
+   - An identical replay returns the stored record unchanged. Any difference
+     gives `ECREDENTIAL_CONFLICT`.
+   - A late create never extends the lifetime.
+3. To revoke, call `revokePrefixCredential` on every node and retry until each
+   node acknowledges.
+   - A node that never saw the ID records the revocation for 12 h, so a create
+     that reaches it later is refused with `ECREDENTIAL_REVOKED`.
+   - `revoked` is `true` only where a live credential was removed.
+
+**Revocation guarantee.**
+- Once a node has acknowledged a revocation, it refuses every request admitted
+  after the acknowledgement. It also refuses any later create for that ID until
+  the ID could no longer be created anyway: an `expiresAt` create must lie
+  within 12 h of the first node's create, and the revocation is kept at least
+  that long.
+- A node that never acknowledged serves the key until the caller's retry
+  reaches it, or until `expiresAt`.
+- The hard bound is `expiresAt`, at most 12 h after the first create. Each node
+  judges expiry by its own clock, so clock offsets shift these instants.
+- `listPrefixCredentials()` lets the caller reconcile nodes.
+
 ## Bucket Backup/Restore
 
 ```typescript
@@ -716,7 +951,9 @@ const metrics = await storage.getMetrics();
 ```
 
 - `getHealth()` reports running state, storage directory and pool observation, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
-- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, and cluster-enabled metrics.
+- `getHealth().auth` also reports `prefixCredentialCount` (live prefix credentials), `prefixCredentialTombstoneCount` (unexpired revocations) and `prefixCredentialStoreAvailable` (`false` while the prefix credential file is unreadable and every prefix credential is refused).
+- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (`objectstorage_prefix_credentials_total`, `objectstorage_prefix_credential_tombstones_total`, `objectstorage_prefix_credential_store_available`), and cluster-enabled metrics.
+- `getHealth()` reads the prefix credential figures through the `getPrefixCredentialStatus` IPC command (no TypeScript method of its own), and `getMetrics()` derives its prefix metrics from `getHealth()`. The command takes no parameters and returns `{ available, credentialCount, tombstoneCount }`: counts and the store state only, never an access key ID, secret, bucket or prefix.
 
 ## Runtime Stats
 
@@ -785,6 +1022,23 @@ interface IClusterHealth {
   drives?: IClusterDriveHealth[];
   erasure?: IClusterErasureHealth;
   repairs?: IClusterRepairHealth;
+  transport?: IClusterTransportHealth;
+}
+
+interface IClusterTransportHealth {
+  mode: 'mutualTls' | 'unauthenticated';
+  certificate?: { sha256: string; notAfter: number };      // only with mutualTls
+  caCertificates?: { sha256: string; notAfter: number }[]; // file order; only with mutualTls
+}
+
+type TDriveMedia = 'rotational' | 'smr' | 'flash' | 'zoned';
+
+interface IClusterDriveHealth {
+  // index, path, status, capacity, error and erasure set fields, plus:
+  media: {
+    declared?: TDriveMedia;              // not reported yet: drives cannot declare their media
+    detected: TDriveMedia | 'unknown';   // detected once at startup
+  };
 }
 ```
 
@@ -792,8 +1046,203 @@ interface IClusterHealth {
 - Standalone mode returns `{ enabled: false }`.
 - Peer status is the local node's current view of cluster membership and heartbeats, so it is best-effort and may lag real network state.
 - Drive health is based on live native probe checks on the configured local drive paths. Capacity values are cheap filesystem snapshots.
+- `drives[].media.detected` is the medium the engine detected for each drive once at startup, never per health call. On Linux it follows the drive root's filesystem to the whole disk behind it (`st_dev` through `/sys/dev/block`; a partition counts as its disk) and reads the disk's `queue/zoned` and `queue/rotational`: `zoned` for host-managed or host-aware devices (SMR or ZNS), otherwise `rotational` or `flash`, as the disk's driver sets `queue/rotational`. Virtual and network block devices report the flag their driver or hypervisor sets: QEMU disks show `rotational` unless they are configured with `rotation_rate=1`, and NVMe volumes show `flash`. Declaring a drive's media, which is planned, is what will correct such a reading. It is `unknown`, never a guess, for filesystems without a block device of their own (tmpfs, overlay, network and FUSE filesystems, btrfs subvolumes), for device-mapper, md, loop and other virtual block devices (LVM, dm-crypt and software RAID included), when sysfs gives no usable answer, and on every platform but Linux. Drive-managed SMR reports itself as rotational and cannot be detected (`smr` can only ever be declared). The detected media change no behaviour today.
 - `quorumHealthy` means the local node currently sees majority quorum and enough available placements in every erasure set to satisfy the configured write quorum.
+- `transport.mode` is `mutualTls` when `cluster.tls` is set and `unauthenticated` otherwise. With mutual TLS, `certificate` and `caCertificates` describe the files the engine loaded at startup: the SHA-256 of each DER certificate (lowercase hex) and its `notAfter` in milliseconds since the Unix epoch. The files are read only at startup, so during a certificate or CA rotation these show what each node actually runs.
 - Repair fields expose the background healer's currently available runtime state. They are best-effort and limited to what the engine tracks today, such as whether a scan is active, the last completed run, and the last error.
+- `repairs.shardsChecked`, `shardsHealed` and `failed` count the last completed scan's placement checks, shards written and failures. `repairs.shardsPending` counts the pending shards (see [Self-Healing](#self-healing)) that scan left pending because their drive was out of reach, refused the write, or their object was busy. While a scan runs, `lastRunStartedAt` is the running scan's start and the counts are still the previous scan's; `runHealingScan()` returns one scan's start and counts together.
+
+## Dashboard
+
+Every node can serve a small read-only dashboard for quick visibility: the
+cluster's nodes, drives, erasure layout and quorum, object and byte totals,
+request, error and byte rates, healing state and recent engine events. It is
+off by default:
+
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  // ...
+  dashboard: { enabled: true, address: '10.0.0.5', port: 3001 },
+});
+console.log(storage.dashboardPort); // the bound port; undefined while off
+```
+
+Open `http://<address>:<port>/` in a browser. The page is bundled into the
+package (`ts_web/`, built into `ts/dashboard.bundled.ts`), so nothing is
+fetched from elsewhere, and it updates itself over one TypedSocket connection
+to the same origin.
+
+- **Configuration.** `dashboard.enabled` must be set. `address` defaults to
+  `127.0.0.1` and `port` to `3001`; `port: 0` asks the OS for a port. A port
+  equal to `server.port` is refused at construction. A dashboard that is
+  enabled but cannot bind fails `start()`, and the engine is stopped: it never
+  runs silently without it. The `dashboard` section is not sent to the engine.
+- **What it shows.** Nodes (status and last heartbeat in this node's view,
+  start time); local drives of every node (status, usage, errors,
+  last probe); the erasure layout and quorum; objects and bytes per node and
+  summed; request, error and byte rates per node and summed, with two-minute
+  sparklines kept in the browser; healing per node (last scan, checked,
+  healed, pending shards, failures); and the newest events of each node (peer
+  status changes, drive status changes, healing scan results, engine start).
+  A standalone node shows its totals, rates and events. Rates are derived
+  from counters of the S3 listener; byte counts are taken at the TCP stream
+  and include HTTP headers. Nothing is stored: a reload starts the sparklines
+  again.
+- **Where the data comes from.** One management command, `getClusterSnapshot`
+  (also `ObjectStorage.getClusterSnapshot()`), built by the engine from memory
+  only: the drive states the last heartbeat probe left, the in-memory runtime
+  stats, healing state, S3 listener counters and a ring of the newest 64
+  events. It never probes a drive, syncs, walks a directory or reads the
+  control store. A cluster node asks each online peer for that peer's own
+  snapshot over QUIC (`NodeSnapshot`), in parallel, with one second per peer;
+  a peer that is not online or does not answer in time shows as unavailable in
+  that sample. Any node with the dashboard on therefore shows the whole
+  cluster, and the other nodes need no dashboard of their own.
+- **Cost.** With no dashboard connected there is no timer, no IPC and no QUIC
+  traffic. While at least one is connected, one shared sampler takes a
+  snapshot every two seconds, keeps at most one in flight and skips ticks
+  while one runs, and pushes the same sample (a few KB per node) to every
+  connected dashboard. At most 16 dashboard sockets are admitted at once; the
+  17th is refused with `503`. On the engine side, request and byte counting is
+  a relaxed atomic add per response and per socket read or write. An enabled
+  dashboard with nobody watching runs no timer and sends no traffic; on a
+  three-node cluster against 11.0.0 its CPU cost per request was within 0.4 %,
+  and the engine's memory was unchanged. Its server modules and bundled page
+  add about 50 MiB to the Node.js process of each node once, at start, and
+  that does not grow with time or stored objects. With 16 dashboards open on
+  every node, large uploads ran about 1.5 % slower and the p99 latency was
+  about 5 % higher.
+- **Security.** The dashboard is read-only by construction: its socket
+  registers one read handler and nothing else, so it cannot start a healing
+  scan, change configuration or touch S3 traffic or stored data. It has no
+  authentication: protect it with the bind address and a firewall, for
+  example by binding it to a private interface only. It shows node IDs,
+  addresses, drive paths, counts and event messages, never credentials or
+  object keys.
+- **Versions.** Run every node on the same version, as clusters already
+  require. An engine that predates `NodeSnapshot` cannot decode the request,
+  logs a stream error and closes the stream, and the asking node shows it as
+  unavailable.
+
+`getClusterHealth()` probes every local drive (a write, sync and read of a
+probe file) on each call; poll `getClusterSnapshot()` instead when you need a
+cheap periodic view.
+
+## Supported S3 Operations
+
+The router fails closed, in standalone and cluster mode alike. A request
+selects one operation from its method, its path (`/`, `/{bucket}`,
+`/{bucket}/{key}`) and its query names, and each query name must be one that
+operation implements. Anything else gets `NotImplemented` (501, with the S3 XML
+error body) before the request takes a lock, reads its body or touches a bucket
+or object. A credential the request is not allowed for still gets
+`AccessDenied` (403) first. Until 11.0.0 such requests were served as another
+operation: `PUT ?tagging` overwrote the object with the XML body,
+`DELETE ?lifecycle` deleted an empty bucket, `DELETE ?versionId=…` deleted the
+current object, and `GET ?versioning` answered with an object listing.
+
+| Operation | Request | Query names besides the selector |
+|---|---|---|
+| ListBuckets | `GET /` | none |
+| CreateBucket, DeleteBucket, HeadBucket | `PUT`, `DELETE`, `HEAD /{bucket}` | none |
+| ListObjects | `GET /{bucket}` | `prefix`, `delimiter`, `max-keys`, `marker`, `encoding-type` |
+| ListObjectsV2 | `GET /{bucket}?list-type=2` | `prefix`, `delimiter`, `max-keys`, `continuation-token`, `start-after`, `fetch-owner`, `encoding-type` |
+| ListObjectVersions | `GET /{bucket}?versions` | `prefix`, `delimiter`, `max-keys`, `key-marker`, `version-id-marker`, `encoding-type` |
+| ListMultipartUploads | `GET /{bucket}?uploads` | `prefix`, `encoding-type` |
+| GetBucketLocation | `GET /{bucket}?location` | none |
+| GetBucketVersioning | `GET /{bucket}?versioning` | none |
+| GetBucketLifecycleConfiguration | `GET /{bucket}?lifecycle` | none |
+| GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy | `GET`, `PUT`, `DELETE /{bucket}?policy` | none |
+| DeleteObjects | `POST /{bucket}?delete` | none |
+| GetObject, HeadObject | `GET`, `HEAD /{bucket}/{key}` | `versionId`, `response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires` |
+| DeleteObject | `DELETE /{bucket}/{key}` | `versionId` |
+| PutObject, CopyObject | `PUT /{bucket}/{key}`, CopyObject with `x-amz-copy-source` | none |
+| CreateMultipartUpload | `POST /{bucket}/{key}?uploads` | none |
+| UploadPart, UploadPartCopy | `PUT /{bucket}/{key}?partNumber=…&uploadId=…` | none |
+| CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId=…` | none |
+| AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId=…` | none |
+| ListParts | `GET /{bucket}/{key}?uploadId=…` | `max-parts`, `part-number-marker` |
+
+Every operation also accepts the `x-id` label that AWS SDKs append and the SigV4
+query-signing parameters of a presigned URL (`X-Amz-Algorithm`,
+`X-Amz-Credential`, `X-Amz-Date`, `X-Amz-Expires`, `X-Amz-SignedHeaders`,
+`X-Amz-Signature`, `X-Amz-Security-Token`, `X-Amz-Content-Sha256`).
+Query-string signatures are not verified: a presigned request is served as an
+unsigned one, which is anonymous when auth is enabled.
+
+An object GET or HEAD (GetObject, HeadObject, ListParts) ignores query names S3
+does not define, as S3 does, so links with cache-busters (`?v=1`) or tracking
+parameters (`?fbclid=…`) keep working. A name S3 defines that the operation
+does not implement (`?acl`, `?partNumber=1`, …) is still refused there, and
+every other request (bucket level, every write and delete) refuses unknown
+names too. The set of names S3 defines is taken from the AWS SDK v3 model
+(3.1039.0). With CORS enabled, the 501 and 405 responses carry the CORS headers.
+
+The `response-*` parameters of GetObject and HeadObject replace the
+`Cache-Control`, `Content-Disposition`, `Content-Encoding`, `Content-Language`,
+`Content-Type` and `Expires` headers of a 200 or 206 response with the given
+value, verbatim. A value that is not a valid HTTP header value (a control
+character such as CR or LF) returns `InvalidArgument` (400). With auth enabled, an anonymous
+request that carries one returns `InvalidRequest` (400), as S3 does, because an
+override such as `response-content-type=text/html` would let a public object
+serve script from the storage origin; a presigned URL is such a request. With
+auth disabled the overrides apply.
+
+Versioning is never enabled on a bucket:
+- `GET ?versioning` returns the empty `<VersioningConfiguration/>` of a bucket
+  that was never versioned. `PUT ?versioning` is `NotImplemented`.
+- `GET ?lifecycle` returns `NoSuchLifecycleConfiguration` (404). `PUT` and
+  `DELETE ?lifecycle` are `NotImplemented`.
+- `GET ?versions` lists every current object as its S3 null version
+  (`VersionId` `null`, `IsLatest` true), with `CommonPrefixes` and paging on
+  `key-marker` and `version-id-marker`. A truncated page returns the last key
+  or common prefix as `NextKeyMarker`, plus `NextVersionIdMarker` `null` after a
+  key. A `version-id-marker` needs a `key-marker`.
+- `versionId` on GET, HEAD and DELETE, and a `?versionId=` after the copy source
+  of CopyObject and UploadPartCopy: `null` is the current object. A
+  well-formed id (a lowercase UUID) names no version and returns
+  `NoSuchVersion` (404; `NoSuchBucket` when the bucket is missing). Anything
+  else, the empty string included, returns `InvalidArgument` (400). A `?`
+  inside a copy-source key must be encoded as `%3F`.
+- These reads are authorized as `s3:GetBucketVersioning`,
+  `s3:GetLifecycleConfiguration` and `s3:ListBucketVersions`. A `versionId`,
+  `null` included, is authorized as `s3:GetObjectVersion` (GET, HEAD, and a
+  copy source's `?versionId=`) or `s3:DeleteObjectVersion` (DELETE), not as
+  `s3:GetObject` or `s3:DeleteObject`, so a policy that allows DeleteObject and
+  denies DeleteObjectVersion keeps protecting history once buckets are
+  versioned. Bucket tenants may use both actions in their bucket.
+
+`encoding-type=url` percent-encodes, on ListObjects, ListObjectsV2,
+ListObjectVersions and ListMultipartUploads, every UTF-8 byte outside
+`A-Z a-z 0-9 - . _ ~` and `/` (a space is `%20`, never `+`) in the elements S3
+encodes for that operation, and the response carries
+`<EncodingType>url</EncodingType>`:
+- ListObjects: `Prefix`, `Marker`, `NextMarker`, `Delimiter`, `Key` and
+  `CommonPrefixes/Prefix`.
+- ListObjectsV2: `Prefix`, `Delimiter`, `StartAfter`, `Key` and
+  `CommonPrefixes/Prefix`; `ContinuationToken` and `NextContinuationToken`
+  are opaque tokens, the same with or without `encoding-type`.
+- ListObjectVersions: `Prefix`, `KeyMarker`, `NextKeyMarker`, `Delimiter`,
+  `Key` and `CommonPrefixes/Prefix`.
+- ListMultipartUploads: `Prefix` and `Key`.
+
+Without the parameter keys are written as they are and no `EncodingType` is
+returned; any other value returns `InvalidArgument` (400). An object key may
+hold any UTF-8 character, also those XML 1.0 cannot carry (U+0001 and the other
+control characters except tab, line feed and carriage return): a listing that
+contains such a key is valid XML only with `encoding-type=url`. ListObjectsV2
+continuation tokens use only `A-Z a-z 0-9 - _`, so they never break the XML.
+
+Accepted but not applied: owners are never returned (`fetch-owner`); responses
+say so by omitting `Owner`. Refused with `NotImplemented` although S3
+defines them: every other subresource (`?acl`, `?tagging`, `?retention`,
+`?legal-hold`, `?cors`, `?website`, `?object-lock`, `?attributes`, `?restore`,
+`?select`, …), `partNumber` on GET and HEAD, the paging parameters of
+ListMultipartUploads (`key-marker`, `upload-id-marker`, `max-uploads`,
+`delimiter`), ListObjects V1 with V2 cursors, the ListBuckets parameters
+(`prefix`, `max-buckets`, `continuation-token`, `bucket-region`), headers
+hoisted into a presigned URL's query (`x-amz-checksum-crc32=…`) on anything but
+an object read, and any `POST` that is not one of the operations above.
 
 ## Usage with AWS SDK v3
 
@@ -999,7 +1448,23 @@ Client ──HTTP PUT──▶ Node A (coordinator)
 2. **Objects are chunked** — large objects split into fixed-size pieces (default 4 MB)
 3. **Each chunk is erasure-coded** — Reed-Solomon produces k data + m parity shards
 4. **Shards are distributed** — placed across different nodes and drives for fault isolation
-5. **Quorum guarantees consistency** — writes need k+1 acks, reads need k shards
+5. **Quorum guarantees consistency** — writes need k+1 acks, reads need k shards.
+   A chunk with fewer than k+m written shards is published with the missing
+   ones as pending shards that healing completes on their assigned drives.
+   A node that membership marks offline is never contacted: its shards of a
+   write are pending at once, reads take their shards from the other nodes,
+   and below either quorum a request fails at once. Until a stopped node is
+   marked offline (five missed heartbeat rounds), each chunk that needs it
+   waits up to the QUIC idle timeout (10 s). Cluster QUIC connections send
+   keep-alives every 2 s from both ends, so a slow but live peer stays
+   connected however long a request takes; there is no whole-request
+   deadline. A heartbeat to a silent peer therefore fails after 10 s. Each
+   heartbeat round goes to every peer at once, so stopped nodes are marked
+   offline after about 50 s by default, however many stop together (a whole
+   location, for example). A node whose restart takes longer than that comes
+   back to pending shards and healing work. `heartbeatTimeoutMs` (default 30 s) still bounds how long a live
+   peer may take to answer a heartbeat, but cannot extend the wait for a
+   silent one beyond 10 s.
 
 ### Cluster Root and Upgrade Requirements
 
@@ -1007,7 +1472,8 @@ Cluster mode never creates configured control or drive roots. Before startup,
 create `storage.directory` and every `cluster.drives.paths` entry, make them
 owned by the ObjectStorage process user, and remove group/world write permission.
 Every configured root must be a real directory with no symlink path component,
-and every drive root must resolve to a distinct device/inode identity. The
+and every drive root must resolve to a distinct device/inode identity and,
+unless `cluster.drives.allowSharedDevice` is set, to a distinct device. The
 control root may be distinct or may be the same identity as one drive root; it
 cannot make two drive entries aliases of one another. Every root must support
 the filesystem durability and locking semantics validated at startup.
@@ -1027,6 +1493,17 @@ has written or migrated the new state, do not restart an older binary and do not
 run a mixed-version cluster. Back up the cluster roots before upgrading; rollback
 requires restoring those roots from the pre-upgrade backup.
 
+Manifests with pending shards (written while a drive refused writes, see
+[Self-Healing](#self-healing)) are refused by engines that predate pending
+shards, because their chunks are short of placements. Before rolling a cluster
+back to such an engine, stop writing, complete or abort every open multipart
+upload (part descriptors record pending shards too, the scan does not count
+them, and an older engine drops them, so such an upload could not complete
+after the rollback), and then, on every node, run
+`runHealingScan()` until it reports `shardsPending` 0. Its scan starts after
+the request arrives, so it covers every write that finished before the call; a
+scan that could not list every manifest fails instead of reporting counts.
+
 ### Erasure Coding
 
 With the default `4+2` configuration:
@@ -1040,14 +1517,124 @@ With the default `4+2` configuration:
 | 6+3 | 9 | 50% | 3 failures | 5 |
 | 2+1 | 3 | 50% | 1 failure | 2 |
 
+### Streaming Reads
+
+A cluster GET streams the object chunk by chunk; nothing is written to the
+control root. It reads from the generation of the manifest it loaded at its
+start, without holding the object's operation gate, so overwrites, deletes and
+other GETs of the same key run while it streams. Each chunk is rebuilt from the
+first k valid shards, read in parallel, and at most one rebuilt chunk waits
+ahead of the client.
+
+- **Errors after the headers abort the response.** The first chunk is rebuilt
+  before the status line, so an unreadable object still gets an error status.
+  A failure in a later chunk ends the body with an error (the connection is
+  closed), never with a clean end of body. A whole-object GET sends its last
+  bytes only after the total length and the object MD5 match the manifest;
+  range GETs are checked for length only.
+- **One deadline for the whole GET.** `cluster.getDeadlineMs` is the longest a
+  single GET may run, counted from its start and including the client's
+  transfer time; a GET that takes longer is aborted, and if its client stopped
+  reading, its connection is closed. The default is 6 hours,
+  the allowed range 1 minute to 7 days. The cleanup grace is derived from it:
+  superseded and deleted generations keep their shards for the deadline plus
+  the smaller of the deadline and one hour (7 hours by default, 2 minutes at
+  the 1-minute floor), so every GET that started on a generation has ended
+  before cleanup frees it. Writes in progress hold their object's gate and are
+  never cleaned up, however short the grace. The setting must be the same on
+  every node; a mismatch can abort streams, but never serves wrong bytes.
+  CopyObject and CompleteMultipartUpload read their sources under the object
+  gate with their own fixed 6-hour limit, counted from the start of the
+  operation.
+- **GET memory budget.** Rebuilding chunks is charged to an engine-wide budget
+  of `max(1 GiB, 16 × chunk footprint)` at the configured chunk size. A chunk's
+  footprint is ⌈chunk/k⌉ × (k + min(k, m)) + chunk: the k fetched shards, the
+  data shards rebuilt for missing ones (a read rebuilds no parity shards) and
+  the decoded chunk; 4+2 at 4 MiB is 10 MiB, 1+15 at 64 MiB is 192 MiB. A GET
+  whose client went away keeps its charge until its local shard reads have
+  ended, and shutdown waits for them before it releases the drive roots. The
+  budget is separate from the upload budget. The first chunk waits at most 5 seconds for
+  budget and is otherwise refused with `503 SlowDown` and `Retry-After: 1`;
+  later chunks wait. Slow downloaders hold up to two rebuilt chunks each, so
+  many of them can exhaust the budget and turn new GETs into `503`s under load.
+
 ### QUIC Transport
 
 Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the `quinn` library:
-- 🔒 **Built-in TLS** — self-signed certs auto-generated at cluster init
+- **Mutual TLS or unauthenticated** — with `cluster.tls`, every connection is TLS 1.3 with a CA-signed certificate on both ends; without it, each node generates a self-signed certificate at startup and accepts any peer (see [Cluster transport trust](#cluster-transport-trust))
 - 🔀 **Multiplexed streams** — concurrent shard transfers without head-of-line blocking
 - ⚡ **Connection pooling** — persistent connections to peer nodes
 - 🌊 **Natural backpressure** — QUIC flow control prevents overloading slow peers
 
+### Cluster transport trust
+
+Without `cluster.tls` the cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's `quicPort` can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port. Run such a cluster only on a network where every host that can reach `quicPort` (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with `quicPort` firewalled from everything else.
+
+With `cluster.tls` set, the transport is mutual TLS (TLS 1.3, no session resumption, no early data) with a cluster CA you run:
+
+- **Identity.** A node's identity is the single DNS subject alternative name (SAN) of its certificate, and it equals `cluster.nodeId`. With `cluster.tls`, `nodeId` is required and must be a lowercase DNS name such as `node-1` or `node-1.storage.internal`; it is never generated, and it must still match the node id persisted in the control root. One certificate serves as both the server and the client certificate, so it must be valid for both: an extended key usage extension listing `serverAuth` and `clientAuth`, or no extended key usage extension at all (the recipe below issues both). Use a dedicated CA per cluster: every certificate it signs with one lowercase DNS SAN is a member of the cluster.
+- **Checks.** A node connects to a known peer only if the peer's certificate names exactly that peer's node id (one DNS SAN, no wildcard), and to a seed (`seedNodes`, known only by its address) only if the seed presents a certificate from the cluster CA. A node accepts a connection only with a client certificate from the cluster CA, and a join or heartbeat only for the node id that certificate names. A peer without such a certificate fails the handshake before it can send any request. Every CA-signed node is a full member: it may sync topology and read, write, tombstone and purge any shard, so the way to remove a compromised node is to stop trusting its certificate (a new CA, see the rotation below).
+- **Startup.** A node checks its own certificate the way its peers will (chain, validity, server and client authentication, the SAN, and that `keyFile` matches `certFile`) and refuses to start if that check or any file check below fails. The error names the file and the reason. A node never falls back to the unauthenticated transport. At startup it logs one line with its certificate's `notAfter` and the SHA-256 of each CA certificate, and `getClusterHealth().transport` reports them (see [Cluster Health](#cluster-health)).
+- **Files.** `caFile` holds one CA certificate, or two during a CA rotation; anything else is refused, which keeps a system CA bundle out. `certFile` holds the node certificate, optionally followed by intermediates, and nothing but certificates. `keyFile` holds exactly one unencrypted PKCS#8, SEC1 or PKCS#1 key; ECDSA P-256 is recommended. Each path must be absolute and canonical, with no symbolic link anywhere, and lie outside the control root and every drive root (roots are backed up and replaced, and a key must not travel with them). That check compares paths, so a bind mount that makes a root reachable under another path is not detected. Each file must be a regular file of at most 64 KiB. The files and their directories must be owned by the engine's user or by root and must not be group- or world-writable, and `keyFile` must be owned by the engine's user and grant no group or world permission (mode 0600 or 0400).
+- **Reload.** The files are read once at startup; a restart is the reload. Certificates are checked at every handshake, so a connection established before a certificate's `notAfter` keeps running until it reconnects, and new handshakes after `notAfter` fail. Renew certificates and restart the nodes before they expire.
+
+#### Making the CA and the node certificates
+
+The engine ships no CA tooling. With OpenSSL 3 or LibreSSL 3.3 (the `openssl` of macOS), create the CA once and keep `ca.key` offline. `ec_param_enc:named_curve` is required for LibreSSL, which otherwise writes the curve as explicit parameters that the engine refuses; OpenSSL 3 writes the named curve either way:
+
+```bash
+openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
+  -pkeyopt ec_param_enc:named_curve -out ca.key
+openssl req -x509 -new -key ca.key -sha256 -days 3650 \
+  -subj "/CN=objectstorage cluster CA" \
+  -addext "basicConstraints=critical,CA:TRUE" \
+  -addext "keyUsage=critical,keyCertSign,cRLSign" \
+  -out ca.pem
+```
+
+Then issue one certificate per node, with the node id as its only DNS SAN:
+
+```bash
+NODE_ID=node-1
+openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
+  -pkeyopt ec_param_enc:named_curve -out "$NODE_ID.key"
+openssl req -new -key "$NODE_ID.key" -subj "/CN=$NODE_ID" -out "$NODE_ID.csr"
+cat > "$NODE_ID.ext" <<EXT
+basicConstraints = critical,CA:FALSE
+keyUsage = critical,digitalSignature
+extendedKeyUsage = serverAuth,clientAuth
+subjectAltName = DNS:$NODE_ID
+EXT
+openssl x509 -req -in "$NODE_ID.csr" -CA ca.pem -CAkey ca.key -CAcreateserial \
+  -days 365 -sha256 -extfile "$NODE_ID.ext" -out "$NODE_ID.pem"
+```
+
+Install `ca.pem` and the node's certificate and key on the node, as the user that runs the engine and outside its control and drive roots:
+
+```bash
+install -d -m 0755 /etc/objectstorage/tls
+install -m 0644 ca.pem "$NODE_ID.pem" /etc/objectstorage/tls/
+install -m 0600 "$NODE_ID.key" /etc/objectstorage/tls/
+```
+
+Then set `cluster.tls` to the three paths (see [Configuration](#configuration)). The test suite runs exactly these commands (`test/test.cluster-tls.node.ts`).
+
+**Docker secrets** are mounted owned by root with mode 0444 by default, which the key check refuses. Set the key secret's `uid` to the engine's user and its `mode` to `0400` in the service's long `secrets` syntax. Docker Compose ignores `uid`, `gid` and `mode` for secrets read from a file, because it bind-mounts the file; there the host file itself must be owned by the engine's uid with mode 0600 or 0400. Kubernetes secret volumes present their files through symbolic links, which the canonical-path rule refuses.
+
+#### Enabling mutual TLS
+
+Nodes with and without `cluster.tls` cannot talk to each other: a TLS node refuses peers without a client certificate and refuses self-signed servers. Enabling TLS is therefore not a rolling change: stop every node, configure `cluster.tls` on every node, then start every node. A node whose `nodeId` was generated keeps that id (the control root persists it, and `getClusterHealth().nodeId` reports it): configure it as `nodeId` and issue the node's certificate for it, or the node refuses to start. Disabling TLS takes the same full stop.
+
+#### Rotating certificates and the CA
+
+A node certificate is renewed by issuing a new one from the same CA and restarting the node; its peers need no change. A new CA takes three rolling restarts, with no cluster stop:
+
+1. Put both CAs into every node's `caFile` (two certificates) and restart the nodes one at a time.
+2. Issue every node a certificate from the new CA and restart the nodes one at a time.
+3. Remove the old CA from every `caFile` and restart the nodes one at a time.
+
+Every pair of nodes stays connected throughout, because trust only grows before the certificates change and only shrinks after. The order matters: a node with a certificate from the new CA cannot connect to a node that trusts only the old one.
+
 ### Cluster Membership
 
 - **Static seed nodes** — initial cluster defined in config
@@ -1057,13 +1644,40 @@ Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the
 
 ### Self-Healing
 
-A background scanner periodically (default: every 24h):
-1. Checks shard checksums (CRC32C) for bit-rot detection
-2. Identifies shards on offline nodes
-3. Reconstructs missing shards from remaining data using Reed-Solomon
-4. Places healed shards on healthy drives
-
-Healing runs at low priority to avoid impacting foreground I/O.
+Every 24 hours, and whenever `runHealingScan()` asks for it, each node scans
+the manifests it coordinated (manifests live on the node that wrote them):
+1. Every shard placement is checked at the drive it names, locally or through
+   its node, against the checksum and size the manifest recorded. The check
+   reads the shard record's header and the record's length only, never its
+   payload, so a scan costs a few small reads per shard. Verifying payloads
+   is not the scan's job: a read detects a rotted payload through its CRC32C
+   and rebuilds the chunk from the other shards.
+2. A missing shard is rebuilt from k valid shards with Reed-Solomon, checked
+   against the recorded checksum, and written to the drive its placement
+   names. The manifest does not change.
+3. A pending shard is written to the drive its manifest recorded and then
+   moved into the chunk's placements, in the same version and generation; the
+   completed manifest is identical to one whose write never failed.
+
+Healing never places a shard on a drive of its own choosing. Placements and
+pending shards on an offline node wait for that node; a pending shard whose
+drive refuses writes stays pending and is reported in `repairs.shardsPending`
+(replacing a permanently lost drive is not supported yet). A shard record that
+cannot be read, such as a truncated one, is reported in `repairs.failed` and
+left in place, because shard records are immutable; a node that cannot read a
+record answers with an error, never as if the shard were missing. A minority
+partition does not heal: its scan fails. Placements are checked without the
+object's operation gate, as a GET reads them. The chunks of an object that need
+a write are healed in batches of up to 1 GiB of chunk data: each batch takes
+the object's gate, reloads the manifest, heals only chunks that are unchanged
+in the same generation, publishes its pending shards with one manifest store
+and releases the gate before the next batch. An object whose gate is busy is
+skipped until the next scan.
+
+Scans never overlap. A scan that outlasts the 24-hour interval delays the next
+one. `runHealingScan()` joins the next scan that starts after its request and
+never a scan that is already running; requests that arrive together share one
+scan.
 
 ### Erasure Set Formation
 
@@ -1083,7 +1697,7 @@ Drives are interleaved across nodes for maximum fault isolation. New nodes form
 
 ```typescript
 import { ObjectStorage } from '@lossless.org/objectstorage';
-import { tap, expect } from '@git.zone/tstest/tapbundle';
+import { tap, expect } from '@git.zone/tsrepo/tapbundle';
 
 let storage: ObjectStorage;
 
@@ -1266,6 +1880,24 @@ List scoped tenant credential metadata without returning secrets.
 
 Return endpoint, port, region, bucket, access key, secret key, SSL flag, legacy descriptor fields, and S3/AWS env values for the bucket tenant.
 
+#### `createPrefixCredential(input: IPrefixCredentialInput): Promise<IPrefixCredential>`
+
+Create an expiring credential limited to one prefix of one bucket, or install
+one created on another node (`accessKeyId`, `secretAccessKey`, `expiresAt`). An
+identical replay returns the stored record. The secret is returned only here.
+See [Prefix Credentials](#prefix-credentials) for the input rules and the
+`ObjectStorageCredentialError` codes.
+
+#### `revokePrefixCredential(options): Promise<{ revoked: boolean }>`
+
+Revoke a prefix credential and keep its ID refused until the latest moment it
+could still be created. Options: `{ accessKeyId }`. `revoked` is `true` only
+when a live credential was removed; unknown IDs are recorded as revoked too.
+
+#### `listPrefixCredentials(options?): Promise<IPrefixCredentialMetadata[]>`
+
+List live prefix credentials without secrets. Options: `{ bucketName? }`.
+
 #### `getBucketMigrationCapability(): Promise<IObjectStorageBucketMigrationCapability>`
 
 Return standalone provider identity and receipt-key evidence plus source seal,
@@ -1324,7 +1956,9 @@ the bucket's exclusive credential. Options:
 
 Delete a fenced bucket. Supplying `accessKeyId` also durably revokes that
 exact tenant credential and deletes the policy; omitting it preserves the
-legacy credential/policy behavior. Options:
+legacy credential/policy behavior. The bucket's prefix credentials are revoked
+first in both cases, and the deletion is refused if that revocation cannot be
+stored. Options:
 `{ bucketName, accessKeyId?, fence, holdPublication? }`. Setting
 `holdPublication: true` returns a durable `resourcePublicationBarrier`; omitted
 or `false` preserves immediate behavior.
@@ -1382,7 +2016,7 @@ Atomically replace the active runtime credential set without restarting the serv
 
 #### `getClusterHealth(): Promise<IClusterHealth>`
 
-Read the Rust core's current cluster, drive, quorum, and repair health snapshot. Standalone mode returns `{ enabled: false }`.
+Read the Rust core's current cluster, drive, quorum, repair, and transport health snapshot. Standalone mode returns `{ enabled: false }`.
 
 In cluster mode, `quicAddress` is the local QUIC/UDP socket's actual bound
 address. With `cluster.quicPort: 0`, the operating system assigns its port; use
@@ -1390,6 +2024,35 @@ address. With `cluster.quicPort: 0`, the operating system assigns its port; use
 Membership advertisements and peer `quicAddress` values contain that same bound
 address, including after a restart.
 
+#### `getClusterSnapshot(): Promise<IObjectStorageClusterSnapshot>`
+
+The [dashboard](#dashboard)'s snapshot, built by the engine from memory only:
+`local` is this node's `IObjectStorageNodeSnapshot` (counters, totals, drives
+from the last heartbeat probe, healing state, recent events) and, in cluster
+mode, `cluster` adds quorum, the erasure layout and every peer, with the peer's
+own snapshot when it answered within a second. One snapshot runs at a time; a
+call while another runs rejects. Standalone mode returns `cluster: null`.
+
+#### `dashboardPort: number | undefined`
+
+The dashboard's bound TCP port while it runs; `undefined` while it is off or
+stopped.
+
+#### `runHealingScan(options?: IRunHealingScanOptions): Promise<IClusterHealingScanResult>`
+
+Run a healing scan on this cluster node (see [Self-Healing](#self-healing)) and
+return its `startedAt`, `completedAt`, `durationMs`, `shardsChecked`,
+`shardsHealed`, `shardsPending` and `failed`. The request joins the next scan
+that starts after it arrives; a scan already running finishes first. It
+rejects in standalone mode, when the scan fails (without a majority of the
+cluster, or when a bucket's manifests cannot be listed) and when the engine
+shuts down before the scan completes, and it is refused while 16 requests
+already wait for a scan to start. The engine answers other management calls
+while the scan runs. `options.timeoutMs` (default 300,000) bounds the wait;
+the engine cannot cancel a scan, so one that outlasts it keeps running, its
+result is discarded, and a later call joins the next scan that starts after
+it.
+
 #### `getHealth(): Promise<IObjectStorageHealth>`
 
 Return running state, storage directory, per-location pool observations, auth
@@ -1416,7 +2079,7 @@ objectstorage uses a **hybrid Rust + TypeScript** architecture:
 │  │   ├─ FileStore (single-node mode)        │
 │  │   └─ DistributedStore (cluster mode)     │
 │  │       ├─ ErasureCoder (Reed-Solomon)     │
-│  │       ├─ ShardStore (per-drive storage)  │
+│  │       ├─ ShardBackend (per-drive shards) │
 │  │       ├─ QuicTransport (quinn)           │
 │  │       ├─ ClusterState & Membership       │
 │  │       └─ HealingService                  │
@@ -1433,7 +2096,7 @@ objectstorage uses a **hybrid Rust + TypeScript** architecture:
 
 **Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves these with bounded-memory streaming I/O, backpressure, and direct `seek()` for range requests.
 
-**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, and `getClusterHealth`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes.
+**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, `createPrefixCredential`, `revokePrefixCredential`, `listPrefixCredentials`, `getPrefixCredentialStatus`, `getClusterHealth`, `getClusterSnapshot`, and `runHealingScan`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes. `runHealingScan` answers when its scan completes, with the request's `id`; the engine serves other commands meanwhile, and a shutdown answers a pending request with an error before the `stop` response. `getClusterSnapshot` is answered the same deferred way, because a cluster node waits up to a second for each peer.
 
 ### S3-Compatible Operations
 
@@ -1483,7 +2146,8 @@ those versions cannot read the canonical v2 objects.
 | ListParts | `GET /{bucket}/{key}?uploadId` | Optional `max-parts` and `part-number-marker` |
 | CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId` | |
 | AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId` | |
-| ListMultipartUploads | `GET /{bucket}?uploads` | |
+| ListMultipartUploads | `GET /{bucket}?uploads` | Optional `prefix` |
+| GetBucketLocation | `GET /{bucket}?location` | Returns `server.region`; an empty constraint for `us-east-1` |
 | GetBucketPolicy | `GET /{bucket}?policy` | |
 | PutBucketPolicy | `PUT /{bucket}?policy` | |
 | DeleteBucketPolicy | `DELETE /{bucket}?policy` | |
@@ -1507,6 +2171,14 @@ response echoes `ContinuationToken` and `StartAfter` when the request carried
 them, reports `KeyCount` as contents plus common prefixes, and returns
 `NextContinuationToken` whenever the page is truncated.
 
+A continuation token is opaque: pass `NextContinuationToken` back unchanged.
+It is the unpadded base64url (RFC 4648 §5) encoding of the UTF-8 key the next
+page resumes after, at most 1,366 characters, and is the same with or without
+`encoding-type=url`, so a client may change `encoding-type` between pages. A
+token the server could not have issued (outside the base64url alphabet, padded,
+not canonical, or not decoding to a 1 to 1,024-byte UTF-8 key) returns
+`InvalidArgument` (400); so do tokens issued before 11.0.0, which were raw keys.
+
 **ListObjects v1** (no `list-type`) reads `prefix`, `delimiter`, `max-keys` and
 `marker`, and ignores the V2 cursors. The response always carries `Marker`,
 empty when the request carried none, and carries `NextMarker` whenever the page
@@ -1523,14 +2195,16 @@ Paging rules shared by both versions:
   cursor.
 - `max-keys` above 1000 is clamped to 1000 and the response echoes the clamped
   value in `MaxKeys`, as S3 does. A `max-keys` value that is not a non-negative
-  integer is rejected with `InvalidRequest`, and so is a cursor longer than the
-  1,024-byte S3 key bound.
+  integer is rejected with `InvalidRequest`, and so is a `start-after` or
+  `marker` longer than the 1,024-byte S3 key bound.
 - An empty `start-after`, `continuation-token` or `marker` value means no cursor.
 - Keys are returned in ascending UTF-8 byte order, and a key is never returned
   twice across pages of an unchanged bucket.
 
 ListMultipartUploads is not paginated: it returns every in-flight upload of the
-bucket with `IsTruncated` false.
+bucket with `IsTruncated` false. An optional `prefix` parameter limits the
+result to uploads whose keys start with it and is echoed in the response;
+prefix credentials must supply one inside their prefix.
 
 ### On-Disk Format
 
@@ -1552,6 +2226,8 @@ bucket with `IsTruncated` false.
     {bucket}.policy.json                 # Bucket policy (IAM JSON format)
   .smartstorage/staging/v1/              # Private staging: a write lands here first; its
                                          # key leaf appears only with the complete payload
+  .smartstorage/credentials.json         # Runtime and bucket tenant credentials (0600)
+  .smartstorage/prefix-credentials.json  # Prefix credentials and revocations (0600)
 ```
 
 Rolling a standalone root back from 10.7.0 or later to 8.0.3–10.6.0: a crash
@@ -1575,7 +2251,8 @@ were never published.
       {object-address}.state.json
                                          # Active/tombstoned object manifest
     .multipart/{upload-id}/               # Descriptor-anchored multipart state
-    .get-tmp/                             # Unlinked GET reconstruction files
+  .smartstorage/credentials.json          # This node's runtime credentials (not replicated)
+  .smartstorage/prefix-credentials.json   # This node's prefix credentials (not replicated)
   .smartstorage/cluster/
     identity.json                         # Node and cluster identity
     topology.json                         # Last durable topology
@@ -1588,8 +2265,12 @@ were never published.
 ```
 
 Superseded physical generations are queued with their exact shard placements.
-Cluster reads are bounded, cleanup waits a 24-hour reader grace period, and the
-hourly cleanup worker purges both shard records and tombstones. Multipart
+They keep their shards for the cleanup grace derived from `cluster.getDeadlineMs`
+(see Streaming Reads), and the cleanup worker, which runs once per grace period
+and at least hourly, then purges both shard records and tombstones. Earlier
+engines spooled GET responses into `.manifests/.get-tmp/`; startup removes a
+leftover spool directory once it holds only spool files, and otherwise leaves
+it in place, unused. Multipart
 uploads older than `multipart.expirationDays`, plus committed uploads whose
 first cleanup attempt failed, are retried during bounded cleanup passes
 scheduled every `multipart.cleanupIntervalMinutes`; large registries are
@@ -1612,7 +2293,7 @@ established.
 
 - [`@push.rocks/smartbucket`](https://code.foss.global/push.rocks/smartbucket) — High-level S3-compatible abstraction layer
 - [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) — TypeScript ↔ Rust IPC bridge
-- [`@git.zone/tsrust`](https://code.foss.global/git.zone/tsrust) — Rust cross-compilation for npm packages
+- [`@git.zone/rsrepo`](https://code.foss.global/git.zone/rsrepo) — Rust builds and cross-compilation for npm packages
 
 ## License and Legal Information